1. Embedded Computing Cluster: NFS Server Analysis
Abraham Murray
Richard Draves and Jitendra Padhye have done significant research to Mesh networks.
Richard Draves did a prototype of IPv6 for Windows NT.
Jitendra Padhye worked for ACIRI, now called ICIR. (ICSI Center for Internet Research)
CS577: Advanced Computer Networks

2. Outline Project Abstract System Overview Measurement Approach
Measured Results
Aggregate Results
Per Client Results
Per File Results
Per User Results
Result Analysis
CS577: Advanced Computer Networks

3. Abstract Any nodes using disproportionate resources?
Our Embedded Computing Cluster:
Runs 80 diskless nodes to save cost
Uses high bandwidth data fabrics
Allows NFS file systems to perform well
1 NFS server for 80 nodes
Runs proprietary software (light on details)
Open-source Linux infrastructure
Intent is to measure and analyze NFS server performance
Any nodes using disproportionate resources?
What are the five busiest client nodes?
Is the workload bursty or constant?
How does the workload change over time?
What files/directories are most heavily requested?
CS577: Advanced Computer Networks

4. Diskless Node (Router)
System Overview
System nodes embedded PowerPC
Combination of Myrinet data fabric, and Gigabit Ethernet
Myrinet is point-to-point switched data fabric, 2 Gbps full duplex
Multi-role system, variety of proprietary software
VoIP, communications support, DSP
“General Purpose Processors”
Myrinet
Gigabit Ethernet
System Server
nfsdump
Diskless Node
HDD
NFS Server
ram disk
NFS Client
Diskless Nodes
Diskless Node (Router)
Diskless Nodes
CS577: Advanced Computer Networks

5. Measurement Approach Measurement Tools: Measurement Approach:
nfsdump modification to support NFS-specific attributes
nfsscan Perl script to automate data analysis of nfsdump outputs
Presented at LISA XVII, October 2003 by Ellard, Seltzer
Measurement Approach:
Task system consistently
Run tools on NFS server (ISSUE!)
Anonymize results for presentation
CS577: Advanced Computer Networks

6. Aggregate Server Results
Table 1 – Aggregate Server Operation Counts[1]
TOTAL
read
write
lookup
getattr
access
create
remove
371282
79326
938
6663
72049
209737
209
411
[1] #cmdline ./nfsscan -t 0 -l -o nfsscan_results ./nfsdump_results.txt
Table 2 – Server Statistics
Operation Type
Operations per Second
Percent of Total
TOTAL
240
read 51
21%
write 1 0%
lookup 4 2%
getattr 47
19%
access
136
56%
create
remove
CS577: Advanced Computer Networks

7. Aggregate Server Results
booting
1st software load
2nd software load !
CS577: Advanced Computer Networks

8. Per Client Results CS577: Advanced Computer Networks
Table 3 – 15 Busiest Clients Operation Counts[1]
client
TOTAL
read
write
lookup
getattr
access
create
remove
102658
8440 21
207
26495
67432 5 12
66523
6517 26
213
16783
42922 4 11
30942
5583 31
251
4561
20446 6
15720
2145 35
290
3247
9905 8 13
12549
6477 16
229
1292
4470 9
12220
6032 48
239
1352
4445 10
9203
4394 28
244
1110
3339 14
8006
4149
184
799
2789 1 7
7854
1926 56
312
1225
4186 18
7758
2235
313
1296
3818
7681
1820 93
230
1083
4295 15
7383
1198 43
197
1314
4536
7224
3881 51
187
641
2376
6700
1853 17
1092
3449
6372
2079
340
1035
2820
[1] #cmdline ./nfsscan -t 0 -BC -l -o nfsscan_results ./nfsdump_results.txt
CS577: Advanced Computer Networks

9. Per Client Results – DSP Nodes
Clients #68, 70, 71 retasked at 18 minutes to run concurrent DSP ! ! !
CS577: Advanced Computer Networks

10. Per Client Results – VoIP Node
Client 90 boots in 1.6 minutes, launches initial software from minutes
CS577: Advanced Computer Networks

11. Per Client Results – Router Node
Client 65 boots in 1.7 minutes. This is a Gigabit-Myrinet routing node
DHCP-relay, IP-forwarding, OSPF routing node
CS577: Advanced Computer Networks

12. Per File Results CS577: Advanced Computer Networks
Table 4 – 20 Busiest Files and Directories Operation Counts[1]
fileid
TOTAL
read
write
lookup
getattr
access
create
remove
1544
43733 10
21972
21751
249
28167 31 49
28067
274
26811 36
134
26641 38
25995 56 6
25914 5 12
340
16343 68 3
16255 4 11
525
12807
189
10285
393 37
9793
1785
7981
166
6996 90 30
6827
1245
5936
3162
1422
1352 55
5063
556
908
3599
161
5042
118 33
4891
115
4695 34 65
4596 91
4409
131 24
4254 7
4385 13 21
4351
103
4232 58
4161
4134 54
4074
640
3656
3563 50 43
1821
3520
3516
1244
3476
405
1561
1510
646
3145
2947 97
101
[1] #cmdline ./nfsscan -t 0 -BF -f -d -l -o nfsscan_results ./nfsdump_results.txt
CS577: Advanced Computer Networks

13. Per File Results - Histogram
CS577: Advanced Computer Networks

14. Per User Results CS577: Advanced Computer Networks
Table 5 – Per User Operation Counts[1]
user
TOTAL
read
write
lookup
getattr
access
create
remove
root
360790
73265
938
5835
69027
209200
209
411
user #1
10491
6061
828
3021
537
[1] #cmdline ./nfsscan -t 0 -BU -l -o nfsscan_results ./nfsdump_results.txt
CS577: Advanced Computer Networks

15. Result Analysis
The top 5 busiest nodes are responsible for 60% of NFS traffic
The NFS workload is bursty, and follows operator-system tasking
Majority of accessed files are touched less than 80 times
Once per client
More regularly accessed files might belong in ramdisks
Reduce # NFS Operations by up to 60%!
CS577: Advanced Computer Networks