Embedded Computing Cluster: NFS Server Analysis Abraham Murray Richard Draves and Jitendra Padhye have done significant research to Mesh networks. Richard Draves did a prototype of IPv6 for Windows NT. Jitendra Padhye worked for ACIRI, now called ICIR. (ICSI Center for Internet Research) CS577: Advanced Computer Networks

Outline Project Abstract System Overview Measurement Approach Measured Results Aggregate Results Per Client Results Per File Results Per User Results Result Analysis CS577: Advanced Computer Networks

Abstract Any nodes using disproportionate resources? Our Embedded Computing Cluster: Runs 80 diskless nodes to save cost Uses high bandwidth data fabrics Allows NFS file systems to perform well 1 NFS server for 80 nodes Runs proprietary software (light on details) Open-source Linux infrastructure Intent is to measure and analyze NFS server performance Any nodes using disproportionate resources? What are the five busiest client nodes? Is the workload bursty or constant? How does the workload change over time? What files/directories are most heavily requested? CS577: Advanced Computer Networks

Diskless Node (Router) System Overview System nodes embedded PowerPC Combination of Myrinet data fabric, and Gigabit Ethernet Myrinet is point-to-point switched data fabric, 2 Gbps full duplex Multi-role system, variety of proprietary software VoIP, communications support, DSP “General Purpose Processors” Myrinet Gigabit Ethernet System Server nfsdump Diskless Node HDD NFS Server ram disk NFS Client Diskless Nodes Diskless Node (Router) Diskless Nodes CS577: Advanced Computer Networks

Measurement Approach Measurement Tools: Measurement Approach: nfsdump modification to support NFS-specific attributes nfsscan Perl script to automate data analysis of nfsdump outputs Presented at LISA XVII, October 2003 by Ellard, Seltzer Measurement Approach: Task system consistently Run tools on NFS server (ISSUE!) Anonymize results for presentation CS577: Advanced Computer Networks

Aggregate Server Results Table 1 – Aggregate Server Operation Counts[1] TOTAL read write lookup getattr access create remove 371282 79326 938 6663 72049 209737 209 411 [1] #cmdline ./nfsscan -t 0 -l -o nfsscan_results ./nfsdump_results.txt Table 2 – Server Statistics Operation Type Operations per Second Percent of Total TOTAL 240   read 51 21% write 1 0% lookup 4 2% getattr 47 19% access 136 56% create remove CS577: Advanced Computer Networks

Aggregate Server Results booting 1st software load 2nd software load ! CS577: Advanced Computer Networks

Per Client Results CS577: Advanced Computer Networks Table 3 – 15 Busiest Clients Operation Counts[1] client TOTAL read write lookup getattr access create remove 0.0.0.70 102658 8440 21 207 26495 67432 5 12 0.0.0.66 66523 6517 26 213 16783 42922 4 11 0.0.0.90 30942 5583 31 251 4561 20446 6 0.0.0.82 15720 2145 35 290 3247 9905 8 13 0.0.0.71 12549 6477 16 229 1292 4470 9 0.0.0.67 12220 6032 48 239 1352 4445 10 0.0.0.72 9203 4394 28 244 1110 3339 14 0.0.0.91 8006 4149 184 799 2789 1 7 0.0.0.65 7854 1926 56 312 1225 4186 18 0.0.0.84 7758 2235 313 1296 3818 0.0.0.80 7681 1820 93 230 1083 4295 15 0.0.0.89 7383 1198 43 197 1314 4536 0.0.0.92 7224 3881 51 187 641 2376 0.0.0.87 6700 1853 17 1092 3449 0.0.0.85 6372 2079 340 1035 2820 [1] #cmdline ./nfsscan -t 0 -BC -l -o nfsscan_results ./nfsdump_results.txt CS577: Advanced Computer Networks

Per Client Results – DSP Nodes Clients #68, 70, 71 retasked at 18 minutes to run concurrent DSP ! ! ! CS577: Advanced Computer Networks

Per Client Results – VoIP Node Client 90 boots in 1.6 minutes, launches initial software from 2-3.5 minutes CS577: Advanced Computer Networks

Per Client Results – Router Node Client 65 boots in 1.7 minutes. This is a Gigabit-Myrinet routing node DHCP-relay, IP-forwarding, OSPF routing node CS577: Advanced Computer Networks

Per File Results CS577: Advanced Computer Networks Table 4 – 20 Busiest Files and Directories Operation Counts[1] fileid TOTAL read write lookup getattr access create remove 1544 43733 10 21972 21751 249 28167 31 49 28067 274 26811 36 134 26641 38 25995 56 6 25914 5 12 340 16343 68 3 16255 4 11 525 12807 189 10285 393 37 9793 1785 7981 166 6996 90 30 6827 1245 5936 3162 1422 1352 55 5063 556 908 3599 161 5042 118 33 4891 115 4695 34 65 4596 91 4409 131 24 4254 7 4385 13 21 4351 103 4232 58 4161 4134 54 4074 640 3656 3563 50 43 1821 3520 3516 1244 3476 405 1561 1510 646 3145 2947 97 101 [1] #cmdline ./nfsscan -t 0 -BF -f -d -l -o nfsscan_results ./nfsdump_results.txt CS577: Advanced Computer Networks

Per File Results - Histogram CS577: Advanced Computer Networks

Per User Results CS577: Advanced Computer Networks Table 5 – Per User Operation Counts[1] user TOTAL read write lookup getattr access create remove root 360790 73265 938 5835 69027 209200 209 411 user #1 10491 6061 828 3021 537 [1] #cmdline ./nfsscan -t 0 -BU -l -o nfsscan_results ./nfsdump_results.txt CS577: Advanced Computer Networks

Result Analysis The top 5 busiest nodes are responsible for 60% of NFS traffic The NFS workload is bursty, and follows operator-system tasking Majority of accessed files are touched less than 80 times Once per client More regularly accessed files might belong in ramdisks Reduce # NFS Operations by up to 60%! CS577: Advanced Computer Networks