1. Performance Analysis and Tuning Presented by Carol ThornborrowPresented by Carol Thornborrow

2. System Performance User “perceived performance”

3. System Performance System performance is the efficiency which system resources respond to current demands on the system.System performance is the efficiency which system resources respond to current demands on the system. Service Level Agreement (SLA) - Agreement between management, the user community and the information technology staff.

4. System Performance What is “normal”?What is “normal”? Create a baseline for comparison.Create a baseline for comparison.

5. Monitoring Resources which affect system performanceResources which affect system performance –CPU –Memory –Disk I/O –Network I/O Resources affect on system performanceResources affect on system performance –Not enough of the resource –Allocation of the resource is being mismanaged

6. Monitoring Identifying bottlenecksIdentifying bottlenecks –What is the problem? –Identify the Cause –What will it take to fix? –Is the problem fixed? –Reevaluate, start over Monitoring ToolsMonitoring Tools –OS utilities vmstat, netstatvmstat, netstat –Vendor products GlancePlus, SarCheckGlancePlus, SarCheck

7. Monitoring - CPU Overall utilizationOverall utilization –vmstat on most systems –sar –u Solaris and HP-UX Load AverageLoad Average –uptime Per process CPU timePer process CPU time –ps –elf on HP-UX, Solaris –ps –aux on Red Hat, Free BSD –top - real time –pstree init-+-alarmdinit-+-alarmd |-7*[kdeinit] |-7*[kdeinit]

8. Monitoring - CPU vmstat 5 4vmstat 5 4 procs cpu r b w us sy id 1 0 0 3 5 92 0 0 00 1 99 0 0 02 0 98 -r > 3-6/CPU -us + sy > 80% sar –u 5 4 %usr %sys %wio %idle 0 0 0 100 0 1 1 98 0 0 0 100 Average 0 0 0 99

9. Tuning - CPU Manually change priorityManually change priority –System V, Free BSD renice new nice# pidrenice new nice# pid –Solaris also has priocntl –s –p new-pri –i pid proc-idpriocntl –s –p new-pri –i pid proc-id Replace CPUReplace CPU Add CPUAdd CPU Redistribute loadRedistribute load –At – run process at specific time –Batch – run processes sequentially –Idle CPU

10. Monitoring - Memory Active virtual memoryActive virtual memory –Solaris swap –l or sar -r –HP-UX swapinfo –Free BSDpstat -s –Red Hatswapon –s Paging ratePaging rate –vmstat # of running processes# of running processes –vmstat SwapsSwaps –vmstat

11. Monitoring - Memory vmstat 5 4vmstat 5 4 r w swap free re po 0 0 509552 38512 7 0 0 0 445184 35600 0 0 0 0 445184 35600 0 0 0 0 445184 35600 0 0 sar –r 5 4sar –r 5 4 16:03:06 freemem freeswap 16:03:11 4413 888080 16:03:16 4409 888080 16:03:21 4409 888080 16:03:26 4409 888080

12. Tuning – Memory Add more memoryAdd more memory Distribute swap spaceDistribute swap space Distribute jobs – at, batchDistribute jobs – at, batch Virtual memory and pagingVirtual memory and paging –Free BSD sysctl Increase vm.v_cache_min and/or vm.v_cache_max to favor fs buffer cache over user processesIncrease vm.v_cache_min and/or vm.v_cache_max to favor fs buffer cache over user processes

13. Monitoring – Disk I/O Avg size of transfersAvg size of transfers iostat iostat dad0 dad0 kps tps kps tps 4 1 4 1 94 26 94 26 kps/tps = Kb/transfer kps/tps = Kb/transfer % of disk utilization% of disk utilization iostat –D iostat –D ssd0 ssd0 rps wps util rps wps util 0 7 4.2 0 7 4.2 0 8 2.6 0 8 2.6 System V –sar –d Need accounting turned on May affect performance

14. Tuning – Disk I/O Spread active files over several disks and controllersSpread active files over several disks and controllers Sequential files are contiguous and near outer edge of diskSequential files are contiguous and near outer edge of disk Place randomly accessed files near center of diskPlace randomly accessed files near center of disk Place root system on own disk if possiblePlace root system on own disk if possible Disk stripingDisk striping Purchase fast disks w fast controllersPurchase fast disks w fast controllers

15. Monitoring and Tuning Network I/O netstat –inetstat –i –Collisions > 5% of total calls pingping nfsstat –cnfsstat –c –retrans > 5% of calls badxid = retrans - > serverbadxid = retrans - > server badxid networkbadxid network Spread the network load between servers Add more bandwidth