1. Acegene IT Co. Ltd.1 Linux 操作系统 系统优化 周炯上海艾基信息技术有限公司

2. Acegene IT Co. Ltd. 2 内容提要 Performance concepts and measurement tools: –CPU –Memory –Disk I/O –Network –Process –Profiling Kernel & Applications

3. Measurement Tools Linux has measurement tools common to UNIX platforms: –System Activity Reporter: sar Historical database –Virtual Memory Statistics: vmstat, free –I/O Statistics: iostat –Top Resource Consumers: top -c –X-based tools: xosview, gnome- system-monitor, ksysguard, gkrellm

4. CPU Measurements Identify the CPU statistics and interpret them: –Idle time: %idle –Executing user code: %user –Executing system code: %system –Load average: runq-sz Start with cat /proc/cpuinfo –Total CPU count/speed There are several ways to see the same information, for example: CPU activity

5. CPU Activity - mpstat # mpstat -P # mpstat -P ALL 1 1 Linux 2.4.21-15.EL (raclinux1) 08/04/2004 CPU %user %nice %system %idle intr/s all 19.50 0.00 31.50 49.00 111.00 0 27.00 0.00 42.00 31.00 111.00 1 12.00 0.00 21.00 67.00 111.00 What is my CPU doing?: mpstat

6. CPU Activity - sar # sar -u 2 3 Linux 2.4.21-15.EL (raclinux1) 08/04/2004 02:02:34 AM CPU %user %nice %system %idle 02:02:36 AM all 1.14 0.00 1.71 97.14 02:02:38 AM all 2.62 0.00 3.66 93.72 02:02:40 AM all 0.54 0.00 3.78 95.68 Average: all 1.45 0.00 3.09 95.46 # sar -u What is my CPU doing?: sar

7. CPU Activity - iostat # iostat -c 2 3 Linux 2.4.21-15.EL (raclinux1) 08/04/2004 avg-cpu: %user %nice %sys %idle 5.73 0.00 17.16 77.11 avg-cpu: %user %nice %sys %idle 4.62 0.00 27.18 68.21 avg-cpu: %user %nice %sys %idle 9.55 0.00 35.96 54.49 # iostat -c What is my CPU doing?: iostat

8. Linux Virtual Memory Each process is assigned a contiguous address space in virtual memory that maps process memory allocations to the real memory –User address space is a contiguous set of pages and is limited to 4GB per process –Physical Address Extension (PAE), allows access to more than 4GB of physical memory RHat: ‘ enterprise ’ kernel (implied SMP) SuSE: 64GB kernel (implied SMP)

9. Swap Swap space (disk) Physical memory (RAM) Virtual memory Swapped-in pages Swapped-out pages

10. Measuring Swap How much is available/used? –swapon -s (cat /proc/swaps) –vmstat so: Amount of memory KB/s SWAP OUT to disk si: Amount of memory KB/s SWAP IN from disk swapd : Amount of virtual memory KB reserved Create/Add more? –# mkswap /dev/sdc3 –fstab: /dev/sdc3 swap swap pri=42 0 0

11. Memory Usage Kernel (not paged) Shared memory Process memory Page cache Buffer cache Kernel modules Dynamically adjusted Paged Paged/locked Not paged

12. Measure memory utilization and paging. Identify the significant memory statistics. –cat /proc/meminfo (units KB) MemTotal : Total physical memory MemFree : Total free memory LowFree : Free memory below 1GB physical Buffers : Linux Buffer Cache Cached : Linux Page Cache BigFree : Free bigpages memory Measuring Memory Usage Page out Page in

13. No Free Memory? free command reports a very low number, should I be worried –It ’ s OK if Linux buffer or page cache are big –Look in -/+ buffer/cache row of free for projected free memory –Very useful stat

14. Memory - sar -B paging statistics -R memory statistics -W swapping statistics #sar -B #sar -R

15. Acegene IT Co. Ltd. 15 Measuring Disk I/O You should monitor the I/O across all devices and look for ‘ slow ’ devices using: –sar, iostat, vmstat –Archive collected statistics for comparison

16. Acegene IT Co. Ltd. 16 Network Traffic Check the following sources of network traffic: –Oracle SQL*Net Servicing SQL connections –Interconnect in RAC environment –Network file system –Samba –HTTP server Start with sar -n DEV iptraf also very useful

17. sar Flags -f read from sar historical file –/var/log/sa/ … -s -e start/end times –-s hh:mm:ss -r memory/swap utilization -c process creation activity -w context switching activity

18. Acegene IT Co. Ltd. 18 Process Specific Specific process is a suspect: –System call trace: strace –p –Library call trace: ltrace –p –Detailed process statistics: ps –o Try: ps -e -o pid,ppid,pcpu,rss,vsz,pri,wchan,cmd Who has my file open? –lsof [-p <pid] For Process Tree, use pstree –p Not seeing a process, it ’ s probably a thread, try: ps -efm

19. Acegene IT Co. Ltd. 19 What Does the Kernel Do? It ’ s possible to profile the kernel and identify where it ’ s spending time: –Boot with “profile=2 nmi_watchdog=1” Either in Lilo or Grub or add manually at boottime –Use readprofile -m –Start with readprofile -r (to reset counters) –Very easy to use, but useful only if you see high system time. To profile user applications use oprofile –http://oprofile.sourceforge.net/

20. Acegene IT Co. Ltd. 20 System Wide RDA (Remote Diagnostic Agent) –http://set.oraclecorp.com/tools/dca/ind ex.html Lshw (Hardware Lister) –http://freshmeat.net/projects/lshw/

21. Acegene IT Co. Ltd. 21 Summary: Linux Monitoring Tools Overall tools –sar, vmstat CPU –/proc/cpuinfo, mpstat, top Memory –/proc/meminfo, /proc/slabinfo Disk I/O –iostat, sar Network –iptraf, netstat, mii-tool Individual process debugging –strace, ltrace, lsof

22. A Q & Q U E S T I O N S A N S W E R S

23. Acegene IT Co. Ltd. 23 练习 执行下列命令 –sar, vmstat –/proc/cpuinfo, mpstat, top –/proc/meminfo, /proc/slabinfo –iostat, sar –iptraf, netstat, mii-tool –strace, ltrace, lsof