1. CIT 470: Advanced Network and System Administration
Performance Monitoring
CIT 470: Advanced Network and System Administration

2. CIT 470: Advanced Network and System Administration
Topics
Performance testing
Performance tuning.
CPU
Memory
Disk
Network
CIT 470: Advanced Network and System Administration

3. What is performance testing?
Performance testing is a type of testing intended to determine the responsiveness, throughput, reliability, and/or scalability of a system under a given workload. -
Performance testing goals:
Assess production readiness
Evaluate against performance criteria
Compare performance characteristics of multiple systems or system configurations
Find the source of performance problems
Support system tuning
Find throughput levels

4. Performance Testing Activities

5. Testing Types
Performance testing: determining performance, scalability, or stability characteristics of system; a superset of the other testing types.
Load testing: determining performance characteristics of system when subjected to work load expected during production.
Stress testing: determining performance characteristics of system when subjected to work loads beyond those expected during production to determine under what conditions system will fail.

6. Baselines
A baseline is a set of data used for comparison. In performance testing, baselines are used to evaluate the effectiveness of subsequent performance-improving changes to the system. Once the system has been changed, a new baseline must be measured.

7. Benchmarking
Benchmarking is the process of measuring system performance using standard tests and comparing it against a well known system.
SPEC CPU2006 (SPECint, SPECfp)
SPEC power2008 (power usage)
SPEC sfs2008 (NFS, CIFS)
SPEC virt2010 (virtualization)
SPEC web2005 (PHP or JSP)
BogoMips
Dhrystone
Whetstone
Weighted TeraFLOPS
NAS Parallel Benchmarks

8. CIT 470: Advanced Network and System Administration
Identify Bottlenecks
Identify which aspect of performance
Latency: delay until initial access.
Throughput: rate of transfer/processing.
Identify which system component
CPU
Memory
Disk
Network
CIT 470: Advanced Network and System Administration

9. Performance Tuning Process
Learn the customer’s problem.
Identify specifically what’s wrong.
Find the problem’s cause and fix it.
When does the problem occur?
Has anything about the system changed?
What critical resource is affecting performance?
Have the right tools.
Historical monitoring data will show what’s normal and identify any trends.
CIT 470: Advanced Network and System Administration

10. CIT 470: Advanced Network and System Administration
Experimenter Effect
Monitoring the system affects performance.
Monitoring tools use system resources.
If you’ve consistently monitored system, then monitoring won’t alter system performance.
CIT 470: Advanced Network and System Administration

11. Performance Problem Solutions
Get more of needed resource.
Ex: Upgrade processor, use striped disk array.
Reduce system requirements.
Ex: Kill processes, move services to other hosts.
Eliminate inefficiency and waste.
Ex: Produce a static home page every 15 minutes instead of regenerating each access.
Ration resource usage.
Ex: Set process priorities with renice.
Ex: Limit process resource usage with limit.
CIT 470: Advanced Network and System Administration

12. CIT 470: Advanced Network and System Administration
Monitoring Processes
uptime
Provides aggregate data about system load. ps
Shows running processes with CPU, mem usage.
top
Updated list of running processes + summaries.
vmstat
Summary data about processes and CPU usage.
CIT 470: Advanced Network and System Administration

13. CIT 470: Advanced Network and System Administration
Uptime
Uptime provides the following data
How long system has been running.
Number of users logged in.
Average number of runnable processes.
In last 1, 5, 15 minutes.
Want a load average under 3.
Uptime example
> uptime
17:40 up 126 days, 8:03, 6 users, load average: 1.40, 1.03, 0.55
CIT 470: Advanced Network and System Administration

14. CIT 470: Advanced Network and System Administration
vmstat
Number of Runnable and Blocked processes.
Memory (virtual, free, buffered, cached)
Blocks/second transferred in (bi) and out (bo)
Interrupts/sec (in) and context switches/sec (cs)
CPU usage by user, system, idle, and waiting.
> vmstat 5 4
procs memory swap io system cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
CIT 470: Advanced Network and System Administration

15. Identifying CPU Shortages
Short-term CPU spikes are normal.
Consistently high number of runnable processes (r) in vmstat.
Consistent high total CPU usage (sy+us).
High system time compared to user time and high context switches indicates system is thrashing between processes instead of doing user work.
CIT 470: Advanced Network and System Administration

16. Changing Process Priorities
Nice values
Positive values lower priorities.
Negative values increase priorities.
If you know a process will be a CPU hog,
nice +5 command_name
If you detect a CPU hog after it’s started,
renice 5 PID
CIT 470: Advanced Network and System Administration

17. Managing Processes with kill
TERM (default)
Terminates process execution (Ctrl-c).
Processes can catch or ignore signal.
KILL (9)
Terminates process execution.
Processes cannot catch or ignore.
Processes waiting on I/O will not die.
STOP
Suspends process execution until SIGCONT (Ctrl-z).
Useful for moving CPU hog out of way temporarily.
CIT 470: Advanced Network and System Administration

18. Imposing Limits on Processes
CPU time
ulimit –t secs
Maximum file size
ulimit –f KB
Maximum data segment
ulimit –d KB
Maximum stack size
ulimit –s KB
Maximum physical mem
ulimit –m KB
Maximum core size
ulimit –c KB
Maximum number procs
ulimit –u n
Maximum virtual mem
ulimit –v KB
CIT 470: Advanced Network and System Administration

19. CIT 470: Advanced Network and System Administration
Monitoring Memory
Use free to see how memory is used.
System will use most free memory for caching.
System will swap out inactive processes.
Don’t worry until free < 5% of total memory.
Use vmstat to detect paging activity.
Page out (so) rate greater than 0 consistently.
High page in (si) rate, as system uses the paging facility to load programs into memory.
CIT 470: Advanced Network and System Administration

20. CIT 470: Advanced Network and System Administration
Managing Memory
Improving paging capacity.
Add new swapfiles with swapon.
Add new swap partitions.
Improving paging performance.
Use swap partitions instead of swap files.
Distribute swap resources across disks.
Migrate memory hogs to another host.
Add more memory.
CIT 470: Advanced Network and System Administration

21. CIT 470: Advanced Network and System Administration
Monitoring Disk I/O
Use iostat to get per disk statistics.
Transactions per second (tps).
Blocks read/written per second.
Managing disk performance problems.
Distribute heavily used data across disks/ctrlers.
Get more or faster disks.
Use RAID or LVM striping.
CIT 470: Advanced Network and System Administration

22. CIT 470: Advanced Network and System Administration
iostat
> iostat 2
Linux (zim) /26/2007
avg-cpu: %user %nice %system %iowait %steal %idle
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
hde
hdh
hdc
hde
hdh
hdc
CIT 470: Advanced Network and System Administration

23. Managing Disk Capacity
Detecting disk resource usage.
List all partition usage with df –h
Identify high usage directories with du
Summary data: du –s
Highest usage directories: du -k /|sort –rn
Use find to detect disk hogs.
Use find –size to search for big files.
Use –atime +X to identify files that haven’t been used in X days.
CIT 470: Advanced Network and System Administration

24. Managing Disk Shortages
Add more disks.
Move files to remote fileservers.
Eliminate unnecessary files.
Compress large infrequently used files.
Impose disk quotas on users.
Soft limit: can be violated temporarily.
Hard limit: cannot be violated.
CIT 470: Advanced Network and System Administration

25. CIT 470: Advanced Network and System Administration
IPTraf
CIT 470: Advanced Network and System Administration

26. CIT 470: Advanced Network and System Administration
iftop
CIT 470: Advanced Network and System Administration

27. References
Mark Burgess, Principles of System and Network Administration, Wiley, 2000.
Aeleen Frisch, Essential System Administration, 3rd edition, O’Reilly, 2002.
Mike Loukides and Gian-Paolo D. Musumeci, System Performance Tuning, 2nd edition, O’Reilly, 2003.
Evi Nemeth et al, UNIX System Administration Handbook, 3rd edition, Prentice Hall, 2001.
patterns & practices, Performance Testing Guidance for Web Applications,