1. Título/Title Nome/Name Cargo/Position Foto/ Picture Linux Performance on Power Breno Leitão Software Engineer

2. 2 Performance concepts. What is performance? –Low Latency –High volume –Low CPU Usage Where can you change? –Change the application –Change the environment –Find where in the hardware is the bottleneck. –At what price?

3. 3 Where to tune? Hardware options Virtual machine parameters Kernel parameters Memory parameters TCP parameters Virtual Database parameters JVM parameters Not an easy and quick process

4. 4 Step by Step 1) Reproduce the problem 1) Find a problem subset that quickly reproduces the problem. 2) Profile and collect information. 1) Creates a script that do run and collect information automatically 3) Analyze the problem, and find the possible points that are affecting the performance. 1) Narrow down 4) One change at time. Brute force is not the right way

5. 5 Linux Performance Linux on Power == Linux

6. 6 Linux Performance Tools and Profilers: –Iostat –Sar –Perf –SystemTap –Oprofile –Linux Performance Customer Profiler Utility (lpcpu)

7. 7 Non Uniform Memory Access Most of the time, servers are NUMA

8. 8 General Linux tips Tune the application to the NUMA environment. Use a recent and supported Linux version Application and IRQ Binding is good. – CPU Migrations is bad. – Taskset and /proc/ /smp_affinity Being able to compile the source code is a good advantage. Disable unused services and modules CPUBinaryHEX 000010x1 100100x2 201000x4 310000x8

9. 9 Compiling tips Different Compilers: - GCC - Advanced Toolchain - IBM XL Compiler Optimization: - Use the latest architecture you are going to use the hardware. (- mcpu=power7 ) - Different level of optimizations (O1, O2, O3) - Hundreds of options (http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html)

10. 10 Other optimizations Libraries IBM MASS (Mathematical Accelerator Subsystem) IBM ESSL (Engineering Scientific Subsystem Library) Link-Time optimization (LTO) Post Time Optimization FDPR (Feedback-Directed Program Re-structuring) IBM SDK for PowerLinux

11. 11 Network Optimizations Memory buffers net.ipv4.tcp_rmem net.ipv4.tcp_wmem TCP Windows Scaling net.ipv4.tcp_window_scaling Congestion Control CUBIC, Vegas, Reno, BIC, Hybla, Tahoe, etc NIC Tunning Queue size, MTU, Offloads (CRC, GSO, LRO)

12. 12 Memory Optimization Huge Pages Libhugepages Memory allocation Optimizations MicroQuill's Smartheap TCMalloc (gperftools) Linux uses all the memory, with cache/buffers, and this is a good thing.

13. 13 Storage Optimization Technologies ISCSI FCoE SAS SSD File System Ext, GPFS, etc Journaling Access time (noatime)

14. 14 IO Scheduler Orders block I/O operations will be submitted to storage volumes. Minimize Hard-disk seeks Prioritize I/O processes Guarantee that a certain request will be issue before a deadline. Schedulers NOOP Anticipatory Deadline

15. 15 Java Optimization Memory (heap, garbage collection) (-Xms, -Xmx, -Xmn*) Compressed 64 bits references (-Xcompressedrefs) Garbage Collection (- Xgcpolicy:{optthruput,subpool,optavgpause,gencon*,bala nced,..}) Garbage Collection Threads -Xgcthreads JIT (Just-in-time) Compiler New technologies available on Linux -Xaggressive IBM HealthCenter profiler

16. 16 Perf Profiling tool that enables you to discover about: Code profiling What code takes longer? Difference between two runs Performance counters: Hardware and Software events Cache (Hit, Miss), TLB, Cycles, Instructions A lot more.

17. 17 Perf timechart

18. 18 Perf demo # perf stat # perf list # perf record # perf diff

19. 19 References DeveloperWorks Wiki http://www.ibm.com/developerworks/wikis/display/LinuxP/Performance+Tuning Perf Introduction http://www.ibm.com/developerworks/br/local/linux/perf_rev5_introduction/index. html SystemTap Introduction http://www.ibm.com/developerworks/br/local/linux/l-systemtap-1/index.html Tuning 10Gb/s Network Cards http://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf IBM PowerLinux SDK https://www-304.ibm.com/webapp/set2/sas/f/lopdiags/sdklop.html Linux Performance and Tuning Guidelines http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/redp4285.html

20. 20