Memory Performance Profiling via Sampled Performance Monitor Event Traces Diana Villa, Patricia J. Teller, and Jaime Acosta The University of Texas at El Paso Department of Computer Science Trevor Morgan Exxon/Mobil Bret Olszewski IBM Corporation-Austin 5 th Annual IBM Austin CAS Conference – 20 February 2004

Outline Motivation Data  Events Profiled  Information Collected Analysis  Approach  Performance Evaluation Framework Results Conclusions and Future Work

5 th Annual IBM Austin CAS Conference – 20 February 2004 Motivation Overall research goal General workload characterization model Project goal  Develop a performance evaluation framework to facilitate analysis of large sampled event traces  Study load access patterns of key applications  Identify and remedy performance impediments

5 th Annual IBM Austin CAS Conference – 20 February 2004 Data Collection Environment IBM eserver p-Series 690 architecture 8- and 32-processor configurations TPC-C benchmark Data collected via event trace sampling: Timestamp Effective instruction and data addresses CPU id Process id Thread id

5 th Annual IBM Austin CAS Conference – 20 February 2004 Platform -1 P X XP XP P X X X X P PP P L2 L3 MCM 0 MCM 1 X 8-processor p690 configuration

5 th Annual IBM Austin CAS Conference – 20 February 2004 Platform - 2 P P PP PP P L2 L3 MCM 0 P P P PP PP P L2 L3 MCM 2 P P P PP PP P L2 L3 MCM 1 P P P PP PP P L2 L3 MCM 3 P 32-processor p690 configuration

5 th Annual IBM Austin CAS Conference – 20 February 2004 Events Resolution of L2-cache data-load misses  L2.5 L2.5 shared L2.5 modified  L2.75 L2.75 shared L2.75 modified  L3  L3.5

5 th Annual IBM Austin CAS Conference – 20 February 2004 L2.5 P X XP XP P X X X X P PP P L2 L3 Penalty: 73 cycles MCM 0 MCM 1 X

5 th Annual IBM Austin CAS Conference – 20 February 2004 L2.75 P X XP XP P X X X X P PP P L2 L3 Penalty: 96 cycles MCM 0 MCM 1 X

5 th Annual IBM Austin CAS Conference – 20 February 2004 L3 P X XP XP P X X X X P PP P L2 L3 Penalty: 112 cycles MCM 0 MCM 1 X

5 th Annual IBM Austin CAS Conference – 20 February 2004 L3.5 P X XP XP P X X X X P PP P L2 L3 Penalty: 143 cycles MCM 0 MCM 1 X

5 th Annual IBM Austin CAS Conference – 20 February 2004 Analysis Identify application-specific sources of performance degradation associated with data references Level of Memory Hierarchy kernel …. text buffer pool data,bss,heap …. Address Space Segment Page Page Offset/ Cache line

5 th Annual IBM Austin CAS Conference – 20 February 2004 Performance Evaluation Framework Database Load DB Java Tool Report Generation Java Tool p690TPC-C Data Collection Environment Reports 5 BufferPool Data,BSS,Heap Kernel Graphs Sampled Event Traces PID TID Timestamp Instr.Addr. DataAddr.

5 th Annual IBM Austin CAS Conference – 20 February 2004 Results

5 th Annual IBM Austin CAS Conference – 20 February 2004 Results - Memory Regions

5 th Annual IBM Austin CAS Conference – 20 February 2004 Results - L3 Cache

5 th Annual IBM Austin CAS Conference – 20 February 2004 Results - Segment

5 th Annual IBM Austin CAS Conference – 20 February 2004 Results - Pages

5 th Annual IBM Austin CAS Conference – 20 February 2004 Results – Cache Lines

5 th Annual IBM Austin CAS Conference – 20 February 2004 Results - Instructions Lock OperationsAtomic Operations simple_lockfetch_and_add simple_lock_ppcfetch_and_add_h simple_unlockfetch_and_addlp disable_lockfetch_and_or unlock_enablefetch_and_orlp simple_unlock_memfetch_and_and unlock_enable_memfetch_and_andlp

5 th Annual IBM Austin CAS Conference – 20 February 2004 Targets for performance improvement of TPC-C are associated mainly with two regions of the address space:  buffer pool  data, bss, heap TPC-C lock instructions are not key to performance degradation 8- and 32-processor data have same reference pattern, thus, a model of TPC-C memory access may be possible Conclusions

5 th Annual IBM Austin CAS Conference – 20 February 2004 Suggest ways to improve performance of applications executed on p690 Enhance performance evaluation framework Quantify representativeness of sampled event traces Expand study of application data load behavior  Process characterization  Process migration  Other performance issues Compulsory vs. capacity/conflict misses False sharing Contention for resources Develop synthetic applications that mimic the behavior of key p690 applications; use these to study application behavior and experiment with modifications to applications that may affect performance Future Work

5 th Annual IBM Austin CAS Conference – 20 February 2004 Questions?