Towards Scalable Performance Analysis and Visualization through Data Reduction Chee Wai Lee, Celso Mendes, L. V. Kale University of Illinois at Urbana-Champaign

Motivation  Event trace-based performance tools help applications scale well.  As applications scale, so must performance tools. Why?

Nature of Event Traces  Tend to be thread or processor-centric.  Volume of data per thread proportional to number of performance events encountered.  Number of performance events per thread depends on duration of run and frequency of events.  Strong Scaling : More threads, more communication events.  Weak Scaling : More threads, more communication events, more work per thread.  More events = more work for Performance Tools.

Reducing the data: Part 1  Baseline: Record events of the entire run.  What are simple ways of reducing the volume of performance data? Cut inconsequential event-blocks (e.g. initialization/end)Keep important snapshots (e.g. important iteration blocks) NAMDStartup First 300 steps with Load Balancing Steps with a load refinement

Quantifying the Problem 92k Atoms327k Atoms1000k Atoms 512 cores827 MB1,800 MB2,800 MB 1024 cores938 MB2,200 MB3,900 MB 2048 cores1,200 MB2,800 MB4,800 MB 4096 cores5,700 MB NAMD molecular dynamics simulations and event trace volume as generated by Projections performance tool over 200 (“interesting”) time steps. Weak ScalingStrong Scaling

Reducing the data: Part 2  Drop “uninteresting” or some specific classes of events.  Compress and/or characterize event patterns. Our Approach: Drop “uninteresting” processors (Threads)

Our Approach  Choose a subset of processors:  Representatives  Outliers  Employ k -Means Clustering for Equivalence- Class discovery.  Chosen processors’ performance data are written to disk at end of run. Which? Why?How?

Equivalence Class Discovery Metric Y Metric X Euclidean Distance Outliers Representatives

Things to Consider  Distance measures may require normalization.  Whether certain metrics are strongly correlated to one another.  Number of initial seeds.  Placement of initial seeds.  Number of representatives chosen.  Number of outliers chosen.

Experimental Methodology  NAMD (NAnoscale Molecular Dynamics) task grain- size performance problem (2002).  Roll-back a performance improvement we made in 2002 to address this problem. Tuned NAMDProblem Injected

Experimental Methodology (2)  1 million atom simulation of the Satellite Tabacco Mosaic Virus.  512 processors to 4096 processors on PSC’s Bigben Cray XT3 supercomputer.  Two criteria for validation:  Amount of data reduced.  Quality of the reduced dataset.

Histogram Quality Measure Bar i orig …… Bar i r educed …… Original Data: 1000 pe Reduced Data: 100 pe HoiHoi HriHri How close is H r i /H o i to on average?

Results: Data Reduction Orig # CoresOriginal DataReduced # CoresReduced Dataset 5122,800 MB51275 MB 10243,900 MB MB 20484,800 MB MB 40965,700 MB MB

Results: Data Reduction

Results: Quality PoPo PrPr P r /P o Average HStd Dev

Conclusion  Approach offers a potential way of controlling volume of performance data generated.  Heuristics have been reasonably good at capturing performance characteristics of the NAMD grain-size problem.

Future Work  Conduct experiments on more problem types and classes for verification.  Find better (more practical) ways for equivalence class discovery.