Dror Feitelson Hebrew University The Forgotten Factor: FACTS on Performance Evaluation and its Dependence on Workloads Dror Feitelson Hebrew University Thanks toparticipants and progam committee; thanks to Monien; abuse hospitality – talk about agenda

Performance Evaluation In system design Selection of algorithms Setting parameter values In procurement decisions Value for money Meet usage goals For capacity planing Important and basic activity

The Good Old Days… The skies were blue The simulation results were conclusive Our scheme was better than theirs Focus on system design. Widely different designs lead to conclusive results. Feitelson & Jette, JSSPP 1997

Their scheme was better than ours! But in their papers, Their scheme was better than ours! But literature is full of contradictory results.

How could they be so wrong? Leads to question of what is the cause for contradictions.

Performance evaluation depends on: The system’s design (What we teach in algorithms and data structures) Its implementation (What we teach in programming courses) The workload to which it is subjected The metric used in the evaluation Interactions between these factors Next: our focus is the workloads.

Performance evaluation depends on: The system’s design (What we teach in algorithms and data structures) Its implementation (What we teach in programming courses) The workload to which it is subjected The metric used in the evaluation Interactions between these factors

Outline for Today Three examples of how workloads affect performance evaluation Workload modeling Research agenda In the context of parallel job scheduling Job scheduling, not task scheduling

Example #1 Gang Scheduling and Job Size Distribution

Gang What?!? Time slicing parallel jobs with coordinated context switching Ousterhout matrix Ousterhout, ICDCS 1982

Gang What?!? Time slicing parallel jobs with coordinated context switching Ousterhout matrix Optimization: Alternative scheduling Ousterhout, ICDCS 1982

Packing Jobs Use a buddy system for allocating processors Feitelson & Rudolph, Computer 1990

Packing Jobs Use a buddy system for allocating processors Start with full system in one block

Packing Jobs Use a buddy system for allocating processors To allocate repeatedly partition in two to get desired size

Packing Jobs Use a buddy system for allocating processors

Packing Jobs Use a buddy system for allocating processors Or use existing partition

The Question: The buddy system leads to internal fragmentation But it also improves the chances of alternative scheduling, because processors are allocated in predefined groups Which effect dominates the other?

The Answer (part 1): Feitelson & Rudolph, JPDC 1996 Answer as function of workload, but not full answer because workload unknown. Dashed lines: provable bounds. Feitelson & Rudolph, JPDC 1996

The Answer (part 2): Note logarithmic Y axis

The Answer (part 2):

The Answer (part 2):

The Answer (part 2): Many small jobs Many sequential jobs Many power of two jobs Practically no jobs use full machine Conclusion: buddy system should work well

Verification Using Feitelson workload Feitelson, JSSPP 1996

Parallel Job Scheduling Example #2 Parallel Job Scheduling and Job Scaling

Variable Partitioning Each job gets a dedicated partition for the duration of its execution Resembles 2D bin packing Packing large jobs first should lead to better performance But what about correlation of size and runtime? First-fit decreasing is optimal

Scaling Models Constant work Constant time Memory bound Parallelism for speedup: Amdahl’s Law Large first  SJF Constant time Size and runtime are uncorrelated Memory bound Large first  LJF Full-size jobs lead to blockout Question is which model applies within the context of a single machine Worley, SIAM JSSC 1990

“Scan” Algorithm Keep jobs in separate queues according to size (sizes are powers of 2) Serve the queues Round Robin, scheduling all jobs from each queue (they pack perfectly) Assuming constant work model, large jobs only block the machine for a short time But the memory bound model would lead to excessive queueing of small jobs Important point: schedule order determined by size Krueger et al., IEEE TPDS 1994

The Data Data: SDSC Paragon, 1995/6

The Data Data: SDSC Paragon, 1995/6 Partitions with equal numbers of jobs; many more small jobs. Data: SDSC Paragon, 1995/6

The Data Data: SDSC Paragon, 1995/6 Similar range, different shape; 80th percentile moves from <1m to several h. Data: SDSC Paragon, 1995/6

Conclusion Parallelism used for better results, not for faster results Constant work model is unrealistic Memory bound model is reasonable Scan algorithm will probably not perform well in practice

User Runtime Estimation Example #3 Backfilling and User Runtime Estimation

Backfilling Variable partitioning can suffer from external fragmentation Backfilling optimization: move jobs forward to fill in holes in the schedule Requires knowledge of expected job runtimes

Variants EASY backfilling Make reservation for first queued job Conservative backfilling Make reservation for all queued jobs

User Runtime Estimates Lower estimates improve chance of backfilling and better response time Too low estimates run the risk of having the job killed So estimates should be accurate, right?

They Aren’t Mu’alem & Feitelson, IEEE TPDS 2001 Short=failed; killed typically exceeded runtime estimate, ~15% Mu’alem & Feitelson, IEEE TPDS 2001

Surprising Consequences Inaccurate estimates actually lead to improved performance Performance evaluation results may depend on the accuracy of runtime estimates Example: EASY vs. conservative Using different workloads And different metrics Will focus on second bullet

EASY vs. Conservative Using CTC SP2 workload

EASY vs. Conservative Using Jann workload model Note: jann model of CTC

EASY vs. Conservative Using Feitelson workload model

Conflicting Results Explained Jann uses accurate runtime estimates This leads to a tighter schedule EASY is not affected too much Conservative manages less backfilling of long jobs, because respects more reservations Relative measure: more by EASY = less by conservative

Conservative is bad for the long jobs Good for short ones that are respected Conservative EASY

Conflicting Results Explained Response time sensitive to long jobs, which favor EASY Slowdown sensitive to short jobs, which favor conservative All this does not happen at CTC, because estimates are so loose that backfill can occur even under conservative

Verification Run CTC workload with accurate estimates

But What About My Model? Simply does not have such small long jobs

Workload Modeling

No Data Innovative unprecedented systems Use an educated guess Wireless Hand-held Use an educated guess Self similarity Heavy tails Zipf distribution

Serendipitous Data Data may be collected for various reasons Accounting logs Audit logs Debugging logs Just-so logs Can lead to wealth of information

NASA Ames iPSC/860 log 42050 jobs from Oct-Dec 1993 user job nodes runtime date time user4 cmd8 32 70 11/10/93 10:13:17 user4 cmd8 32 70 11/10/93 10:19:30 user42 nqs450 32 3300 11/10/93 10:22:07 user41 cmd342 4 54 11/10/93 10:22:37 sysadmin pwd 1 6 11/10/93 10:22:42 user4 cmd8 32 60 11/10/93 10:25:42 sysadmin pwd 1 3 11/10/93 10:30:43 user41 cmd342 4 126 11/10/93 10:31:32 Feitelson & Nitzberg, JSSPP 1995

Distribution of Job Sizes

Distribution of Job Sizes

Distribution of Resource Use

Distribution of Resource Use

Degree of Multiprogramming

System Utilization

Job Arrivals

Arriving Job Sizes

Distribution of Interarrival Times

Distribution of Runtimes

Job Scaling

User Activity

Repeated Execution

Application Moldability Of jobs run more than once

Distribution of Run Lengths

Predictability in Repeated Runs For jobs run more than 5 times

Recurring Findings Many small and serial jobs Many power-of-two jobs Weak correlation of job size and duration Job runtimes are bounded but have CV>1 Inaccurate user runtime estimates Non-stationary arrivals (daily/weekly cycle) Power-law user activity, run lengths

Research Agenda

The Needs New systems tend to be more complex Differences tend to be finer Evaluations require more detailed data Getting more data requires more work Important areas: Internal structure of applications User behavior

Generic Application Model Iterations of Compute granularity Memory working set / locality I/O Interprocess locality Communicate Pattern, volume Option of phases with different patterns of iterations compute I/O communicate At least for SPMD; not for pipeline

Consequences Model the interaction of the application with the system Support for communication pattern Availability of memory Application attributes depend on system Effect of multi-resource schedulers Common occurrence in benchmarking that different benchmarks rank systems differently

Missing Data There has been some work on the characterization of specific applications There has been no work on the distribution of application types in a complete workload Distribution of granularities Distribution of working set sizes Distribution of communication patterns

Effect of Users Workload is generated by users Human users do not behave like a random sampling process Feedback based on system performance Repetitive working patterns

Feedback User population is finite Users back off when performance is inadequate Negative feedback Better system stability Need to explicitly model this behavior

Locality of Sampling Users display different levels of activity at different times At any given time, only a small subset of users is active These users repeatedly do the same thing Workload observed by system is not a random sample from long-term distribution Can this be exploited by a scheduler?

Final Words…

We like to think that we design systems based on solid foundations…

But beware: the foundations might be unbased assumptions!

Computer Systems are Complex We should have more “science” in computer science: Collect data rather than make assumptions Run experiments under different conditions Make measurements and observations Make predictions and verify them Science = experimental scince, like physics, chemistry, biology

Advice from the Experts “Science if built of facts as a house if built of stones. But a collection of facts is no more a science than a heap of stones is a house” -- Henri Poincaré

Advice from the Experts “Science if built of facts as a house if built of stones. But a collection of facts is no more a science than a heap of stones is a house” -- Henri Poincaré “Everything should be made as simple as possible, but not simpler” -- Albert Einstein

Acknowledgements Students: Ahuva Mu’alem, David Talby, Uri Lublin Larry Rudolph / MIT Data in Parallel Workloads Archive Joefon Jann / IBM CTC SP2 log SDSC Paragon log SDSC SP2 log NASA iPSC/860 log