Metrics and Techniques for Evaluating the Performability of Internet Services Pete Broadwell

Outline 1.Introduction to performability 2.Performability metrics for Internet services Throughput-based metrics (Rutgers) Latency-based metrics (ROC) 3.Analysis and future directions

Goal of ROC project: develop metrics to evaluate new recovery techniques Problem: concept of availability assumes system is either “up” or “down” at a given time Availability doesn’t capture system’s capacity to support degraded service –degraded performance during failures –reduced data quality during high load Motivation

What is “performability”? Combination of performance and dependability measures Classical defn: probabilistic (model- based) measure of a system’s “ability to perform” in the presence of faults 1 –Concept from traditional fault-tolerant systems community, ca –Has since been applied to other areas, but still not in widespread use 1 J. F. Meyer, Performability Evaluation: Where It Is and What Lies Ahead, 1994

Performability Example Discrete-time Markov chain (DTMC) model of a RAID-5 disk array 1 1 Hannu H. Kari, Ph.D. Thesis, Helsinki University of Technology, 1997 p i (t) = probability that system is in state i at time t = failure rate of a single disk drive D = number of data disks  = disk repair rate w i (t) = reward (disk I/O operations/sec)

Performability for Online Services: Rutgers Study Rich Martin (UCB alum) et al. wanted to quantify tradeoffs between web server designs, using a single metric for both performance and availability Approach: –Performed fault injection on PRESS, a locality-aware, cluster-based web server –Measured throughput of cluster during simulated faults and normal operation

REPAIR (human operator) DETECT Degraded Service During a PRESS Component Fault FAILURE STABILIZE RECOVER RESET (optional) Time Throughput Requests/sec

Calculation of Average Throughput, Given Faults Throughput Time Degraded throughput Requests/sec Average throughput Normal throughput

Behavior of a Performability Metric Effect of improving degraded performance PerformabilityPerformance during faults

Behavior of a Performability Metric Effect of improving component availability (shorter MTTR, longer MTTF) MTTRMTTF Performability Aavailability = MTTF MTTF + MTTR

Behavior of a Performability Metric Effect of improving overall performance PerformabilityOverall performance (includes normal operation) Most performability metrics scale linearly as component availability, degraded performance and overall performance increase

Results of Rutgers Study: Design Comparisons

An Alternative Metric: Response Latency Originally, performability metrics were meant to capture end-user experience 1 Latency better describes the experience of an end user of a web site –response time >8 sec = site abandonment = lost income $$ 2 Throughput describes the raw processing ability of a service –best used to quantify expenses 1 J. F. Meyer, Performability Evaluation: Where It Is and What Lies Ahead, Zona Research and Keynote Systems, The Need for Speed II, 2001

Effect of Component Failure on Response Latency Time Response latency (sec) REPAIR 8s Abandonment region FAILURE Annoyance region?

Issues With Latency As a Performability Metric Modeling concerns: –Human element: retries and abandonment –Queuing issues: buffering and timeouts –Unavailability of load balancer due to faults –Burstiness of workload Latency is more accurately modeled at service, rather than end-to-end 1 Alternate approach: evaluate an existing system 1 M. Merzbacher and D. Patterson, Measuring End-User Availability on the Web: Practical Experience, 2002

Analysis Queuing behavior may have a significant effect on latency-based performability evaluation –Long component MTTRs = longer waits, lower latency-based score –High performance in normal case = faster queue reduction after repair, higher latency-based score More study is needed!

Future Work Further collaboration with Rutgers on collecting new measurements for latency-based performability analysis Development of more realistic fault and workload models, other performability factors such as data quality Research into methods for conducting automated performability evaluations of web services

Metrics and Techniques for Evaluating the Performability of Internet Services Pete Broadwell

Back-of-the-Envelope Latency Calculations Attempted to infer average request latency for PRESS servers from Rutgers data set –Required many simplifying assumptions, relying upon knowledge of PRESS server design –Hoped to expose areas in which throughput- and latency-based performability evaluations differ Assumptions: –FIFO queuing w/no timeouts, overflows –Independent faults, constant workload (also the case for throughput-based model) Current models do not capture “completeness” of data returned to user

Comparison of Performability Metrics

Rutgers calculations for long-term performability Goal: metric that scales linearly with both - performance (throughput) and - availability [MTTF / (MTTF + MTTR)] T n = normal throughput for server A I = ideal availability (.99999) Average throughput (AT) = T n during normal operation + per- component throughput during failure Average availability (AA) = AT / T n Performability = T n x [log(A I ) / log(AA)]

Results of Rutgers study: performance comparison

Results of Rutgers study: availability comparison

Results of Rutgers study: performability comparison