Fault Tolerance Benchmarking

2 Owerview What is Benchmarking? What is Dependability? What is Dependability Benchmarking? What is the relation between Dependability & Fault tolerance Benchmarking? Tsai (A Fault tolerance Benchmark) / 20

3 What is Benchmarking? Benchmarking is testing of several computer systems or peripheral devices with the aim to compare their performance-cost relationship. Benchmarking means measuring something In an automated, reproducible, portable, and truthful way, and this can only probably be achieved by relying on direct results. (not by model analysis). / 20

4 What is a Benchmark ? A specification of all elements necessary for performing the benchmark tests. What elements? See later! / 20

5 Benchmarking Context Definition of benchmark elements depends on:  the system under benchmark  the benchmark scope (internal/external)  the application area  the phase of system life cycle in mind  design  Implementation  integration  test  production  maintenance / 20

6 Dependability is the trustworthiness of a computer system such that reliance can justifiably be placed on the service it delivers. Dependability is normally described by a set of dependability attributes. What is Dependability? / 20

7 Dependability Attributes  Reliability Measure of continuous correct service delivery(dependability with respect to continuity of service).  Availability Measure of correct service delivery with respect to the alternation of correct and incorrect service (dependability with respect to readiness for usage).  Safety Measure of continuous delivery of either correct service or incorrect service after benign failure (dependability with respect to the non- occurrence of catastrophic failures).  Security Dependability with respect to the prevention of unauthorized access and/or handling of information.  Robustness The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environment conditions. / 20

8 Why Benchmark Availability?  System availability is a pressing problem –modern applications demand near-100% availability  e-commerce, enterprise apps, online services, ISPs at all scales and price points –we don’t know how to build highly-available systems!  except at the very high-end  Few tools exist to provide insight into system availability –most existing benchmarks ignore availability  focus on performance, and under ideal conditions –no comprehensive, well-defined metrics for availability / 20

9 Availability metrics  Traditionally, percentage of time system is up –time-averaged, binary view of system state (up/down)  This metric is inflexible –doesn’t capture degraded states  a non-binary spectrum between “up” and “down”  time-averaging discards important temporal behavior –compare 2 systems with 96.7% traditional availability:  system A is down for 2 seconds per minute  system B is down for 1 day per month / 20

10 Dependability Attributes How do we achiev these attributes? –start by understanding them –figure out how to measure them –evaluate existing systems and techniques –develop new approaches based on what we’ve learned –and measure them as well! Dependability Benchmarks make these tasks possible! / 20

11 What is Dependability Benchmarking? Dependability benchmarking is performing a set of tests to quantify computer dependability. This quantification is supported by the evaluation of dependability attributes (e.g., reliability, availability, safety) through the assessment of direct measures related to the behavior of a computer in the presence of faults. Examples of these direct measures are failure modes, error detection coverage, error latency, diagnosis efficiency, recovery time, and recovery losses. / 20

12 Dependability Benchmarking The goal of dependability benchmarking is to quantify the dependability features of a computer or a computer component in a truthful and reproducible way. Unlike functionality and performance features, that are normally available to the customers or can be certified (functionality) or measured (performance), component or system dependability cannot be easily assessed today. The objective of dependability benchmarking is to change this picture, especially for COTS components and COTS-based systems. / 20

13 What is expected from a Software Dependability Benchmark?  Software characterization with respect to: – Internal faults – External faults  other software component(s)  hardware  Aims – Properties + quantification of some specific measures – Avoid undesirable behavior → “environment” modification – Enhancement  Correction  Wrapping / 20

14 Availability Benchmark / 20

15 Software Testing  Aims – Activate / identify faults correction – Characterize software behavior  Various kinds of tests for characterization – Functional (statistical / operational) testing Validate functionality under ordinary operating conditions (typical load) Software reliability evaluation – Load testing Performance under heavy load (peak, worst case) – Robustness / stress testing Pushes the software beyond its specified limits (invalid inputs, stressful environmental conditions) / 20

16 Software Reliability  Aim – Evaluate software reliability measures (MTTF, failure intensity)  Means – Failure data collection and processing From functional test  Advantage Early estimation  Limitation Representative operational profile Duration accuracy / 20

17 Software Performance Benchmark Most of the time Performance benchmark = system benchmark Aims of software performance benchmark Measure system performance (or price / performance) to compare: –different software products on the same machine Competition between software vendors –different releases/versions of the same product on the same machine Performance improvement? Two categories of benchmarks – Coarse-grain benchmarks => execution times of the entire application – Fine-grain benchmarks => execution time (rate) of specific operations / 20

18 Fault Tolerance & Robustness Benchmarks  Fault tolerance benchmarks – Ability to tolerate faults – Effectiveness of error detection and recovery mechanisms – Performance degradation due to faults  Robustness benchmarks – Ability to tolerate / resist to unexpected conditions caused by:  Hardware failures  Other user programs (system call with illegal parameters) – Failure modes = Robustness test + repeatable  Fault injection / 20

19 The Fault Tolerance Benchmark (Tsai et al., 1996)  This fault tolerance benchmark uses a two-phase procedure: 1) Determine whether the system tolerates the faults that it is intended to tolerate, and evaluate the effect these faults have on the fault tolerance mechanisms. 2) Evaluate the reaction of the system to faults that it is not designed to handle (a demonstration of the degree of fault tolerance beyond what is expected).  For phase 1 injections, three types of measures are obtained: –Error/fault ratio: the ratio of the number of errors detected to the number of faults injected. –Performance degradation: two times are measured:  The time to execute the benchmark with faults.  The time to execute the benchmark without faults. –Number of catastrophic incidents. / 20

20 Conclusion Dependability benchmarks rely on: development / validation process & field information + additional specific work What about COTS? well tested BUT information not available all the time / 20