On Dynamic Resource Availability in Grids A. Iosup, O. Sonmez, D.H.J. Epema M. Jan PDS Group, ST/EWI, TU Delft LRI/INRIA Futurs Paris, INRIA November 24, 2018 IEEE Grid 2007, Austin, TX, USA

Grids are far from being reliable job execution environments Server 99.99999% reliable Small Cluster 99.999% reliable In today’s grids, reliability is more important than performance! 99.9% reliable 5x decrease in failure rate after first year [Sch06] Production Cluster CERN LCG jobs now 25.29% failures that’s after 5 years (3 ~production) Source: dboard-gr.cern.ch, May’07. >10% jobs fail [Ios06] DAS-2 20-45% failures [Kha06] TeraGrid 27% failures, 5-10 retries [Dum05] Grid3 We see each day that a commercially acceptable server needs to be 99.99999% reliable. That’s one error every … hours! As soon as we increase in size, failures become more frequent. A top cluster manufacturer advertises only “three nines” reliability. We are not unhappy with this value. [goal for grids] It is well known that the reliability of a system is lower in the beginning, and towards the end of its life (the famous bath-tub shape). Production clusters have been found to be xx to yy % reliable, after passing the ??-year margin [Sch06]. With Grids, we seem to still be in the beginning: errors occur very frequently. Recent studies have shown that ... [Ios06, Kha06] [Ios06] A. Iosup and D. H. J. Epema. Grenchmark: A framework for analyzing, testing, and comparing grids. In CCGRID, pages 313–320. IEEE Computer Society, 2006. [Kha06] O. Khalili, Jiahue He, et al. Measuring the performance and reliability of production computational grids. In GRID. IEEE Computer Society, 2006. [Dum05] C. Dumitrescu, I. Raicu, and I. T. Foster. Experiences in running workloads over grid3. In H. Zhuge and G. Fox, editors, GCC, volume 3795 of LNCS, pages 274–286, 2005. Large scale clusters – 1 error per day for a few hours (4 hours?) at 1000 resources -> 999 resources + (24-4)/24 resources per day = 999.8 November 24, 2018

Sources of failure in grids and what to do about them Failures [HLi07] Error propagation [Tha02] User applications 1 We don’t know much about our work-horse, the cluster-based grid! Grid + User Middleware 2 Grid middleware tests [Ios07] Machine + OS (resource availability) Desktop grids [Kon04] Cluster-based grids? 3 November 24, 2018

Research Questions Q1: What are the characteristics of the resource (un)availability in grids? Q2: What is the performance impact of the dynamic grid resource availability? November 24, 2018

Outline Introduction and Motivation Resource Availability in Grids Environment and Traces Availability Characteristics Availability Model Performance Impact of Resource Availability Conclusion and Future Work November 24, 2018

2.1. Grid’5000: Environment and Traces Experimental platform Grid’5000: 9 sites, 15 clusters Target: 5000 nodes, >5000 CPUs All clusters managed by OAR Traces: jobs 05/2004-11/2006 (30 mo.) resource availability 05/2005-11/2006 (18 mo.) Jobs: 951 K Availability status change events: 600 K Users: 473, Groups: 10 CPUs: ~ 2500, Consumed CPU time: 651 years November 24, 2018

2.2. Availability Characteristics (1/3) Number of Resources Over Time Average availability Grid-level view Grid-level availability Availability range: 35-92% Average availability: 69% Cluster-level availability Availability range: 32-98% November 24, 2018

2.2. Availability Characteristics (2/3) Mean Time Between Failures (MTBF) Mean Time to Repair (MTTR) Grid-level ~12 minutes Node-level ~2 days MTTR Node-level ~half day CDF MTBF Grid-level view TBF [s] November 24, 2018

2.2. Availability Characteristics (3/3) Correlated Failures Maximal set of failures (ordered according to increasing event time), of time parameter in which for any two successive failures E and F, where returns the timestamp of the event; = 1-3600s. Grid-level view Range: 1-339 Average: 11 Cluster span Range: 1-3 Average: 1.06 Failures “stay” within cluster Average CDF Grid-level view Size of correlated failures November 24, 2018

see article for per-cluster parameter values 2.3. Availability Model MTBF MTTR Correl. Assume no correlation of failure occurrence between clusters Which site/cluster? fs, fraction of failures at cluster s Weibull distribution for IAT Shape parameter > 1: increasing hazard rate the longer a node is online, the higher the chances that it will fail Reboot machines just to keep them in “safe zone”? Software recovers from accumulating problems Hardware may crash during reboot; otherwise won’t help November 24, 2018

Outline Introduction and Motivation Resource Availability in Grids Performance Impact of Resource Availability Models of Availability Information Performance Evaluation Conclusion and Future Work November 24, 2018

3.1. Models of Availability Information 4 types of systems: Resource availability Static Dynamic Availability Information Delay On-Time (0) Short period Long period SA KA AMA HMA Availability Info. Delay Real? 1 SA Steady Availability Static On-Time N 2 KA Known Availability Dynamic On-Time N Y 3 AMA Automated Monitoring of Availability Dynamic Short-Time Y Grids vs. clusters: resource availability information is not always present 4 HMA Human Monitoring of Availability Dynamic Long-Time Y November 24, 2018

3.2. Performance Evaluation (1/2) Overview Scenario: SA vs. KA vs. AMA vs. HMA AMA: information update 60s, 1h HMA: information update 1 week, 1 month, never (fixed availability information) Metrics Traditional: AWT, ART Adapted for dynamic availability: utilization, throughput, goodput Failures, number and reason (submission or execution) Experimental setup Custom trace-driven discrete event simulator Simulate Grid’5000 Use job and availability traces: 06/2005 - 10/2006 (739K jobs) November 24, 2018

3.2. Performance Evaluation (2/2) Sample Results see article for detailed results Avg. Norm. G’put. [cpus/day/proc] Goodput decreases with intervention delay Avg. Norm. T’put. [jobs/day/proc] Throughput decreases with intervention delay KA = AMA > HMA 1wk > HMA 1mo >> SA Job Failures [%] Human intervention leads to sub. failures Any intervention better than No intervention November 24, 2018 SA KA AMA 60s AMA 1h HMA 1w HMA 1mo HMA Never Model

4. Conclusion and Future Work Q1: What are the characteristics of the resource (un)availability in grids? Location, Inter-arrival (MTBF), Duration (MTTR), Size (Cor.Failures) Model for grid resource availability Q2: What is the performance impact of the dynamic grid resource availability? Four models for grid resource availability information Trace-based performance evaluation through simulations KA = AMA > HMA >> SA Future Work Validate the model for other grids Model-based performance evaluation Availability-aware scheduling policies for grids November 24, 2018

Thank you! Questions? Remarks? Observations? Help building our community’s Grid Workloads Archive: http://gwa.ewi.tudelft.nl/ Add your Job and Resource Availability Traces! November 24, 2018