1. Design and Analysis of an Energy Agile Cluster Computing System Andrew Krioukov, Prashanth Mohan, Stephen Dawson- Haggerty, Sara Alspaugh, David Culler, Randy Katz 1

2. Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional reactive, mostly power proportional T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 2

3. Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional reactive, mostly power proportional T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 3

4. Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional power proportional, reactive, grid-aware T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 4

5. Grid Internet SUPPLIES: provide power communicate renewable availability, price LOADS: adapt demand communicate forecast electricity information Pieces Needed 5

6. Non-dispatchable, variable supply Power proportional, grid-aware loads NREL Western Wind and Solar Integration Study Dataset http://wind.nrel.gov/Web_nrel/ Pacheco wind farm Scientific computing cluster Figure of merit: amount of wind used. How do we get here? Renewable Integration 6

7. P OWER T IME oblivious, flat load dispatchable supply power proportionality grid- awareness 7

8. Data Center Loads data center consumption dominated by IT load IT load driven by workload need power proportionality need load shaping mechanism ServerIdle:Peak HP ProLiant DL160 63.5% Apple XServe 3.151.8% IBM System x3450 51.6% Dell PowerEdge 2950 57.9% Pelley, et. al, Understanding and Abstracting Total Data Center Power, 2009 Barroso et. al. The Case for Energy-Proportional Computing, 2007 SPECpower Results http://www.spec.org/power_ssj2008/results/power_ssj2008.html 5,000 servers at Google average 30% utilization IT equipment is not power proportional power (W) utilization 8

9. Power Proportionality Spinning Reserve 9

10. Architecture 10

11. Outline Motivation Enabling technology Methodology Algorithms Evaluation 11

12. Renewable Energy Component 12

13. Formulation We assume the wind farm is sized for the data center. Option 1: grid blend (open system) Wind Other Requires assuming load is negligible fraction of grid – not realistic Option 2: dedicated wind farm (closed system) Fit load to specific wind farm 13 http://www.greenhousedata.com/

14. Wind Wind power over 48 hours from a wind farm in Monterrey County, California. Variation in wind power for month long intervals at multiple wind farms. 14

15. Workload Component 15

16. Workloads Torque jobs Num Jobs Batch: Less latency sensitive, longer jobs e.g., analytics, scientific computing Request Rate Wikipedia traffic Interactive: Latency sensitive, generally short jobs e.g., web app server, email server, etc. 16

17. Interactive Workloads Trackable request pattern Easier to shift work spatially Less temporal slack Time Request Rate 17

18. WorkloadExamplePower Proportionality Load Shaping InteractiveWikipediarequires work e.g., napsac no temporal slack, trade QoS for energy BatchTorquealmost given e.g., on-demand lots of temporal slack, larger design space 18

19. Grid-Aware Interactive example goal: respond to price spikes method: decrease work at a given point in time by returning fewer page items 19

20. Load shed at peak:50% Power reduction:50% Reduction in cost of running cluster: 50% Reduction in daily energy costs: 6% Improved grid stability 20

21. Batch Workloads Jobs 35% avg. utilization Lots of temporal slack Easy to make power proportional More scheduling freedom 21

22. WorkloadExamplePower Proportionality Load Shaping InteractiveWikipediarequires work e.g., napsac no temporal slack, trade QoS for energy BatchTorquealmost given e.g., on-demand lots of temporal slack, larger design space 22

23. Slack slack = max run time – job duration 23

24. Cluster: NERSC Franklin Average duration: 98 min Average slack: 68 min Cluster: EECS PSI Average duration: 55 min Average slack: 17 hours Slack in Real Systems 24

25. Grid-Aware Batch Scheduling example goal: shape load to match wind availability method: exploit temporal slack Pacheco wind farm Scientific computing cluster 25

26. Greedy Algorithm B(t) = power budget for next 10 min Sort jobs by slack Schedule all jobs with no remaining slack Schedule other eligible jobs in least-remaining- slack order until B(t) is exceeded 26

27. Run-immediately, grid-oblivious scheduler Greedy, grid-aware scheduler Grid-aware scheduling increases wind energy use. Correspondingly, reduces grid dependence. 27

28. When wind farm is sized to match data center, we reduce grid dependence by 50%. This comes very close to optimal. 28

29. Reduction in grid dependence is robust to choice of wind farm. 29

30. 30

31. As slack increases, grid dependence diminishes. PSI Franklin 31

32. Grid-aware scheduling is equivalent to 5 hours worth of data center-sized batteries. 32

33. Grid-aware scheduling is equivalent to 5 hours worth of data center-sized batteries. 4 33

34. Summary Power proportionality and grid-aware scheduling Energy savings, renewable integration, grid stability reduce grid dependence by half equivalent to 5 hours of batteries Next steps slack in other systems...? 34

35. Q UESTIONS ? T HE E ND 35

36. Power Proportional Torque have working implementation here also – see demo tonight (these guys) 36

37. Old Grid easy to match supplies follow loads by reacting to voltage sags and frequency shifts P OWER T IME oblivious, flat load dispatchable supply 37

38. Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional reactive, mostly power proportional T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 38

39. Currently statistical multiplexing to smooth load limited renewables and special pricing Mix dispatchable and non-dispatchable supply Mix flat and oblivious power proportional loads Soda Hall Power (kW) CA Grid Power (kW) 39

40. Problem Symptoms $/MWh €/MWh -200€ to 750€ EPEX Spot Price Germany -$50 to $160 CAISO Oasis RTM Price California 40

41. Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional reactive, mostly power proportional T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 41

42. Wind Scale Impact 42

43. Low Wind Most of the available wind is used All algorithms perform comparably 17.68% wind energy20.63% wind energy Cluster Grid Energy Cluster Wind Energy Unused Wind Energy 43

44. High Wind Scheduling yields highest improvement Cluster Grid Energy Cluster Wind Energy Unused Wind Energy 63.43% wind energy81.55% wind energy 44

45. Power Proportionality 45 Requests Availability Forecasts

46. Grid Challenges Keith I. Farkas, et. al. Quantifying the Energy Consumption of a Pocket Computer and a Java Virtual Machine. 2000 NREL Western Wind and Solar Integration Study Dataset http://wind.nrel.gov/Web_nrel/ 8x Power (W) ARM based mobile computer - High dynamic variability in loads Altamont Pass, CA wind farm - Renewable supplies are non-dispatchable - Transmission and distribution bottlenecks 46

47. 47

48. Data Center Loads Chiller CRACUPS PDU Chiller Availability Forecasts Power Usage Effectiveness = data center power consumption is driven by IT load Pelley, et. al, Understanding and Abstracting Total Data Center Power, 2009. Barroso et. al, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, 2009. What determines data center load? 48

49. IT Load SAN / DB Requests What determines IT power? Ideally, the workload. Barroso et. al. The Case for Energy-Proportional Computing, 2007 Utilization Fraction of time 5,000 servers at Google. Average 30% utilization. 49

50. Non-Power Proportional Servers Power (W) ServerIdle:Peak HP ProLiant DL160 63.5% Apple XServe 3.1 51.8% IBM System x3450 51.6% Dell PowerEdge 2950 57.9% Utilization Source: SPECpower Results http://www.spec.org/power_ssj2008/results/power_ssj2008.html 30% Utilization 75% Peak Power 40% Efficiency 50