Design and Analysis of an Energy Agile Cluster Computing System Andrew Krioukov, Prashanth Mohan, Stephen Dawson- Haggerty, Sara Alspaugh, David Culler, Randy Katz 1

Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional reactive, mostly power proportional T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 2

Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional reactive, mostly power proportional T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 3

Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional power proportional, reactive, grid-aware T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 4

Grid Internet SUPPLIES: provide power communicate renewable availability, price LOADS: adapt demand communicate forecast electricity information Pieces Needed 5

Non-dispatchable, variable supply Power proportional, grid-aware loads NREL Western Wind and Solar Integration Study Dataset Pacheco wind farm Scientific computing cluster Figure of merit: amount of wind used. How do we get here? Renewable Integration 6

P OWER T IME oblivious, flat load dispatchable supply power proportionality grid- awareness 7

Data Center Loads data center consumption dominated by IT load IT load driven by workload need power proportionality need load shaping mechanism ServerIdle:Peak HP ProLiant DL % Apple XServe % IBM System x % Dell PowerEdge % Pelley, et. al, Understanding and Abstracting Total Data Center Power, 2009 Barroso et. al. The Case for Energy-Proportional Computing, 2007 SPECpower Results 5,000 servers at Google average 30% utilization IT equipment is not power proportional power (W) utilization 8

Power Proportionality Spinning Reserve 9

Architecture 10

Outline Motivation Enabling technology Methodology Algorithms Evaluation 11

Renewable Energy Component 12

Formulation We assume the wind farm is sized for the data center. Option 1: grid blend (open system) Wind Other Requires assuming load is negligible fraction of grid – not realistic Option 2: dedicated wind farm (closed system) Fit load to specific wind farm 13

Wind Wind power over 48 hours from a wind farm in Monterrey County, California. Variation in wind power for month long intervals at multiple wind farms. 14

Workload Component 15

Workloads Torque jobs Num Jobs Batch: Less latency sensitive, longer jobs e.g., analytics, scientific computing Request Rate Wikipedia traffic Interactive: Latency sensitive, generally short jobs e.g., web app server, server, etc. 16

Interactive Workloads Trackable request pattern Easier to shift work spatially Less temporal slack Time Request Rate 17

WorkloadExamplePower Proportionality Load Shaping InteractiveWikipediarequires work e.g., napsac no temporal slack, trade QoS for energy BatchTorquealmost given e.g., on-demand lots of temporal slack, larger design space 18

Grid-Aware Interactive example goal: respond to price spikes method: decrease work at a given point in time by returning fewer page items 19

Load shed at peak:50% Power reduction:50% Reduction in cost of running cluster: 50% Reduction in daily energy costs: 6% Improved grid stability 20

Batch Workloads Jobs 35% avg. utilization Lots of temporal slack Easy to make power proportional More scheduling freedom 21

WorkloadExamplePower Proportionality Load Shaping InteractiveWikipediarequires work e.g., napsac no temporal slack, trade QoS for energy BatchTorquealmost given e.g., on-demand lots of temporal slack, larger design space 22

Slack slack = max run time – job duration 23

Cluster: NERSC Franklin Average duration: 98 min Average slack: 68 min Cluster: EECS PSI Average duration: 55 min Average slack: 17 hours Slack in Real Systems 24

Grid-Aware Batch Scheduling example goal: shape load to match wind availability method: exploit temporal slack Pacheco wind farm Scientific computing cluster 25

Greedy Algorithm B(t) = power budget for next 10 min Sort jobs by slack Schedule all jobs with no remaining slack Schedule other eligible jobs in least-remaining- slack order until B(t) is exceeded 26

Run-immediately, grid-oblivious scheduler Greedy, grid-aware scheduler Grid-aware scheduling increases wind energy use. Correspondingly, reduces grid dependence. 27

When wind farm is sized to match data center, we reduce grid dependence by 50%. This comes very close to optimal. 28

Reduction in grid dependence is robust to choice of wind farm. 29

30

As slack increases, grid dependence diminishes. PSI Franklin 31

Grid-aware scheduling is equivalent to 5 hours worth of data center-sized batteries. 32

Grid-aware scheduling is equivalent to 5 hours worth of data center-sized batteries. 4 33

Summary Power proportionality and grid-aware scheduling Energy savings, renewable integration, grid stability reduce grid dependence by half equivalent to 5 hours of batteries Next steps slack in other systems...? 34

Q UESTIONS ? T HE E ND 35

Power Proportional Torque have working implementation here also – see demo tonight (these guys) 36

Old Grid easy to match supplies follow loads by reacting to voltage sags and frequency shifts P OWER T IME oblivious, flat load dispatchable supply 37

Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional reactive, mostly power proportional T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 38

Currently statistical multiplexing to smooth load limited renewables and special pricing Mix dispatchable and non-dispatchable supply Mix flat and oblivious power proportional loads Soda Hall Power (kW) CA Grid Power (kW) 39

Problem Symptoms $/MWh €/MWh -200€ to 750€ EPEX Spot Price Germany -$50 to $160 CAISO Oasis RTM Price California 40

Grid Evolution S UPPLIES L OADS mostly dispatchable renewable, variable, intermittent, greatly non- dispatchable oblivious, stochastic, mostly non-power proportional reactive, mostly power proportional T ODAY I DEAL F UTURE oblivious, flat O LD G RID non-renewable, reactive, dispatchable 41

Wind Scale Impact 42

Low Wind Most of the available wind is used All algorithms perform comparably 17.68% wind energy20.63% wind energy Cluster Grid Energy Cluster Wind Energy Unused Wind Energy 43

High Wind Scheduling yields highest improvement Cluster Grid Energy Cluster Wind Energy Unused Wind Energy 63.43% wind energy81.55% wind energy 44

Power Proportionality 45 Requests Availability Forecasts

Grid Challenges Keith I. Farkas, et. al. Quantifying the Energy Consumption of a Pocket Computer and a Java Virtual Machine NREL Western Wind and Solar Integration Study Dataset 8x Power (W) ARM based mobile computer - High dynamic variability in loads Altamont Pass, CA wind farm - Renewable supplies are non-dispatchable - Transmission and distribution bottlenecks 46

47

Data Center Loads Chiller CRACUPS PDU Chiller Availability Forecasts Power Usage Effectiveness = data center power consumption is driven by IT load Pelley, et. al, Understanding and Abstracting Total Data Center Power, Barroso et. al, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines, What determines data center load? 48

IT Load SAN / DB Requests What determines IT power? Ideally, the workload. Barroso et. al. The Case for Energy-Proportional Computing, 2007 Utilization Fraction of time 5,000 servers at Google. Average 30% utilization. 49

Non-Power Proportional Servers Power (W) ServerIdle:Peak HP ProLiant DL % Apple XServe % IBM System x % Dell PowerEdge % Utilization Source: SPECpower Results 30% Utilization 75% Peak Power 40% Efficiency 50