1. MANAGING DISTRIBUTED UPS ENERGY FOR EFFECTIVE POWER CAPPING IN DATA CENTERS ISCA 2012 Vasileios Kontorinis, L.Zhang, B.Aksanli, J.Sampson, H.Homayoun, E. Pettis*, D. Tullsen, T. Rosing *Google UCSD

2. Datacenter market is growing World is becoming more IT dependent. Internet users increased from 16% to 30% of world population in 5 years [Internet World Stats] Smart phones are projected to jump from 500M in 2011 to 2B in 2015 [Inter.Telecom.Union] Internet heavily depends on Datacenters Data center power will double in 5 years Expected worldwide Datacenter Investment in 2012: 35B$ (equivalent to GDP of Lithuania) [DataCenterDynamics] 2 Important to build cost-effective Datacenters

3. Power Oversubscription - Opportunity 3 Datacenter Supporting equipment No Oversubscription With Oversubscription One time capital expenses Recurring Costs More servers Same infrastructure Power Oversubscription  More Cost-effective Data centers Total Cost of Ownership / Server Servers Server Cost

4. Power Oversubscription – Opportunity 4 [Barroso et al. + APC TCO calc] Assumptions: Server cost: 1500$ 28000 servers (10MW) Energy: 4.7c/KWh Power: 12$/kW Amort. Time DC: 10y, servers: 4y Distributed LA-based UPS Available at: http://cseweb.ucsd.edu/~tullsen/DCmodeling.html

5. Power Oversubscription using Stored Energy 5 Leverage diurnal patterns of web services Discharge UPS batteries during high activity (once per day) Recharge during high (once per day) Power Time Power Time Peak Power Pulse Model Diurnal Power Profile Peak Power Pulse Low Power Pulse Power Profile Shaping Peak Power Reduction … M Tu W … Su + _ + _ UPS stored Energy

6. Centralized UPS Used in most small / medium data centers Scales poorly High losses in AC-DC-AC conversion (5-10%) Centralized single point of failure, requires redundancy 6 Increasingly cost-inefficient for large data centers X

7. Distributed UPS Used in large data centers Scales with data center size Avoids AC-DC-AC conversion Distributed points of failure 7 Cheaper UPS solution Facebook GoogleGoogle

8. Place more servers under same power infrastructure Related work and our proposal Centralized UPSs for power capping [Govindan, ISCA 2011] Distributed UPSs for rare power emergencies [Govindan, ASPLOS 2012] Our proposal: Provision distributed UPS for peak power capping Different battery technology Shave power on daily basis 8 Utility … Diesel Generator PDUs Racks + _ + _ UPS Better amortize capex costs

9. Outline Introduction Choosing the right battery for power shaving Datacenter workload and power modeling Policies and results Conclusions 9

10. Outline Introduction Choosing the right battery for power shaving Datacenter workload and power modeling Policies and results Conclusions 10

11. Competing Battery Technologies 11 Lead Acid (LA) Lithium Cobalt Oxide (LCO) Lithium Iron Phosphate (LFP) Electric

12. Metrics 12 Backup UPS batteries rarely used (3-4 times per year) Proper metrics: Cost Size Backup + peak shaving UPS batteries used on daily basis Proper metrics: Charge cycles Cost Size Recharge speed Wh / $ Volumetric Density (Wh / liter) Wh * cycles / $ Volumetric Density (Wh / litre) ( % charge / hour)

13. Battery Technology Comparison 13 Backup: Lead Acid (cheaper) Backup+Peak Shaving: Lithium Iron Phosphate (cost effective)

14. Datacenter Shaved Energy Server level Shaved Energy Number of servers Power supply efficiency Capacity of server level battery: Battery discharge properties DoD Lifetime capacity loss Size UPS Cost + UPS Depreciation UPS Cost = Bat.Cap.*$/Ah UPS depr. = UPS Cost/expected battery life Battery Capacity-Cost Estimation 14 LFPLead Acid Power Time Peak Reduction Peak Duration

15. Assumptions 15

16. TCO savings with peak duration 16 LFP more space,energy efficient than LA, can shave more! The more we shave, the more we gain! LA LFPLA LFP size constraintLA size constraint

17. TCO savings with battery DoD 17 (a) LA(b) LFP Sweet DoD spot for TCO savings (LA: 40%, LFP: 60%) + _ + _ High DoD Low DoD When shaving same energy: + _ + _

18. Key points for battery selection When using batteries for peak power shaving: Shave as much power as possible (reasonably sized battery) There is a DoD sweet spot, maximizing TCO savings LFP better technology because: lots of recharges more efficient discharge higher energy density cheaper in the future 18 What if: - Servers with unbalanced load? - Day-to-day variation in demand?

19. Outline Introduction Choosing the right battery for power shaving Datacenter workload and power modeling Policies and results Conclusions 19

20. Workload Modeling Whole year traffic data from Google Transparency Report Apply weights according to web presence: (Search 29.2%, Social Networking 55.8%, Map Reduce 15%) Present results for 3 worst consecutive days (11/17/2010-11/19/2010) 20

21. Service Time Workload Modeling (cont.) Model 1000 machine cluster, with 5 PDUs, 10 racks per PDU, 20 servers (2u) per rack. We simulate load based on M/M/8 queues and scale inter-arrival time according to workload traffic Job Scheduler (Round Robin or Load-aware) Job …….. Interarrival Time 8 Cores (consumers)/ Server 21

22. Outline Introduction Choosing the right battery for power shaving Datacenter workload and power modeling Policies and results Conclusions 22

23. Policy goals Guarantee power budget at specific level of power hierarchy Discharge during only high activity, charge during only low activity Effective irrespective of job scheduling Make uniform battery usage 23

24. AvailableIn Use Recharge Not Available Power over Threshold Power below Threshold Reached DoD Goal Recharge Complete (Power + Bat. Recharge Power) below Threshold Uncoordinated Policy Applied at the server level Easy to implement Runs independently per server DoD goal set to 60% of battery capacity (LFP) 24

25. 25 Round Robin Scheduling Uncoordinated Policy Results Batteries discharge when not required Batteries recharge during peak Fails to guarantee budget Budget violation

26. Uncoordinated Policy Results (cont.) 26 Coordination is required!! Load-aware Scheduling Batteries discharge all together (wasteful) Recharge all together (violates budget) Fails to guarantee budget Budget violation

27. Applied at higher levels (PDU, Cluster) Requires remote battery enable/disable, initiate recharge Number of batteries enabled proportional to peak magnitude Batteries used spatially distributed Coordinated Control 27 Overall Power Day1Day2Day3 100 server equivalent 200 server equivalent 0 server equivalent 200 server equivalent 300 server equivalent rack1rack2

28. Peak power reduction of 19%  23% more servers  6.2% TCO/server reduction Coordinated Policies 28 Power cap close to Average power (ideal) of 250W Pdu-level Cluster-level

29. Discussion: Energy proportionality Sharper, thinner peaks We can shave more power, with same stored energy Peak power reduction of up to 37.5% with the 40Ah LFP battery Energy Proporional Servers Modern Servers Overall Power Day1Day2Day3 29

30. Concluding remarks Battery provisioning of distributed UPS topologies to cap power and oversubscribe data center is beneficial Critical to reconsider battery properties (technology, capacity, DoD) Coordination of charges and discharges is required We cap peak power by 19%, allow 23% more servers and better amortize capex costs Achieve 6.2% reduction in TCO/server ($15M -- 28k server DC) 30

31. BACKUP SLIDES 31

32. TCO savings with battery cost 32 TCO savings increase over time with LFP! LA is stable technology LFP advancements expected, due to electric vehicles

33. Scenario 1: Unexpected daily traffic We use the additional 35% capacity in our batteries (DoD optimized for TCO savings at 60%) Scenario 2: Batteries are not replaced immediately With 50% of batteries dead we can still reduce peak by 15% When things go wrong? 33 Grouping battery maintenance/replacement for cost savings possible

34. Exploration of Dead Batteries 34

35. No DVFSWITH DVFS Discussion: DVFS To DVFS or not DVFS? Datacenter SLAs violations likely during peak load DVFS bad during high demand Great during low demand Creates higher margins for aggressive battery capping Potential SLA violation SLA violation unlikely Overall Power Day1Day2Day3 35

36. = = = Battery Capacity-Cost Estimation 36 LFP Lead Acid (~twice volume) Power Time Peak Reduction Peak Duration = PeakReduction * PeakDuration

37. Battery Related Assumptions 37

38. Workload partitioning 38

39. 39 Distributed Algorithm