Where Does the Power go in DCs & How to get it Back Foo Camp 2008 2008-07-12 James Hamilton email :JamesRH@microsoft.com web: http://mvdirona.com blog: http://perspectives.mvdirona.com

Agenda Power is the important measure Power drives costs in that Data Center costs are 80% providing power and cooling infrastructure Increasing concern about DC power consumption Work done/watt Power In: Power Distribution & Optimizations Servers: Critical Load & Optimizations Heat Out: Mechanical Systems & Optimization 7/12/2008 http://perspectives.mvdirona.com

Power Distribution: Utility to CPU Power Conversions to server (each roughly 98%) High (115kVAC) to medium(13.2kVAc) [differs by geo] Uninterruptable Power Supply & Generators: Running at 13.2VAC UPS: can be rotary or battery Good ones in 97% range. Much more common 93 to 94% Common: rectify to DC, trickle to batteries, then invert to AC (~93%) No loss at generators (please don’t start them: ~130gallons/hour * 10 or so) 13.2kVACto 480VAC 480VAC to 208VAC Conversions in Server to CPU & Memory: Power Supply: 208VAC to 12VDC (80% common, ~95% affordable) VRM: 12VDC to ~1.5VDC (80% common, 90% affordable) 7/12/2008 http://perspectives.mvdirona.com

Power Redundancy at Geo-Level Over 20% of entire DC costs is in power redundancy Batteries able to supply up to 15 min at some facilities N+2 generation (2.5MW) at over $2M each Instead, use more smaller, cheaper data centers Eliminate redundant power & bulk of shell costs Average UPS in the 93% range Over 1MW wasted in 15MW facility 7/12/2008 http://perspectives.mvdirona.com

Power Distribution Optimization Rules to minimize power distribution losses: Avoid conversions (Less transformer steps & efficient or no UPS) Increase efficiency of conversions High voltage as close to load as possible Size voltage regulators (VRM/VRDs) to load & use efficient parts DC distribution potentially a small win With regulatory issues Two interesting approaches: 480VAC (or higher) to rack & 48VDC (or 12VDC) within 480VAC to PDU and 277VAC to load 1 leg of 480VAC 3-phase distribution Common design: 44% lost in distribution 1*.98*.98*.93*.98*.8*.8 => 56% (~4.4MW lost on 10MW total) Affordable technology: 1*.99*.99*.95*.95 => 88% (~1.2MW total) 7/12/2008 http://perspectives.mvdirona.com

Critical Load Optimization Power proportionality is great but “off” is even better Today: Idle server consumes ~60% power of full load Industry secret: “good” data center server utilization around ~30% Off requires changing workload location What limits 100% dynamic workload distribution? Networking constraints VIPs can’t span L2 nets, ACLs are static, manual configuration, etc. Data Locality Hard to efficiently move several TB & workload needs to be close to data Workload management: Scheduling work over resources optimizing for power with SLA constraint Server power management Most workloads don’t fully utilize all resources on server Need ability to shut off or de-clock unused server resources Very low power states recover more quickly Move from 30% utilization to 80% 7/12/2008 http://perspectives.mvdirona.com

CEMS: Thin Slice Computing Cooperating Expendable Micro-Slice Servers Correct system balance problem with less-capable CPU Too many cores, running too fast for memory, bus, disk, … Power consumption scales with cube of clock frequency Goal: ¼ the price & much less than ½ the power Utilize high-volume client parts in server environment Goal: 20 to 50W at under $500 1U form factor or less with service-free design Longer term goals High-density, shared power supply & boot disk Eliminate non-server required components Establish viability of service free designs 7/12/2008 http://msblogs/JamesRH

Conventional Mechanical Design Server fans (from components to air) CRACs: (from air to chilled water) Air moving over long distance expensive Air control often poor with hot/cold mixing Secondary water circuit (variable flow) Primary water circuit (fixed flow) Water side economizer & A/C evaporator Condensate circuit A/C condenser Water side economizer Cooling tower 7/12/2008 http://perspectives.mvdirona.com

Mechanical Optimization Simple rules to minimize cooling costs: Raise data center temperatures Tight control of airflow with short paths Cooling towers rather than A/C Air side economization (open the window) Low grade, waste heat energy reclamation Best current designs have water close to load but don’t use direct water cooling Lower heat densities could be 100% air cooled but density trends suggest this won’t happen Common mechanical designs: 24% lost in cooling Assume reduction to 1/3 current 24% to 8% for 16% savings 7/12/2008 http://perspectives.mvdirona.com

Summary Some low-scale facilities incredibly bad Assuming current high-scale installation: Power distribution savings ~32% Save 8% in power distribution to server Save further 24% power distribution losses in server Cooling Savings: ~16% Conservatively estimate 1/3 the power using air-side economization 24% loss down to 8% for a 16% power savings Server Utilization: ~90% Move from 30% to 80% through DC-wide workload scheduling 30% load @ 60% of full load power to 80% load @ 100% of full load power 2.6x work at 1.7x more power for a gain of 90% Cooperative, Expendable, Micro-slice Servers: ~12% ½ the power but less capable server (most workloads are memory or disk I/O bound) Conservatively assume .8x work done .5x power => 30% savings 4.0x gains in work done/watt look attainable: 1*1.32*1.16*1.90*1.30 => 3.8x (some overlap between CEMS & power dist savings) Power is #3 expense in DC behind server h/w, power distribution & cooling Data center capital expense savings nearly 100% driven by power Reductions in power reduce capex, opex & is good for environment 7/12/2008 http://perspectives.mvdirona.com

Slides Perspectives Blog: These Slides: http://mvdirona.com/jrh/TalksAndPapers/JamesRH_DCPo werSavingsFooCamp08.ppt Perspectives Blog: http://perspectives.mvdirona.com 7/12/2008 http://msblogs/JamesRH Microsoft