CIT 470: Advanced Network and System AdministrationSlide #1 CIT 470: Advanced Network and System Administration Data Centers

CIT 470: Advanced Network and System AdministrationSlide #2 Topics Data Center: A facility for housing a large amount of computer or communications equipment. 1.Environment 2.Power 3.Racks 4.Containers

CIT 470: Advanced Network and System AdministrationSlide #3 Environmental Requirements Temperature: 64-72F (17-22C) Temperature inside case around 40F higher. Chips and hard disks fail around 120F. Humidity 35-65% Too low: static discharges. Too high: water condenses, short circuits. Power Reliable, conditioned power. Physical Security Prevent accidents, hacking, theft, vandalism.

CIT 470: Advanced Network and System AdministrationSlide #4 Environment Features HVAC: Heating, Ventilation, Air Cond UPS: Power conditioning, <1hr battery Generator: For long term power outages. Accessibility: For large equipment. Card Lock: SAs only, records entrances/exits. Fire Suppression System: Not water based. Humidity/Water Detection

CIT 470: Advanced Network and System AdministrationSlide #5 HVAC HVAC systems can fail due to Power loss. Coolant leakage. Mechanical failure. Detect HVAC failures early. Many HVACs have sensors to report failure. Temperature monitor in data center otherwise. HVAC maintenance n+1 redundancy to avoid immediate disaster. Usually handled by external contractor.

CIT 470: Advanced Network and System AdministrationSlide #6 Advanced Cooling Devices HP Smart Cooling –Sensors within machine room positions louvers in the floor to direct cool air at hot spots. –Run data center at 27C instead of 13C. IBM Heat eXchanger –Water cooled rear door for rack. –Removes ~15 kW (50000 BTU) from rack. –Useful for dealing with hot spots.

CIT 470: Advanced Network and System AdministrationSlide #7 Space Aisles Must be wide enough to move equipment. Hot spots Result from poor air flow. Servers can overheat when avg room temp low. Work space A place for SAs to work on servers. Desk space, tools, etc. Capacity Ensure that you have enough room to grow.

CIT 470: Advanced Network and System AdministrationSlide #8 Power UPS provides power. Generator provides auxiliary power. ATS switches to generator if building power fails. Liebert UPS and battery bank.

CIT 470: Advanced Network and System AdministrationSlide #9 Power Distribution Under floor power –Susceptible to water; requires water sensors. –Overhead power bus is preferrable. Overhead Power Bus –Drop power down to each rack individually. –Don’t run power cords between racks. –Each rack will have its own PDU.

CIT 470: Advanced Network and System AdministrationSlide #10 Power Distribution Units (PDU) Different power sockets can be on different circuits. Individual outlet control (pwr cycle.) Current monitoring and alarms. Network management (web or SNMP.) APC Power Distribution Unit

Typical Power Usage DevicePower Usage Intel Xeon W GHz Quad Core130 W Intel Xeon E GHz Quad Core80W Intel Xeon E GHz Dual Core80W 7200RPM Hard Drive7W 10,000RPM Hard Drive14W 15,000RPM Hard Drive20W DDR2 DIMM1.65W Video Card20-120W CIT 470: Advanced Network and System AdministrationSlide #11 Power supplies range from 50-90% efficiency, with the 80 PLUS program pushing recent ones towards those higher figures.

CIT 470: Advanced Network and System AdministrationSlide #12 The Power Problem Servers require much more power today. 4-year power cost = server purchase price. Upgrades may have to wait for electricity. Power is a major data center cost –$5.8 billion for server power in –$3.5 billion for server cooling in –$20.5 billion for purchasing hardware in –Google, MS building centers near cheap power on the Columbia and Colorado rivers.

CIT 470: Advanced Network and System Administration Power Usage Effectiveness (PUE) PUE = Data center power / Computer power –PUE=2 indicates that for each watt of power used to power IT equipment, one watt used for HVAC, power distribution, etc. –Decreases towards 1 as DC is more efficient. PUE variation –Industry average = 2 –Microsoft = 1.22 –Google = 1.19

CIT 470: Advanced Network and System AdministrationSlide #14 Physical Security Keycard + code to restrict who can access. Biometrics are becoming popular alternative. Electronically log all accesses. Surveillance cameras. No windows. No floor/ceiling access.

CIT 470: Advanced Network and System AdministrationSlide #15 Racks 19” rack standard –EIA-310D –Other standard numbers. NEBS 21” racks –Telecom equipment. 2-post or 4-post 1U = 1.75 in Air circulation (fans) Cable management Doors or open

CIT 470: Advanced Network and System AdministrationSlide #16 Rack Purposes Organize equipment –Increase density with vertical stacking. Cooling –Internal airflow within rack cools servers. –Airflow determined by arrangement of racks. Wiring –Cable guides keep cables within racks. Power infrastructure –PDUs in racks distribute power to servers.

CIT 470: Advanced Network and System AdministrationSlide #17 Buying a Rack Buy the right size –Space for servers. –+ space for power, patch panel, etc. Be sure it fits your servers. –Appropriate mounting rails. –Shelves for non-rack mountable servers. Physical/environment security –Locking front and back doors if needed. –Sufficient power and cooling. –Power/environment monitors if needed.

CIT 470: Advanced Network and System AdministrationSlide #18 Wiring It’s important to be neat. –So you can get the cable for the right server. –Label both ends of each cable. Prewiring –Run power, net, serial cables when rack setup. –Greatly reduces time to setup a new server. Hiding messiness isn’t being neat. –Raised floors can hide cables. –Some sites don’t remove old cables to avoid accidentally taking down the wrong server.

CIT 470: Advanced Network and System AdministrationSlide #19 Containers Data Center in a shipping container. –4-10X normal data center density. –Up to 2500 servers. –Up to 12.9 kW/m 2. MS/Google fill entire data centers with containers. –Industrialization of IT. –Low PUE.

CIT 470: Advanced Network and System AdministrationSlide #20 References 1.“19-inch Rack”, u, “Top 5 Things to Keep in Mind When Buying a 19” Rack,” Mark Burgess, Principles of System and Network Administration, Wiley, Data Center Journal, “Data Center,” Dan Goodin, “IT confronts the Data Center Power Crisis,” Google, 8.Hurgh, “Rack Layout,” “Project Blackbox,” Thomas A. Limoncelli and Christine Hogan, The Practice of System and Network Administration, Addison-Wesley, Microsoft Data Centers blog,