1. Power Measurements on Mira Argonne Leadership Computing Facility Susan Coghlan Mira Project Manager Argonne National Laboratory June 18, 2012

2. The Mira System Argonne National Laboratory - ISC 2012 2 Mira Statistics 10 PF Peak8.1 PF HPL 48 Racks48K compute nodes 768K cores3.1 B threads 768 TB memory16 GB memory per node 28 PB disk240 GB/s 4.8MW peak48 racks at 100kw each

3. Mira 480V Compute Power Distribution System Argonne National Laboratory - ISC 2012 3 13.2KV Substations Transformers 480V Distribution Panels Eaton Digitrip Optim 1050 BG/Q Compute Racks Measuring points Bulk Power Modules (BPM) NodesDCAs 7 Domains Bulk Power Modules (BPM) NodesDCAs 7 Domains Bulk Power Modules (BPM) NodesDCAs 7 Domains 36 64 1024

4. ALCF Power Measurement Methodology Challenges Challenge #1: Length of time required for HPL run –Mira integration not complete, system still in hands of vendor –Difficult to justify so many hours just to measure power –Recommend using a benchmark that runs in an hour or two Challenge #2: Access to external AC power data required for L2 and L3 –Building and all infrastructure is owned by a 3 rd party Trust –Distribution panel power measurement data only visible to the building management company –All contact with building management company must go through the Argonne Data Center Manager (ADCM) –Still no data after months of working through ADCM –Attempted to find boxes to measure between panels and racks Impossible to find existing boxes – 480V 3 Phase, receptacle HBL560R7W Custom ones cost just under $10K per rack to rent Argonne National Laboratory - ISC 2012 4

5. Mira 480V Compute Power Distribution System Argonne National Laboratory - ISC 2012 5 13.2KV Substations Transformers 480V Distribution Panels Eaton Digitrip Optim 1050 BG/Q Computes Measuring points Bulk Power Modules (BPM) NodesDCAs 7 Domains Bulk Power Modules (BPM) NodesDCAs 7 Domains Bulk Power Modules (BPM) NodesDCAs 7 Domains 36 64 1024 Accessible to ALCF

6. Data from 48 racks running HPL Argonne National Laboratory - ISC 2012 6