Buying into “Summit” under the “Condo” model Pat Burns, VP for IT Rick Casey, HPC Manager H.J. Siegel, Chair of the ISTeC MAC

Agenda 1. Welcome, introductions – HJ 2. Review NSF Summit award – Pat 3.      Review Summit configuration – Rick 4. Present “Condo” buy-in models – Rick 5.      Q&A – HJ

Welcome from ISTeC ISTeC: CSU's Information Science & Technology Center Current HPC at CSU: ISTeC Cray from 9/09 $630K NSF grant NEW NSF award to ISTeC for greatly enhanced HPC Thanks to those who helped us get this award through ISTeC

The Joint NSF MRI Award NSF MRI proposal submitted under the RMACC (http://www.rmacc.org) Joint award to CSU and CU: 450 TFLOPs (~200 on top 500) CSU: $850k (23%) CU: $2.5m (67%) 10% of cycles offered to RMACC participants Giant RFP process: award to Dell/DDN: “Summit” system Housed and operated at CU (CSU fiber connected), no cost to CSU Undergoing final acceptance testing now Limited opportunity to buy Into the system, subsidized for common infrastructure

Summary of Benefits Great pricing via large scale purchase Zero operational costs Hardware subsidy Central user and application support

Summit: Schematic Rack Layout Storage rack Compute rack 1 Compute rack 2 Compute rack 3 Compute rack 4 Compute rack 5 Compute rack 6 Compute rack 7 1 PB scratch GPFS DDN SFA14K HiMem nodes (5) 2 TB RAM / node Ethernet mgt. nodes OmniPath leaf nodes Nvidia K80 GPU nodes (10) core nodes Intel Knights Landing Phi nodes (20) OPA fabric Gateway nodes Intel Haswell nodes (376) Note: actual rack layout may differ from this schematic

CPU Nodes 376 CPU nodes Dell Poweredge C6320 9,024 total Intel Haswell CPU cores 4 nodes, 96 cores per chassis 200 GB SATA SSD / chassis 2X Intel Xeon E5-2680v3; 2.5 GHz, per node 24 CPU cores / node 128 GB RAM / node 5.3 GB RAM / CPU core

GPU Nodes 10 GPU nodes Dell Poweredge C4130 99,840 GPU cores 2X Nvidia K80 GPU cards / node 2X Intel Xeon E5-2680v3; 2.5 GHz / node 200 GB SATA SSD / node 24 CPU cores / node 128 GB RAM / node 5.3 GB RAM / CPU core Nvidia K80

HiMem Nodes 5 HiMem nodes Dell Poweredge R930 4X Intel Xeon E7-4830v3; 2.1 GHz 2 TB RAM / node (DDR4) 48 CPU cores / node 42 GB RAM / CPU core 200 GB SAS SSD / node 12 TB SAS HDD / node

Interconnect Intel OmniPath interconnect 100 Gbyte / sec. bandwidth Fat tree topology, 2:1 blocking

Storage 1 Petabyte (PB) scratch storage DDN SFA14K block storage appliance GRIDScaler (GPFS integration) Direct native connect to OmniPath

Access to Summit (In Process) Via CSU’s fiber infrastructure Identical for CU and CSU users Simple account applications required Start-up, small allocations “automatic” Goal is to support widespread usage Larger allocations will be given in accordance with needs Limited “whole machine” runs will be available Will require eID/Duo for authentication Will require Globus for file transfers Recommended from CSU’s Research DMZ network/storage

“Condo” Model Buy-in Buy-in in units of chasses CPU chassis has 4 nodes (band together to buy?) Allocations will be given equal to 8,760 hrs./year x purchased size Ex.: 1 CPU node purchased, allocation = 8,760 x 24 core-hrs./yr. All resources are shared when available: scale up to larger sizes All shared, common elements are subsidized, until $$$ run out Power, cooling, data center space, staff Power distribution units (PDU’s) Ethernet switches Omnipath common fabric (you must purchase card for each node) Omnipath cabling

Deadlines (for both CU and CSU) Nov. 10 for commitments Get commitments (specs and account numbers) to Richard.Casey@colostate.edu, (970) 980-5975 PO by 12/1

Excel Spreadsheet Discussed

Q&A Most Welcome Thank You!