February 25, 2008 The Emerging Front Range HPC Collaboratory Dr. Rich Loft: Director, Technology Development Computational and Information Systems Laboratory National Center for Atmospheric Research

February 25, 2008 Front Range HPC Collaboratory Components… Facilities HPC Systems Grid Technology Applications/Workflows Science Collaborations Science Drivers Networking

February 25, 2008 The National Center for Atmospheric Research Climate Modeling Expertise Nobel Prize winning IPCC AR-4 Community Modeling Approach Scalable Models Grid Expertise TeraGrid Resource Provider Earth System Grid Grid-BGC (BioGeoChemistry) New Facility Plans (2012) >1 PFLOPS 8 MW >100 PB archive High Performance Computing 40 years of experience 25 TFLOPS peak computers 5 Pbyte archive

February 25, 2008 Climate Modeling… 40 km version credit: J. Hack, NCAR

February 25, 2008 Climate of the last Millennium Credit: Caspar Amman NCAR

February 25, 2008 This is a Critical Moment in Climate Change Research… Reproduce historical trends Investigate climate change Run IPCC Scenarios Investigate solutions Assess impacts Simulate adaptation strategies Work with energy industry Before IPCC AR4 After 2007

February 25, 2008 Mitigation: What we do next can change the amount of warming that occurs. Likelihood of warming (Knutti et al. 2005)

February 25, 2008 L. Buja Dimensions Slide

February 25, 2008 Why High Resolution? Ocean component of CCSM (Collins et al, 2006) Eddy-resolving POP (Maltrud & McClean,2005)

February 25, 2008 The Case for an HPC Collaboratory in the Western Interior l An integrated approach to climate change will involve –Climate modeling with regional capabilities –Mitigation efforts via, for example, carbon sequestration l Most sequestration reservoir formations in the Western Interior l Unconventional oil and gas locations in Western Interior l Regional climate change effects on the Great Plains biomass l Half of all EPSCOR states are in the Western Interior l Power Costs DOE FutureGen Project in Texas

February 25, 2008 Experimental Program to Stimulate Competitive Research (EPSCoR) Most of the Western Interior is EPSCoR

February 25, 2008 National Security Agency - The power consumption of today's advanced computing systems is rapidly becoming the limiting factor with respect to improved/increased computational ability."

February 25, 2008 Moore’s Law = More Cores: Quad Core “Barcelona” AMD Processor… Can 8, 16, 32 cores be far behind?

February 25, 2008 The Path to Petascale is Through Massive Parallelism… Cray-2/8 1986Blue Gene/L K CPU’s 367 TFLOPS 3.9 GFLOPS

February 25, 2008

Cheap Power in the Western Interior

February 25, 2008 Front-Range Collaboratory Resources Current Systems l GECO (CSM+NREL) –2144, 2.66 GHz Intel CPU Dell Cluster (22.8 TFLOPS) l NCAR –2048, 700 MHz PwrPC CPU Blue Gene/L (5.7 TFLOPS) Planned Systems (FY2009) l l NCAR – –~30 TFLOPS system upgrade – –1.5 PB HPSS-based archive l l CU – –~100 TFLOPS MRI Proposal to NSF Farther Out NCAR Supercomputing Center (NSC)

February 25, 2008 Blue NCAR: A platform for scalability research “Frost”

February 25, 2008 NCAR is part of the TeraGrid - NSF’s HPC Grid SDSC TACC UC/ANL NCSA ORNL PU IU PSC TeraGrid is a facility that integrates computational, information, and analysis resources at the San Diego Supercomputer Center, the Texas Advanced Computing Center, the University of Chicago / Argonne National Laboratory, the National Center for Supercomputing Applications, Purdue University, Indiana University, Oak Ridge National Laboratory, the Pittsburgh Supercomputing Center, and the National Center for Atmospheric Research. NCAR Caltech USC-ISI Utah Iowa Cornell Buffalo UNC-RENCI Wisc

February 25, 2008 HPC Infrastructure End State Integrated HPC Resource & Visualization Systems WAN Network HPC Storage Cluster 10Gb/s MSS Archive TG Shared Filesystems Science gateways Data Preservation Switch Data SAN

February 25, 2008 Compute cabinets I/O nodes Interactive nodes Network Switch cabinets Future HPC Systems Are Likely to be Heterogeneous I/O Subsystem Front End Switch ComputeAccelerators Accelerator cabinets

February 25, 2008 NCAR Summer Internships in Parallel Computational Science (SIParCS) l Open to: –Upper division undergrads –Graduate students l In Disciplines such as: –CS, Software Engineering –Applied Math, Statistics –ES Science l Support: –Travel, Housing, Per diem –10 weeks salary l 2008 application deadline: –Feb 1st l Number of interns selected: –8-10 l Members of traditionally underrepresented groups encouraged to apply.

February 25, 2008 Thanks! Any Questions?

February 25, 2008 Front Range HPC Collaboratory: A geographical view Ring Topology Multiple Network Providers Level3 McCleod Platte River Power Auth. ICG Upgradable to multiple 10 Gb/s waves NCAR: ~30 Tflops FY2009 CU: 100 Tflops FY2009 GECO: 11.4 Tflops NSC: 1 Pflops 2011

February 25, 2008 l Over 3000 registered users make it the most used Meteorological Research model; l Operational use by the National Weather Service, US Navy, US Army, USAF, South Korean Meteorological Service, Indian Meteorological Department; l Special forecasts are made by NCAR over the Antarctic in support of international operations there. 36 h Reflectivity Forecast 4 km WRF Model Radar Reflectivity 10 June Z l State-of-the-art Coupled Climate System Model l Open Source Code and freely available data l Significant development collaborations with: –22 US universities –US Department of Energy (DOE) n LANL, LLNL, LANL, ORNL, ANL, LBL, NERSC –NASA l Largest contributor to the 2007 IPCC AR4 –1.4° resolution –11,000 model years simulated >100 TBytes –of output data WRF: Weather and Research Forecast Model CCSM: Community Climate System Model NCAR’s Community Model Approach to Enabling Collaborative Research

February 25, 2008 IPCC

February 25, 2008 Atmospheric Resolution: Number of Northern Hemisphere Cyclones T255 ERA T159 T95 Credit: Jung et al. 2006