SciDAC CCSM Consortium: Software Engineering Update Patrick Worley Oak Ridge National Laboratory (On behalf of all the consorts) Software Engineering Working.

Slides:



Advertisements
Similar presentations
DOE Global Modeling Strategic Goals Anjuli Bamzai Program Manager Climate Change Prediction Program DOE/OBER/Climate Change Res Div
Advertisements

O AK R IDGE N ATIONAL L ABORATORY U. S. D EPARTMENT OF E NERGY Center for Computational Sciences Cray X1 and Black Widow at ORNL Center for Computational.
CCSM Testing Status Tony Craig Lawrence Buja Wei Yu CCSM SEWG Meeting Feb 5, 2003.
Dynamical Downscaling of CCSM Using WRF Yang Gao 1, Joshua S. Fu 1, Yun-Fat Lam 1, John Drake 1, Kate Evans 2 1 University of Tennessee, USA 2 Oak Ridge.
O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Global Climate Modeling Research John Drake Computational Climate Dynamics Group Computer.
Phil’s Promised Presentation on POP’s Present Progress, Performance and Penultimate Post-present Plan POP People P. Malone, P. Smith, P. Maltrud, P. Jones,
11Sahalu JunaiduICS 573: High Performance Computing5.1 Analytical Modeling of Parallel Programs Sources of Overhead in Parallel Programs Performance Metrics.
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn.
A Parallel Structured Ecological Model for High End Shared Memory Computers Dali Wang Department of Computer Science, University of Tennessee, Knoxville.
WRF-VIC: The Flux Coupling Approach L. Ruby Leung Pacific Northwest National Laboratory BioEarth Project Kickoff Meeting April 11-12, 2011 Pullman, WA.
Why Climate Modelers Think We Need a Really, Really Big Computer Phil Jones Climate, Ocean and Sea Ice Modeling (COSIM) Climate Change Prediction Program.
NSF NCAR | NASA GSFC | DOE LANL ANL | NOAA NCEP GFDL | MIT Adoption and field tests of M.I.T General Circulation Model (MITgcm) with ESMF Chris Hill ESMF.
Slide 1 Auburn University Computer Science and Software Engineering Scientific Computing in Computer Science and Software Engineering Kai H. Chang Professor.
ESMF Development Status and Plans ESMF 4 th Community Meeting Cecelia DeLuca July 21, 2005 Climate Data Assimilation Weather.
CESM/RACM/RASM Update May 15, Since Nov, 2011 ccsm4_0_racm28:racm29:racm30 – vic parallelization – vic netcdf files – vic coupling mods and “273.15”
Computational Design of the CCSM Next Generation Coupler Tom Bettge Tony Craig Brian Kauffman National Center for Atmospheric Research Boulder, Colorado.
Chapter 3 Parallel Algorithm Design. Outline Task/channel model Task/channel model Algorithm design methodology Algorithm design methodology Case studies.
ESMF Performance Evaluation and Optimization Peggy Li(1), Samson Cheung(2), Gerhard Theurich(2), Cecelia Deluca(3) (1)Jet Propulsion Laboratory, California.
CSEG Update Mariana Vertenstein CCSM Software Engineering Group Mariana Vertenstein CCSM Software Engineering Group.
Overview of Recent MCMD Developments Jarek Nieplocha CCA Forum Meeting San Francisco.
BG/Q vs BG/P—Applications Perspective from Early Science Program Timothy J. Williams Argonne Leadership Computing Facility 2013 MiraCon Workshop Monday.
CESM/ESMF Progress Report Mariana Vertenstein NCAR Earth System Laboratory CESM Software Engineering Group (CSEG) NCAR is sponsored by the National Science.
ARGONNE NATIONAL LABORATORY Climate Modeling on the Jazz Linux Cluster at ANL John Taylor Mathematics and Computer Science & Environmental Research Divisions.
Earth System Modeling Framework Status Cecelia DeLuca NOAA Cooperative Institute for Research in Environmental Sciences University of Colorado, Boulder.
1 CCSM Component Performance Benchmarking and Status of the CRAY X1 at ORNL Patrick H. Worley Oak Ridge National Laboratory Computing in Atmospheric Sciences.
Introduction to the Earth System Modeling Framework International Workshop on Next Generation Climate Models for Advanced High Performance Computing Facilities.
Petascale –LLNL Appro AMD: 9K processors [today] –TJ Watson Blue Gene/L: 40K processors [today] –NY Blue Gene/L: 32K processors –ORNL Cray XT3/4 : 44K.
Presented by Global Coupled Climate and Carbon Cycle Modeling Forrest M. Hoffman Computational Earth Sciences Group Computer Science and Mathematics Division.
Lecture 4 TTH 03:30AM-04:45PM Dr. Jianjun Hu CSCE569 Parallel Computing University of South Carolina Department of.
CCSM Portability and Performance, Software Engineering Challenges, and Future Targets Tony Craig National Center for Atmospheric Research Boulder, Colorado,
CCSM Tutorial CCSM Software Engineering Group June
August 2001 Parallelizing ROMS for Distributed Memory Machines using the Scalable Modeling System (SMS) Dan Schaffer NOAA Forecast Systems Laboratory (FSL)
MESQUITE: Mesh Optimization Toolkit Brian Miller, LLNL
CS 484 Designing Parallel Algorithms Designing a parallel algorithm is not easy. There is no recipe or magical ingredient Except creativity We can benefit.
Belgrade, 26 September 2014 George S. Markomanolis, Oriol Jorba, Kim Serradell Overview of on-going work on NMMB HPC performance at BSC.
CCSM Performance, Successes and Challenges Tony Craig NCAR RIST Meeting March 12-14, 2002 Boulder, Colorado, USA.
On the Road to a Sequential CCSM Robert Jacob, Argonne National Laboratory Including work by: Mariana Vertenstein (NCAR), Ray Loy (ANL), Tony Craig (NCAR)
Presented by The SciDAC CCSM Consortium Project John B. Drake Computer Science and Mathematics Division Computational Earth Sciences Group.
SDM Center High-Performance Parallel I/O Libraries (PI) Alok Choudhary, (Co-I) Wei-Keng Liao Northwestern University In Collaboration with the SEA Group.
Warren M. Washington NCAR The Parallel Climate Model (PCM) and Transition to a Climate Change Version of the Community Climate System Model (CCSM)
A Pattern Language for Parallel Programming Beverly Sanders University of Florida.
O AK R IDGE N ATIONAL L ABORATORY U.S. D EPARTMENT OF E NERGY Data Requirements for Climate and Carbon Research John Drake, Climate Dynamics Group Computer.
Presented by LCF Climate Science Computational End Station James B. White III (Trey) Scientific Computing National Center for Computational Sciences Oak.
CCSM Software Engineering Update Tony Craig CCSM SEWG Meeting Feb 4, 2003.
March 31, 2004BGC Working Group Interactive chemistry in CAM Jean-François Lamarque, D. Kinnison and S. Walters Atmospheric Chemistry Division NCAR.
To Connect to Wireless Network Boot up with wifi enabled, no encryption Open browser, go to or
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH C. DeLuca/NCAR, J. Anderson/NCAR, V. Balaji/GFDL, B. Boville/NCAR, N. Collins/NCAR,
The Community Climate System Model (CCSM): An Overview Jim Hurrell Director Climate and Global Dynamics Division Climate and Ecosystem.
Mirin – AMWG 2006 – Slide 1 Coupled Finite-Volume Simulations at One-Degree Resolution Art Mirin and Govindasamy Bala Lawrence Livermore National Laboratory.
March 9, 2004CCSM AMWG Interactive chemistry in CAM Jean-François Lamarque, D. Kinnison S. Walters and the WACCM group Atmospheric Chemistry Division NCAR.
Overview of the CCSM CCSM Software Engineering Group June
VisIt Project Overview
Introduction to Parallel Computing: MPI, OpenMP and Hybrid Programming
GMAO Seasonal Forecast
Community Earth System Model (CESM) for CMIP6
Parallel Programming By J. H. Wang May 2, 2017.
Parallel Algorithm Design
Software Practices for a Performance Portable Climate System Model
GFDL Climate Model Status and Plans for Product Generation
Parallel Programming in C with MPI and OpenMP
Highlights from CCSM3’s Special Issue of the Journal of Climate
Joint GEOS-Chem and NCAR Modeling Workshop:
Q: What Does the Future Hold for “Parallel” Languages?
Mariana Vertenstein CCSM Software Engineering Group NCAR
CS 584.
Hybrid Programming with OpenMP and MPI
Department of Computer Science, University of Tennessee, Knoxville
Progress of Interactions Among CCSM and Other Modeling Efforts
HPC User Forum: Back-End Compiler Technology Panel
Parallel Programming in C with MPI and OpenMP
Presentation transcript:

SciDAC CCSM Consortium: Software Engineering Update Patrick Worley Oak Ridge National Laboratory (On behalf of all the consorts) Software Engineering Working Group Meeting 11th Annual CCSM Workshop June 22, 2006 Village at Breckenridge Breckenridge, CO

Project Goals Software –Performance portability –Software engineering (repositories, standardized testing – No Code Left Behind initiative) Model Development –Better algorithms –New physical processes (esp. chemistry, biogeochemistry)

Talk Overview Recent and Ongoing Activities SciDAC-1 performance engineering retrospective SciDAC-2 software engineering plans

SciDAC Software Engineering Activities Since Last Workshop Addition of interactive carbon & sulfur cycles to CCSM Implementation of several ocean ecosystem trace gases through coupler Revised/continuation run of BGC in conjunction with working group Explicit typing of all variables and constants in CAM and CLM New release of MCT (2.2.0) in Dec, Parallel NetCDF testing at ORNL and NCAR

SciDAC Software Engineering Activities Since Last Workshop Single-executable of any components combination (dead model, data model or active model) of CCSM3.0. Introduction of MCT into surface model interfaces in standalone CAM. Support for non-lon/lat grids in physics/dynamics interface and in physics load balancing algorithms. (See talk by Brian Eaton.)

SciDAC Performance Engineering Activities Since Last Workshop CCSM porting and performance optimization on Cray X1E and XT3 –including standalone CAM/CLM, POP, and POP/CICE Additional MCT vectorization for X1E –Next public release will contain all vectorization changes (MCT 2.2.2, late July) Analysis of scalability of FVCAM with respect to horizontal resolution and processor count Performance evaluation and optimization of CAM for large numbers of tracers

SciDAC-1 Performance Engineering Retrospective SciDAC-1 ends on June 30, 2006 Performance engineering activities included –improving serial performance and decreasing parallel overhead –adding compile- and runtime options that improve performance portability –porting and optimizing on new platforms –interface design that allows new dycores to be integrated with CAM efficiently. Next set of slides describe CAM/CLM performance evolution. Similar results can be shown for POP/CICE.

1. Physics data structures –Index range, dimension declaration 2. Physics load balance –Variety of load balancing options, with different communication overheads –SMP-aware load balancing options 3. Communication options –MPI protocols (two-sided and one-sided) –Co-Array Fortran –SHMEM protocols and choice of pt-2-pt implementations or collective communication operators CAM Performance Optimization Options

4. OpenMP parallelism –Instead of some MPI parallelism –In addition to MPI parallelism 5. Aspect ratio of dynamics 2D domain decomposition (FV-only) –1D is latitude-decomposed only –2D is latitude/longitude-decomposed in one part of dynamics, latitude/vertical-decomposed in another part, with remaps to/from the two decompositions during each timestep. CAM Performance Optimization Options

CAM EUL Performance History: to Nov. 2002

CAM EUL Performance History: to May 2004

CAM FV Performance: 1D vs. 2D decomp.

· Performance impact of SciDAC check-ins from March 2004 to April 2006 on the Cray X1E, plotting performance for both named version tag and for immediately preceding version. · Not all check-ins improved performance, nor were expected to - some improved portability, added new performance tuning options, or fixed bugs. CAM Performance History: Vectorization

Current CAM EUL Performance Maximum number of MPI processes is 128 for T85 L26. IBM systems use OpenMP to increase scalability.

Current CAM FV Performance Earth Simulator results courtesy of D. Parks. SP results courtesy of M. Wehner. Maximum number of MPI processes is 960 for 0.5x0.625 L26. IBM systems and Earth Simulator use OpenMP to increase scalability.

Over the last 5 years, SciDAC-funded activities have improved performance of CAM/CLM significantly, by –a factor of 4.5 on IBM p690 cluster for T42 L26; –a factor of at least 2 on current IBM SMP clusters for T85L26 due to parallel algorithm improvements; –a factor of over 3 on the XT3 and X1E for 0.5x0.625 L26 due to the 2D domain decomposition; –a factor of 5 for T85 L26, 1.9x2.5 L26, and 0.5x0.625 L26 on the X1E from vectorization modifications and parallel algorithm improvements; –a factor of at least 4 from porting to new platforms. Summary

Funding of SciDAC-2 proposals will be announced in July, 2006 Two coordinated SciDAC-2 projects proposed that include CCSM software engineering activities: –SEESM: A Scalable and Extensible Earth System Model for Climate Change Science (Science Application: Climate Modeling and Simulation) o Immediate software (and performance) engineering needs, 5 year duration –PENG: Performance Engineering for the Next Generation Community Climate System Model (Science Application Partnership: Computer Science) o More speculative, longer-term performance engineering activities, 3 year duration SciDAC-2 Plans

” Develop, test, and exploit a first generation of Earth system models based upon the CCSM. ” –Similar to current consortium, with Drake and Jones as PIs 1. Extend CCSM to include representations of biological, ecological, chemical,and aerosol processes. 2. Provide the necessary software and modeling expertise to rapidly integrate new methods and model improvements. 3. Pursue the development of innovative methods with the evaluation of these in the coupled context of the CCSM 4. Continue to improve the performance, portability and scalability of the CCSM on available and future computing architectures for use in national and international assessments of climate change. SEESM

Large project, with many software engineeringactivities, including 1. Frameworks for integration and unit testing of new parameterizations 2. Frameworks for integration of new dynamics and components, including Working with SCD and CSEG on HOMME Working with NASA and GFDL on next generation FV 3. Frameworks for model evaluation and performance engineering activities, including 1. Baseline performance instrumentation and analysis, and performance evolution tracking 2. Continued porting and optimization on XT3, X1E, p575 cluster, and on future target platforms SEESM Software Engineering

”Optimize CCSM so that it can simulate tomorrow’s science at the same throughput as CCSM simulates today’s science.” –Loy (ANL), Mirin (LLNL), Worley (ORNL, PI) –Project needs to be responsive to SEESM needs, so research plan is flexible 1. Performance model-driven scalability analysis and performance optimization: CAM, CLM, POP, CSIM/CICE, full CCSM 2. Optimization of CAM in chemistry-dominated regime Physics scalability Physics load balancing Tracer advection optimization PENG

3. CCSM BG/L and BG/P port and optimization Help CSEG with single executable and sequential CCSM Work with SCD on BG/L porting Eliminate remaining global arrays and other sources of memory footprint problems Work with CSEG on introducing parallel I/O 4. Performance analysis and optimization when using new dycores with non-lon/lat grids Help SCD and CSEG with HOMME Help NASA/GFDL with FV 5. Improve scalability and performance of other CCSM components and of full CCSM, as directed by SEESM. PENG