1. Challenges in analyzing a High-Resolution Climate Simulation John M. Dennis, Matthew Woitaszek dennis@ucar.edudennis@ucar.edu, mattheww@ucar.edu mattheww@ucar.edu dennis@ucar.edumattheww@ucar.edu October 26, 2010 October 26-28, 2010GG in Data-Intensive Discovery 1

2. Climate Modeling? Coupled models Atmosphere-Ocean-Sea Ice- Land Challenges of climate: Need many years --> Limits resolution --> Limits parallelism Community Earth System Model (CESM) Formally: Community Climate System Model (CCSM) October 26-28, 2010GG in Data-Intensive Discovery2

3. IPCC & AR5 Intergovernmental Panel on Climate Change (IPCC) “The IPCC assesses the scientific, technical and socio-economical information relevant for the understanding of the risk of human- induced climate change” The Fifth Assessment Report (AR5) CESM plans Control:1 run x 1300 years Historical:40 runs x 55 years Future:180 runs x 95 years 20,600 years of simulation October 26-28, 2010GG in Data-Intensive Discovery3

4. Would you like to Supersize that? October 26-28, 2010GG in Data-Intensive Discovery4

5. Absolutely, But how will it clog up my networks, and fill up my archive? October 26-28, 2010GG in Data-Intensive Discovery5

6. PetaApps project PetaApps project: Interactive Ensemble Kinter, Stan (COLA) Kirtman (U of Miami) Collins, Yelick (Berkley) Bryan, Dennis, Loft, Vertenstein (NCAR) Bitz (U of Washington) Ultra-High resolution climate Explore impact of weather noise on Climate Explore technical/Computer Science issues ~99,000 core Cray XT5 system at NICS [Kraken] Large TG allocation: 35M CPU hours October 26-28, 2010GG in Data-Intensive Discovery6

7. Wendell Sea: Ice thickness (1°) October 26-28, 2010GG in Data-Intensive Discovery7

8. October 26-28, 2010GG in Data-Intensive Discovery8

9. Impact of Supersizing October 26-28, 2010GG in Data-Intensive Discovery9 RegularSupersized #1 (0.5x0.1) Supersized #2 (0.25x0.1) ATM0.2 GB0.9 GB3.6 GB LND0.1 GB0.2 GB0.9 GB ICE0.7 GB72 GB OCN1.2 GB122.5 GB Total564 TB48,357 TB49,187 TB 16x 100x

10. Outline Climate Modeling Production Data challenges: AR5 & IPCC PetaApps simulation Generating the Data Analyzing the Data Conclusions October 26-28, 2010GG in Data-Intensive Discovery10

11. Large scale PetaApps run 155 year control run 0.1° Ocean model [ 3600 x 2400 x 42 ] 0.1° Sea-ice model [3600 x 2400 x 20 ] 0.5° Atmosphere [576 x 384 x 26 ] 0.5° Land [576 x 384 Statistics ~18M CPU hours 5844 cores for 4-5 months ~100 TB of data generated 0.5 to 1 TB per wall clock day generated October 26-28, 2010GG in Data-Intensive Discovery11 4x current production 100x current production

12. Large scale PetaApps run (con’t) Work flow Run on Kraken (NICS) Transfer output from NICS to NCAR (100 – 180 MB/sec sustained) Caused noticeable spikes in TG network traffic Archive on HPSS Data analysis using 55 TB project space at NCAR October 26-28, 2010GG in Data-Intensive Discovery12

13. Issues/challenges with runs Very large variability with I/O performance 2-10x slowdown common 300x slowdown was observed Interference with other jobs? October 26-28, 2010GG in Data-Intensive Discovery13

14. Outline Climate Modeling Production Data challenges: AR5 & IPCC PetaApps simulation Generating the Data Analyzing the Data Conclusions October 26-28, 2010GG in Data-Intensive Discovery14

15. Issues with data analysis Miss-match between structure and form of data needed for analysis Model generated file: Timestep = n :U,V,T,PS,CLDLOWfile.n.nc Timestep = n+step=m:U,V,T,PS,CLDLOWfile.m.nc Analysis: Average T (temperature) for last 40 years Sub-setting and average operation necessary October 26-28, 2010GG in Data-Intensive Discovery15

16. Issues with data analysis (con’t) Slightly different form of input for different analysis/tool Diversity of analysis tools NCL (NCAR Command Language) NCO (NetCDF operators FeretIDLMatlab Cascade of intermediate products “Beware the data cascade, forever your disk will it fill.” -Yoda- October 26-28, 2010GG in Data-Intensive Discovery16

17. Impact of self describing format All variables [~300 varaibles]for single timestep + Matches raw output from model - Does not match needs of analysis tools One variable for single timestep + Matches analysis tool needs + Conceptually simple - Replication of self describing metadata - larger number of files…[~75M files] One variable multiple timesteps + Matches analysis tool needs - Lose of generality October 26-28, 2010GG in Data-Intensive Discovery17

18. Parallelizing Analysis Parallelizing analysis of atmospheric data Goal: Speedup analysis of high-resolution data Analysis no slower then low resolution AMWG diagnostic package NCO operators [calculating statistics] NCL ploting Parallelization using Swift [T. Sines] October 26-28, 2010GG in Data-Intensive Discovery18

19. AMWG Diagnostics 19 Improved Computing = increasing amounts of data o Data generation multi-threaded o Data analysis single-threaded Climate Model (Community Earth System Model) October 26-28, 2010GG in Data-Intensive Discovery

20. Swift Workflow system from U of Chicago (http://www.ci.uchicago.edu/swift/index.php) Allows parallel execution of scripts Relatively easy to use Dependency driven Variety of jobs submission options October 26-28, 2010GG in Data-Intensive Discovery20

21. Issues with Swift Full generality generates “excessive” copying of data Climate workflow has small number of flops relative to input/output data size Copy-in data: 15 sec Calculate: 60 sec Copy-out data: 15 sec Used “un-published” option to perform direct-IO [0-copy] Unresolved issue with deleting out of scope ‘files’ October 26-28, 2010GG in Data-Intensive Discovery21

22. Swift workflow on Dash (8 cores) October 26-28, 2010GG in Data-Intensive Discovery22

23. Data analysis Platforms Twister@NCAR: 8 core, Intel box, GPFS filesystem Dash@SDSC: Compute nodes with SSD disk vSMP node with SSD + ScaleMP GPFS-WANNautilus@NICS 1024 core SGI Ultra Violet GPFS filesystem October 26-28, 2010GG in Data-Intensive Discovery23

24. Execution time for AMWG diagnostics Twister (8 cores) Dash (8 cores) Nautilus (8 cores) 2 deg/10 yrs243 sec78 sec277 sec 1 deg/10 yrs663 sec621 sec 0.5 deg/2 yrs1085 sec October 26-28, 2010GG in Data-Intensive Discovery24 Nice/Interesting result Not so interesting result

25. Performance Issues with AMWG diagnostic Twister@NCAR: No specialized hardware Tiny system Dash@SDSC: vSMP system has network driver problem 7.7x too slower compute node vs vSMP Nautilus@NICS: Apparently excessive Java exec overhead October 26-28, 2010GG in Data-Intensive Discovery25

26. October 26-28, 201026 Conclusions PetaApps project [Interactive Ensembles] 155 year CCSM control run 0.5° ATM & LND + 0.1° OCN ICE 0.5 to 1 TB/day of data to analysis Increased ability to generated data creates analysis bottleneck core Encouraging results for parallel workflow with Swift No transformative benefit yet for expensive hardware ! Overnight After coffee break Nearly Instant No solution yet to data volume growth GG in Data-Intensive Discovery

27. 27Acknowledgements NCAR:NCAR: D. BaileyD. Bailey F. BryanF. Bryan T. CraigT. Craig B. EatonB. Eaton J. Edwards [IBM]J. Edwards [IBM] N. HearnN. Hearn K. LindsayK. Lindsay N. NortonN. Norton M. VertensteinM. Vertenstein COLA:COLA: J. KinterJ. Kinter C. StanC. Stan U. MiamiU. Miami B. KirtmanB. Kirtman U.C. BerkeleyU.C. Berkeley W. CollinsW. Collins K. Yelick (NERSC)K. Yelick (NERSC) U. WashingtonU. Washington C. BitzC. Bitz Grant Support: Grant Support: DOE DE-FC03-97ER62402 [SciDAC] DE-PS02-07ER07-06 [SciDAC] NSF Cooperative Grant NSF01 OCI-0749206 [PetaApps] CNS-0421498 CNS-0420873 CNS-0420985 Computer Allocations: Computer Allocations: TeraGrid TRAC @ NICS DOE INCITE @ NERSC LLNL Grand Challenge Thanks for Assistance: Thanks for Assistance: Cray, NICS, and NERSC NICS: M. Fahey P. Kovatch ANL: R. Jacob R. Loy LANL: E. Hunke P. Jones M. Maltrud LLNL D. Bader D. Ivanova J. McClean (Scripps) A. Mirin ORNL: P. Worley and many more… October 26-28, 2010GG in Data-Intensive Discovery

28. Questions dennis@ucar.edu October 26-28, 2010GG in Data-Intensive Discovery28