Presented by End-to-End Computing at ORNL Scott A. Klasky Scientific Computing National Center for Computational Sciences In collaboration with Caltech:

Slides:



Advertisements
Similar presentations
Network II.5 simulator ..
Advertisements

NGAS – The Next Generation Archive System Jens Knudstrup NGAS The Next Generation Archive System.
Hasan Abbasi Matthew Wolf Jay Lofstead Fang Zheng Greg Eisenhauer Karsten Schwan Analyzing large data sets quickly Scott Klasky Ron Oldfield Norbert Podhorszki.
MapReduce Online Created by: Rajesh Gadipuuri Modified by: Ying Lu.
Workflow automation for processing plasma fusion simulation data Norbert Podhorszki Bertram Ludäscher Scientific Computing Group Oak Ridge National Laboratory.
1 OBJECTIVES To generate a web-based system enables to assemble model configurations. to submit these configurations on different.
1 Coven a Framework for High Performance Problem Solving Environments Nathan A. DeBardeleben Walter B. Ligon III Sourabh Pandit Dan C. Stanzione Jr. Parallel.
SWIM WEB PORTAL by Dipti Aswath SWIM Meeting ORNL Oct 15-17, 2007.
ProActive Task Manager Component for SEGL Parameter Sweeping Natalia Currle-Linde and Wasseim Alzouabi High Performance Computing Center Stuttgart (HLRS),
Summary Role of Software (1 slide) ARCS Software Architecture (4 slides) SNS -- Caltech Interactions (3 slides)
Distributed Processing, Client/Server, and Clusters
Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.
Astrophysics, Biology, Climate, Combustion, Fusion, Nanoscience Working Group on Simulation-Driven Applications 10 CS, 10 Sim, 1 VR.
Slide 1 of 9 Presenting 24x7 Scheduler The art of computer automation Press PageDown key or click to advance.
GMD German National Research Center for Information Technology Innovation through Research Jörg M. Haake Applying Collaborative Open Hypermedia.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
Apache Airavata GSOC Knowledge and Expertise Computational Resources Scientific Instruments Algorithms and Models Archived Data and Metadata Advanced.
Data Acquisition at the NSLS II Leo Dalesio, (NSLS II control group) Oct 22, 2014 (not 2010)
M. Taimoor Khan * Java Server Pages (JSP) is a server-side programming technology that enables the creation of dynamic,
Research on cloud computing application in the peer-to-peer based video-on-demand systems Speaker : 吳靖緯 MA0G rd International Workshop.
Scott Klasky SDM Integration Framework in the Hurricane of Data SDM AHM 10/07/2008 Scott A. Klasky ANL: Ross CalTech: Cummings GT: Abbasi,
Christopher Jeffers August 2012
SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, Terence Critchlow, Xiaowen Xin, Bertram.
Alok 1Northwestern University Access Patterns, Metadata, and Performance Alok Choudhary and Wei-Keng Liao Department of ECE,
Presented by XGC: Gyrokinetic Particle Simulation of Edge Plasma CPES Team Physics and Applied Math Computational Science.
M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.
AUTOBUILD Build and Deployment Automation Solution.
Copyright © 2008 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Chapter 3: Operating Systems Computer Science: An Overview Tenth Edition.
9 Chapter Nine Compiled Web Server Programs. 9 Chapter Objectives Learn about Common Gateway Interface (CGI) Create CGI programs that generate dynamic.
Extending Petascale I/O with Data Services Hasan Abbasi Karsten Schwan Matthew Wolf Jay Lofstead Scott Klasky (ORNL)
Introduction to Apache OODT Yang Li Mar 9, What is OODT Object Oriented Data Technology Science data management Archiving Systems that span scientific.
Introduction GOALS:  To improve the Quality of Service (QoS) for the JBI platform and endpoints  E.g., latency, fault tolerance, scalability, graceful.
Presented by On the Path to Petascale: Top Challenges to Scientific Discovery Scott A. Klasky NCCS Scientific Computing End-to-End Task Lead.
CHAPTER TEN AUTHORING.
Chapter 4 Realtime Widely Distributed Instrumention System.
Accelerating Scientific Exploration Using Workflow Automation Systems Terence Critchlow (LLNL) Ilkay Altintas (SDSC) Scott Klasky(ORNL) Mladen Vouk (NCSU)
Introduction to the Adapter Server Rob Mace June, 2008.
Frontiers in Massive Data Analysis Chapter 3.  Difficult to include data from multiple sources  Each organization develops a unique way of representing.
GEM Portal and SERVOGrid for Earthquake Science PTLIU Laboratory for Community Grids Geoffrey Fox, Marlon Pierce Computer Science, Informatics, Physics.
Opportunities in Parallel I/O for Scientific Data Management Rajeev Thakur and Rob Ross Mathematics and Computer Science Division Argonne National Laboratory.
Fusion-SDM (1) Problem description –Each run in future: ¼ Trillion particles, 10 variables, 8 bytes –Each time step, generated every 60 sec is (250x10^^9)x8x10.
_______________________________________________________________CMAQ Libraries and Utilities ___________________________________________________Community.
 Apache Airavata Architecture Overview Shameera Rathnayaka Graduate Assistant Science Gateways Group Indiana University 07/27/2015.
Framework for MDO Studies Amitay Isaacs Center for Aerospace System Design and Engineering IIT Bombay.
CCGrid 2014 Improving I/O Throughput of Scientific Applications using Transparent Parallel Compression Tekin Bicer, Jian Yin and Gagan Agrawal Ohio State.
Presented by Scientific Annotation Middleware Software infrastructure to support rich scientific records and the processes that produce them Jens Schwidder.
Presented by Scientific Data Management Center Nagiza F. Samatova Network and Cluster Computing Computer Sciences and Mathematics Division.
Your name here SPA: Successes, Status, and Future Directions Terence Critchlow And many, many, others Scientific Process Automation PNNL.
Jay Lofstead Input/Output APIs and Data Organization for High Performance Scientific Computing November.
System Center Lesson 4: Overview of System Center 2012 Components System Center 2012 Private Cloud Components VMM Overview App Controller Overview.
1 BBN Technologies Quality Objects (QuO): Adaptive Management and Control Middleware for End-to-End QoS Craig Rodrigues, Joseph P. Loyall, Richard E. Schantz.
INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.
Marcelo R.N. Mendes. What is FINCoS? A set of tools for data generation, load submission, and performance measurement of CEP systems; Main Characteristics:
Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.
AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.
The Gateway Computational Web Portal Marlon Pierce Indiana University March 15, 2002.
An Architectural Approach to Managing Data in Transit Micah Beck Director & Associate Professor Logistical Computing and Internetworking Lab Computer Science.
SDM Center Experience with Fusion Workflows Norbert Podhorszki, Bertram Ludäscher Department of Computer Science University of California, Davis UC DAVIS.
Addressing Data Compatibility on Programmable Network Platforms Ada Gavrilovska, Karsten Schwan College of Computing Georgia Tech.
Simulation Production System Science Advisory Committee Meeting UW-Madison March 1 st -2 nd 2007 Juan Carlos Díaz Vélez.
LIOProf: Exposing Lustre File System Behavior for I/O Middleware
Breaking the frontiers of the Grid R. Graciani EGI TF 2012.
1 Copyright © 2008, Oracle. All rights reserved. Repository Basics.
ADIOS – adiosapi.org1 Jay Lofstead Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS) Jay Lofstead (GT),
Onedata Eventually Consistent Virtual Filesystem for Multi-Cloud Infrastructures Michał Orzechowski (CYFRONET AGH)
OpenPBS – Distributed Workload Management System
Threads vs. Events SEDA – An Event Model 5204 – Operating Systems.
SDM workshop Strawman report History and Progress and Goal.
Overview of Workflows: Why Use Them?
MapReduce: Simplified Data Processing on Large Clusters
Presentation transcript:

Presented by End-to-End Computing at ORNL Scott A. Klasky Scientific Computing National Center for Computational Sciences In collaboration with Caltech: J. Cummings Georgia Tech: K. Schwan, M. Wolf, H. Abbasi, G. Lofstead LBNL: A. Shoshani NCSU: M. Vouk, J. Ligon, P. Mouallem, M. Nagappan ORNL: R. Barreto, C. Jin, S. Hodson PPPL: S. Ethier Rutgers: M. Parashar, V. Bhat, C. Docan Utah: S. Parker, A. Kahn UC Davis: N. Podhorszki UTK: M. Beck, S. Sellers, Y. Ding Vanderbilt: M. DeVries

2 Klasky_End--to-End_SC07 Massively parallel simulation Petascale data workspace

3 Klasky_End--to-End_SC07 The End-to-End framework Metadata-rich output from components SRMLNAsync. NXM streaming Workflow Automation Applied Math Data Monitoring CCA Dashboard Applications (GTC, GTC-S, XGC, M3D, S3D, Chimera) Visualization Analysis

4 Klasky_End--to-End_SC07 Unified APIs for MPI/AIO (Lofstead)  Single, simplified API capable of supporting various low- level implementations (MPI-IO, HDF5, POSIX, asynchronous methods)  Transmits buffered data only during non-communication phases of HPC codes  External XML configuration file describing data formats and the storage approach and parameters for each  Implements best practices for underlying implementations  Adds data tagging and annotation  Enables complex inline processing with DataTap and DART (off compute node)  e.g., custom compression, filtering, transformation, multiple output organizations from single write, real-time analysis DART DataTap POSIX... AIO API HPC Codes HDF-5 MPI-IO

5 Klasky_End--to-End_SC07 Asynchronous I/O API usage example XML configuration file: … … srv=ewok001.ccs.ornl.gov Fortran90 code : ! initialize the system loading the configuration file aio_init (100) ! 100 MB of buffer ! retrieve a declared type for writing aio_get_type (t1, “restart”) ! open a write path for that type aio_open (h1, t1, “restart.n1”) ! write the data items aio_write (h1, “mi”, mi) aio_write (h1, “zion”, zion) … ! write more variables ! commit the writes for asynchronous transmission aio_close (h1) … ! do more work ! shutdown the system at the end of my run aio_finalize ()

6 Klasky_End--to-End_SC07 Asynchronous petascale I/O for data in transit  High-performance I/O  Asynchronous  Managed buffers  Respect firewall constraints  Enable dynamic control with flexible MxN operations  Transform using shared-space framework (Seine) User applications Seine coupling framework interface Other program paradigms Shared space management Load balancing Directory layerStorage layer Communication layer (buffer management) Operating system

7 Klasky_End--to-End_SC07  Adding a DataTap to an HPC code reduces I/O overhead tremendously.  Rather than writing directly, the client HPC code notifies the DataTap server to read data asynchronously when resources are available.  The DataTap server scheduler manages data transfer to reduce I/O impact:  Guarantees available memory and egress bandwidth consumption does not exceed a user specified limit. Other considerations, such as CPU usage, are also possible.  The DataTap server is the gateway to I/O graph processing for storage to disk or additional processing--even on another cluster. Lightweight data extraction and processing using a DataTap and I/O Graph DataTap Servers (service nodes) …… I/O Graph & Disk Storage … HPC Code (compute nodes) HPC Code (compute nodes)

8 Klasky_End--to-End_SC07 Data streaming and in-transit processing  Requirements  High-throughput, low- latency transport with minimized overheads  Adapt to application and network state  Schedule and manage in-transit processing  Approach – Cooperative self-management  Application-level data streaming  Proactive management using online control and policies  In-transit data manipulation  Quick, opportunistic, and reactive In-Transit node Data Consumer or Sink Data Producer ProcessBuffer Forward In-Transit Level Reactive Management LLC Controller Service Manager Application-Level Proactive Management Buffer Data Blocks Data Consumer Data Transfer Coupling  Experimental evaluation  ORNL and NERSC -> Rutgers -> PPPL  Adaptive in-transit processing reduced idle time from 40% to 2%.  Improved end-to-end data streaming –  Reduced data loss.  Improved data quality at sink. Simulatio n

9 Klasky_End--to-End_SC07 Workflow automation  Automate the data processing pipeline  Transfer of simulation output to the e2e system, execution of conversion routines, image creation, data archiving  And the code coupling pipeline  Check linear stability and compute new equilibrium on the e2e system  Run crash simulation if needed  Using the Kepler workflow system  Requirements for Petascale computing – Easy to use– Parallel processing – Dashboard front-end– Robustness – Autonomic– Configurability

10 Klasky_End--to-End_SC07 CPES workflow automation  NetCDF files  Transfer files to e2e system on-the-fly  Generate images using grace library  Archive NetCDF files at the end of simulation  Proprietary binary files (BP)  Transfer to e2e system using bbcp  Convert to HDF5 format  Generate images with AVS/Express (running as service)  Archive HDF5 files in large chunks to HPSS  M3D coupling data  Transfer to end-to-end system  Execute M3D : compute new equilibrium  Transfer back the new equilibrium to XGC  Execute ELITE : compute growth rate, test linear stability  Execute M3D-MPP : to study unstable states (ELM crash)

11 Klasky_End--to-End_SC07 Kepler components for CPES Watch simulation output Execute remote command Archive stream of files in large chunks

12 Klasky_End--to-End_SC07 Kepler workflow for CPES code coupling Combines data from AVS/Express, Gnuplot, IDL, Xmgrac. Allows us to monitor the weak code coupling of XGC0 (Jaguar) to M3D-OMP (ewok) to ELITE (ewok) to M3D-MPP (ewok).

13 Klasky_End--to-End_SC07 S3D workflow automation  Restart/analysis files  Transfer files to e2e system  Morph files using existing utility  Archive files to HPSS  Transfer files to Sandia  NetCDF files  Transfer files to e2e system on- the-fly  Generate images using grace library and AVS/Express  Send images to dashboard system  Min/max log files  Transfer to e2e system at short intervals  Plot with gnuplot  Send to dashboard for real-time monitoring

14 Klasky_End--to-End_SC07 S3D graphs on the dashboard Graphs are generated and updated as the model is running.

15 Klasky_End--to-End_SC07  Proprietary binary files (BP)  Convert to HDF5 format  Generate images  With custom processing programs (bp2h5-png)  With connection to VisIt  Archive files in HPSS  Key actor: ProcessFile (N. Podhorszki)  Check-perform-record checkpoint pattern  Operates on a stream of operands (remote files) in a pipeline of processors  Logging and error logging of operations provided within the component; just configure for location of log files GTC workflow automation

16 Klasky_End--to-End_SC07 Scientist with limited knowledge about dashboard technology  Simulation monitoring involves the successful integration of several sub-tasks:  Monitoring of DOE machines  Visualization of simulation data: graphs, movies, provenance data, input files etc.  Database integration and High Performance Storage System:  Annotating images and runs taking e-notes and maintaining an e-book  High-speed data delivery services  Workflow system that pieces these tasks together Simulation monitoring  Check machine status and queues  Submit job through dashboard and workflow  Visualize simulation data from provenance information to output files and graphs  Analyze data  Keep notes on runs  Download selected information or move to specific storage  Interact with workflow

17 Klasky_End--to-End_SC07  Back end: shell scripts, python scripts and PHP  Machine queues command  Users’ personal information  Services to display and manipulate data before display  Dynamic front end:  Machine monitoring: standard web technology + Ajax  Simulation monitoring: Flash  Storage: MySQL (queue-info, min- max data, users’ notes…) Machine and job monitoring

18 Klasky_End--to-End_SC07 Provenance tracking  Collects data from the different components of the W/F.  Provides the scientist easy access to the data collected through a single interface.  APIs have been created in Kepler to support real-time provenance capture of simulations running on leadership- class machines. Manticore Orbitty (Python adapter) Orbitty (Python adapter) SSH XMLRPC Scientists Super- Computer Super- Computer MySQL DB XMLRPC (Zope) HTTP (Apache)

19 Klasky_End--to-End_SC07 Ewok cluster Ewok cluster Depots NYU PPPL UCI MIT Portals Directory server Logistical networking: High-performance ubiquitous and transparent data access over the WAN Jaguar Cray XT4 Jaguar Cray XT4

20 Klasky_End--to-End_SC07 Data distribution via logistical networking and LoDN  Logistical Distribution Network (LoDN) directory service adapted to run in NCCS environment:  User control of automated data mirroring to collaborative sites on per file or (recursive) per folder basis.  Firewall constraints require mirroring of metadata to outside server.  User libraries enable program access to LN storage through standard interfaces (POSIX, HDF5, NetCDF).  User control over data placement and status monitoring will be integrated with dashboard.  Download of data to local system for offline access.

21 Klasky_End--to-End_SC07 Contact Scott A. Klasky Lead, End-to-End Solutions Scientific Computing National Center for Computational Sciences (865) Klasky_End--to-End_SC07