SDM Center Experience with Fusion Workflows Norbert Podhorszki, Bertram Ludäscher Department of Computer Science University of California, Davis UC DAVIS.

Slides:

Advertisements

Similar presentations

Provenance GGF18 Kepler/COW+RWS, Kepler/COW+RWS, Bowers, McPhiilips et al. Provenance Management in a COllection-oriented Scientific Workflow.

Advertisements

Workflow automation for processing plasma fusion simulation data Norbert Podhorszki Bertram Ludäscher Scientific Computing Group Oak Ridge National Laboratory.

1 OBJECTIVES To generate a web-based system enables to assemble model configurations. to submit these configurations on different.

SWIM WEB PORTAL by Dipti Aswath SWIM Meeting ORNL Oct 15-17, 2007.

ProActive Task Manager Component for SEGL Parameter Sweeping Natalia Currle-Linde and Wasseim Alzouabi High Performance Computing Center Stuttgart (HLRS),

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.

Technical Aids for Mirroring Sites and China Popular Science Exhibition(CPSE) Chen Hui CNIC,CAS (Computer Network Information Center, Chinese Academy of.

Experiences in Integration of the 'R' System into Kepler Dan Higgins – National Center for Ecological Analysis and Synthesis (NCEAS), UC Santa Barbara.

6th Biennial Ptolemy Miniconference Berkeley, CA May 12, 2005 Distributed Computing in Kepler Ilkay Altintas Lead, Scientific Workflow Automation Technologies.

Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.

Astrophysics, Biology, Climate, Combustion, Fusion, Nanoscience Working Group on Simulation-Driven Applications 10 CS, 10 Sim, 1 VR.

BY: SACHIN SHRIVASTAVA Operating System By : Sachin Shrivastava 1.

The 6713 DSP Starter Kit (DSK) is a low-cost platform which lets customers evaluate and develop applications for the Texas Instruments C67X DSP family.

 Scientific workflow management system based on Ptolemy II  Allows scientists to visually design and execute scientific workflows  Actor-oriented.

®® Microsoft Windows 7 for Power Users Tutorial 8 Troubleshooting Windows 7.

SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, Terence Critchlow, Xiaowen Xin, Bertram.

Operating Systems CS3502 Fall 2014 Dr. Jose M. Garrido

CHAPTER FOUR COMPUTER SOFTWARE.

Introduction to Interactive Media Interactive Media Tools: Software.

Workflow Project Luciano Piccoli Illinois Institute of Technology.

 H.M.BILAL Operating System Concepts.  What is an Operating System?  Mainframe Systems  Desktop Systems  Multiprocessor Systems  Distributed Systems.

Kepler/pPOD: Scientific Workflow and Provenance Support for Assembling the Tree of Life UC DAVIS Department of Computer Science The Kepler/pPOD Team Shawn.

C++ Programming Language Lecture 1 Introduction By Ghada Al-Mashaqbeh The Hashemite University Computer Engineering Department.

Chapter 1: Overview of Workflow Management Dr. Shiyong Lu Department of Computer Science Wayne State University.

1 Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory.

Presented by On the Path to Petascale: Top Challenges to Scientific Discovery Scott A. Klasky NCCS Scientific Computing End-to-End Task Lead.

Chapter 4 Realtime Widely Distributed Instrumention System.

SDM Center End-to-end data management capabilities in the GPSC & CPES SciDAC’s: Achievements and Plans SDM AHM December 11, 2006 Scott A. Klasky End-to-End.

Accelerating Scientific Exploration Using Workflow Automation Systems Terence Critchlow (LLNL) Ilkay Altintas (SDSC) Scott Klasky(ORNL) Mladen Vouk (NCSU)

Silberschatz, Galvin and Gagne  Operating System Concepts Chapter 3: Operating-System Structures System Components Operating System Services.

Fusion-SDM (1) Problem description –Each run in future: ¼ Trillion particles, 10 variables, 8 bytes –Each time step, generated every 60 sec is (250x10^^9)x8x10.

CE Operating Systems Lecture 3 Overview of OS functions and structure.

Grid Architecture William E. Johnston Lawrence Berkeley National Lab and NASA Ames Research Center (These slides are available at grid.lbl.gov/~wej/Grids)

Presented by End-to-End Computing at ORNL Scott A. Klasky Scientific Computing National Center for Computational Sciences In collaboration with Caltech:

 Apache Airavata Architecture Overview Shameera Rathnayaka Graduate Assistant Science Gateways Group Indiana University 07/27/2015.

Silberschatz and Galvin  Operating System Concepts Module 1: Introduction What is an operating system? Simple Batch Systems Multiprogramming.

A Collaborative Framework for Scientific Data Analysis and Visualization Jaliya Ekanayake, Shrideep Pallickara, and Geoffrey Fox Department of Computer.

Your name here SPA: Successes, Status, and Future Directions Terence Critchlow And many, many, others Scientific Process Automation PNNL.

1 Chapter 34 Internet Applications (Telnet, FTP).

University of California, Davis Daniel Zinn 1 University of California, Davis Daniel Zinn 1 Daniel Zinn Bertram Ludäscher University of California at Davis.

1 SRM-Lite: overcoming the firewall barrier for data movement Arie Shoshani Alex Sim Viji Natarajan Lawrence Berkeley National Laboratory SDM Center All-Hands.

Toward interactive visualization in a distributed workflow Steven G. Parker Oscar Barney Ayla Khan Thiago Ize Steven G. Parker Oscar Barney Ayla Khan Thiago.

Application Software System Software.

August 2003 At A Glance The IRC is a platform independent, extensible, and adaptive framework that provides robust, interactive, and distributed control.

Scientific Workflows for the Sensor Web ICT for Earth Observation Anwar Vahed.

Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.

Experiences Running Seismic Hazard Workflows Scott Callaghan Southern California Earthquake Center University of Southern California SC13 Workflow BoF.

A computer contains two major sets of tools, software and hardware. Software is generally divided into Systems software and Applications software. Systems.

1.1 Sandeep TayalCSE Department MAIT 1: Introduction What is an operating system? Simple Batch Systems Multiprogramming Batched Systems Time-Sharing Systems.

Ocean Observatories Initiative OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Scientific Workflows for OOI Ilkay Altintas Charles.

Climate-SDM (1) Climate analysis use case –Described by: Marcia Branstetter Use case description –Data obtained from ESG –Using a sequence steps in analysis,

Workflow-Driven Science using Kepler Ilkay Altintas, PhD San Diego Supercomputer Center, UCSD words.sdsc.edu.

Workflow Management Concepts and Requirements For Scientific Applications.

Large-scale accelerator simulations: Synergia on the Grid turn 1 turn 27 turn 19 turn 16 C++ Synergia Field solver (FFT, multigrid) Field solver (FFT,

Silberschatz and Galvin  Operating System Concepts Module 1: Introduction What is an operating system? Simple Batch Systems Multiprogramming.

Ganga/Dirac Data Management meeting October 2003 Gennady Kuznetsov Production Manager Tools and Ganga (New Architecture)

1 Chapter 1 Basic Structures Of Computers. Computer : Introduction A computer is an electronic machine,devised for performing calculations and controlling.

INTRODUCTION TO XSEDE. INTRODUCTION  Extreme Science and Engineering Discovery Environment (XSEDE)  “most advanced, powerful, and robust collection.

Chapter 1: Introduction What is an Operating System? Mainframe Systems Desktop Systems Multiprocessor Systems Distributed Systems Clustered System Real.

Applied Operating System Concepts

SDM workshop Strawman report History and Progress and Goal.

Operating System Concepts

Chapter 2: Operating-System Structures

Introduction to Operating Systems

A Semantic Type System and Propagation

Overview of Workflows: Why Use Them?

Gordon Erlebacher Florida State University

Chapter 2: Operating-System Structures

Operating System Concepts

Overview of Computer system

Presentation transcript:

SDM Center Experience with Fusion Workflows Norbert Podhorszki, Bertram Ludäscher Department of Computer Science University of California, Davis UC DAVIS Department of Computer Science kepler-project.org

SDM Center New Challenges The CPES project brought new challenges for Kepler and workflow automation people Remote computations, services and tools Long running simulations, large amounts of data One-time-passwords Workflow = “Glue” Scientists only need to connect individual components together Automate tedious processes (logins, copies of data, control, start-stop) Do it reliably Show what is going on

SDM Center Workflows Real-time Monitoring of Simulation: Transfer current data set to a secondary resource Execute short analysis/visualization routines Display result Archival and post-processing Transfer, pack and archive data sets on the fly

SDM Center Kepler actors for CPES Job submission to various resource managers Permanent SSH connection to perform tasks on a remote machine Generalized actors (workflows themselves) for specified tasks: Watch a remote directory for simulation timesteps Execute an external command on a remote machine Tar and archive data in large junks to HPSS Transfer a remote image file and display on screen Control a running SCIRun server remotely Above actors do logging/checkpointing the final workflow can be stopped / restarted

SDM Center Convert Archive Monitor Transfer Archival Workflow Plasma physics simulation on 2048 processors on (LBL) Gyrokinetic Toroidal Code (GTC) to study energy transport in fusion devices (plasma microturbulence) Generating 800GB of data (3000 files, 6000 timesteps, 267MB/timestep), 30+ hour simulation run Under workflow control: Monitor (watch) simulation progress (via remote scripts) Transfer from NERSC to ORNL concurrently with the simulation run Convert each file to HDF5 file Archive files to 4GB chunks into HPSS

SDM Center Monitoring Workflow

SDM Center Future Plans Currently we have specialized actors that should be generalized for other disciplines and systems “watching for” simulation output safe and robust transfer, recovery from failure archiving to different MSS, with different security policies, robust to failures and maintenance periods Next workflow is cyclic, not just streaming couple two simulations on two resources, transfer data and control between them use local job manager for code execution What about provenance management? main reason to use scientific workflow system e.g. in bioinformatics workflows – needed for debugging runs, interpreting results, etc.

SDM Center There is more, e.g., how to get from messy to neat & reusable designs? Author: Tim McPhillips, UC Davis

SDM Center The Answer (YMMV) Collection-Oriented Modeling & Design (COMAD) embrace an assembly line metaphor data = tagged nested collections e.g. represented as flattened, pipelined token streams: