IU/GA/ORNL Summary. Most of Monday Wael explaining his script and configuration file design  Intent to follow Don's design as closely as possible  Single.

Slides:



Advertisements
Similar presentations
TeraGrid Deployment Test of Grid Software JP Navarro TeraGrid Software Integration University of Chicago OGF 21 October 19, 2007.
Advertisements

Configuration management
Configuration management
Designing Reusable Frameworks for Test Automation
Earth System Curator Spanning the Gap Between Models and Datasets.
Metadata Development in the Earth System Curator Spanning the Gap Between Models and Datasets Rocky Dunlap, Georgia Tech.
Global MP3 Geoffrey Beers Deborah Ford Mike Quinn Mark Ridao.
ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.
Pages A File System based eScience Workbench Roger Menday Jülich Supercomputing Centre, Germany.
SWIM WEB PORTAL by Dipti Aswath SWIM Meeting ORNL Oct 15-17, 2007.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.
The Role of DANSE at SNS Steve Miller Scientific Computing Group Leader January 22, 2007.
What is workflow?  A workflow is a structured way of defining and automating structures and procedures within an organization. What is workflow management.
Astrophysics, Biology, Climate, Combustion, Fusion, Nanoscience Working Group on Simulation-Driven Applications 10 CS, 10 Sim, 1 VR.
Presented by IBM developer Works ibm.com/developerworks/ 2006 January – April © 2006 IBM Corporation. Making the most of Creating Eclipse plug-ins.
Page 1 Building Reliable Component-based Systems Chapter 18 - A Framework for Integrating Business Applications Chapter 18 A Framework for Integrating.
Enterprise Workflow CPSC 476 Lightening Talk Brenda Griffith/Katie Soto.
An Automated Component-Based Performance Experiment and Modeling Environment Van Bui, Boyana Norris, Lois Curfman McInnes, and Li Li Argonne National Laboratory,
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
Grappa: Grid access portal for physics applications Shava Smallen Extreme! Computing Laboratory Department of Physics Indiana University.
GRAPPA Part of Active Notebook Science Portal project A “notebook” like GRAPPA consists of –Set of ordinary web pages, viewable from any browser –Editable.
CCA Port, Component & Application Build Skeleton Templates “A new script toolkit for generating CCA build skeletons” Torsten Wilde and James Kohl Oak Ridge.
Metadata Creation with the Earth System Modeling Framework Ryan O’Kuinghttons – NESII/CIRES/NOAA Kathy Saint – NESII/CSG July 22, 2014.
Software Engineering Chapter 8 Fall Analysis Extension of use cases, use cases are converted into a more formal description of the system.Extension.
Grids and Portals for VLAB Marlon Pierce Community Grids Lab Indiana University.
1 Evolution of OSG to support virtualization and multi-core applications (Perspective of a Condor Guy) Dan Bradley University of Wisconsin Workshop on.
The Network Performance Advisor J. W. Ferguson NLANR/DAST & NCSA.
Accelerating Scientific Exploration Using Workflow Automation Systems Terence Critchlow (LLNL) Ilkay Altintas (SDSC) Scott Klasky(ORNL) Mladen Vouk (NCSU)
CS 320 Assignment 1 Rewriting the MISC Osystem class to support loading machine language programs at addresses other than 0 1.
Grid Computing Research Lab SUNY Binghamton 1 XCAT-C++: A High Performance Distributed CCA Framework Madhu Govindaraju.
Virtual Data Grid Architecture Ewa Deelman, Ian Foster, Carl Kesselman, Miron Livny.
The european ITM Task Force data structure F. Imbeaux.
Section 11: Implementing Software Restriction Policies and AppLocker What Is a Software Restriction Policy? Creating a Software Restriction Policy Using.
© 2007 UC Regents1 Track 1: Cluster and Grid Computing NBCR Summer Institute Session 1.1: Introduction to Cluster and Grid Computing July 31, 2007 Wilfred.
Tools for collaboration How to share your duck tales…
Wrapping Scientific Applications As Web Services Using The Opal Toolkit Wrapping Scientific Applications As Web Services Using The Opal Toolkit Sriram.
Ames Research CenterDivision 1 Information Power Grid (IPG) Overview Anthony Lisotta Computer Sciences Corporation NASA Ames May 2,
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Page 1© Crown copyright 2004 FLUME Metadata Steve Mullerworth 3 rd -4 th October May 2006.
Scalable Systems Software for Terascale Computer Centers Coordinator: Al Geist Participating Organizations ORNL ANL LBNL.
Portable Infrastructure for the Metafor Metadata System Charlotte Pascoe 1, Gerry Devine 2 1 NCAS-BADC, 2 NCAS-CMS University of Reading PIMMS provides.
March 2004 At A Glance autoProducts is an automated flight dynamics product generation system. It provides a mission flight operations team with the capability.
CCA Common Component Architecture CCA Forum Tutorial Working Group CCA Status and Plans.
STAR Collaboration, July 2004 Grid Collector Wei-Ming Zhang Kent State University John Wu, Alex Sim, Junmin Gu and Arie Shoshani Lawrence Berkeley National.
Dr. Rebhi S. Baraka Advanced Topics in Information Technology (SICT 4310) Department of Computer Science Faculty of Information Technology.
August 2003 At A Glance The IRC is a platform independent, extensible, and adaptive framework that provides robust, interactive, and distributed control.
Distributed Data for Science Workflows Data Architecture Progress Report December 2008.
Interactive Data Analysis on the “Grid” Tech-X/SLAC/PPDG:CS-11 Balamurali Ananthan David Alexander
CS-SWIM Meeting Planning 6 March Short term pre-PSACI Users/advisory panel identified, invited (CS victims needed; Al Sussman, Andrew Seigel) Schedule.
Curator: Gap Analysis (from a schema perspective) Rocky Dunlap Spencer Rugaber Georgia Tech.
Overview of Grid Webservices in Distributed Scientific Applications Dennis Gannon Aleksander Slominski Indiana University Extreme! Lab.
AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.
SPI NIGHTLIES Alex Hodgkins. SPI nightlies  Build and test various software projects each night  Provide a nightlies summary page that displays all.
David Foster LCG Project 12-March-02 Fabric Automation The Challenge of LHC Scale Fabrics LHC Computing Grid Workshop David Foster 12 th March 2002.
Climate-SDM (1) Climate analysis use case –Described by: Marcia Branstetter Use case description –Data obtained from ESG –Using a sequence steps in analysis,
Metadata Development in the Earth System Curator Spanning the Gap Between Models and Datasets Rocky Dunlap, Georgia Tech 5 th GO-ESSP Community Meeting.
Origami: Scientific Distributed Workflow in McIDAS-V Maciek Smuga-Otto, Bruce Flynn (also Bob Knuteson, Ray Garcia) SSEC.
Perfecto We help customers deliver exceptional digital experiences.
Breaking the frontiers of the Grid R. Graciani EGI TF 2012.
Enables businesses achieve greater efficiency by sharing data and processes Shared application data across legal entities – Party, Location, Products,
De Rigueur - Adding Process to Your Business Analytics Environment Diane Hatcher, SAS Institute Inc, Cary, NC Falko Schulz, SAS Institute Australia., Brisbane,
By Jeremy Burdette & Daniel Gottlieb. It is an architecture It is not a technology May not fit all businesses “Service” doesn’t mean Web Service It is.
Automatic launch and tracking the computational simulations with LiFlow and Sumatra Evgeniy Kuklin.
iCrawl – Master Thesis and Hiwi Jobs
iCrawl – Hiwis Jobs and Master Thesis
Execute your Processes
Metadata Development in the Earth System Curator
Overview of Workflows: Why Use Them?
Palestinian Central Bureau of Statistics
Presentation transcript:

IU/GA/ORNL Summary

Most of Monday Wael explaining his script and configuration file design  Intent to follow Don's design as closely as possible  Single configuration file with subsections instead separate one for each component, then global one for overall simulation  Driver is just another component in the design

Monday Framework:  parses config file  instantiates components  Initializes  Call driver.init( ) [the overall driver]  Call driver.step( )  Call driver.finalize( ) Framework also provides services that instantiated components can use...

Monday Framework services are extensive. Some intended to be used by physics components, some only by driver component, but no enforcement of this Widespread agreement (except one ORNL guy)  Need a design document describing this  Documentation in/with code  Complete neophytes (Samantha Foley and Bramley) will help with nagging questions and incorrect comments in scripts

Monday Other discussions:  Timestepping, particularly when combined with non- time stepping components Label with start or end time of the interval? Issue is not going to be settled here, just assure that  Scripting mechanisms will allow whatever is chosen  Data management systems can provide data to physicists  Requires clearly stated conventions if they are chosen (e.g., always timestamp with start of interval)  A data schema implicitly chosen by directory/subdirectory naming convention in current driver/set of components

Monday Portal:  Initially for monitoring and data presentation  Job launch capabilities needed, but first need scripts to launch them...  Data management schemata...

CS Research Goals Running multiple codes as part of a single overarching simulation  Issues of data consistency across components which before did not need to cooperate  How to provide mechanisms to allow signaling and potentially interrupts across components running as separate executables  Running multiple codes with drastically different resource requirements (short term: ignored because near term science not needing this)

CS Research Goals Slow MHD case will require single executable;  move towards integration between components being linked together directly (not necessarily a single executable but fewer executables)  Some need for this likely even now in fast MHD; e.g. TSC is being hobbled by run mode chosen in SWIM

CS Research Goals Overall data management system  Overwhelming even with single code  Needs automated File handling Archiving of results Metadata creation  Needs ability to flexibly allow components to identify new/different files, added to data mgmt seamlessly

CS Research Goals User/developer interaction  Need job monitoring, event notification, accounting  Easy interface to monitoring of computational results ElViz tool of choice here, but need to define interactions with data management system Allow users to identify variables/entities to monitor  Portal launch  When/how will system be for non-SWIM developers

CS Research Goals Comprehensible build systems, automated regression testing  Stage in by addressing key components: component scripts, plasma state, driver component  Exhaustive regression testing unlikely to be possible but some limited regressions on couple of simple cases might be addressible – and would be useful May still be able to provide some tools for limited use by vital components like Plasma State

CS Research Issues Comprehensible build systems, automated regression testing, ctnd  Already facing this with linking of extant components and PS changes. If not full solution, some protocols to help notify when changes made that require component developer's intervention

Collaborations to Consider CS  CCA/TASCS: already have overlap  CPEDS: (Ian Foster, Jenny Schopf): grid logging, MDS, data placement, rehashed GriPHYN stuff. Working with LIGO, Earth Systems Grid (Bramley)  CsSCADS (Ken Kennedy) multicore systems  PERI (Bob Lucas). Pat Worley will seep info to SWIM CS folks as useful.  Earth Systems Grid (Dean Williams) David Bernholdt available as contact point

Collaborations to Consider CS  PDSI: Petascale Data Storage Initiative ?  Scientific Data Management (Ari Shoshani): parallel I/O, some workflow via Kepler (see Scott Klasky)  OSG: Open science grid, Miron Livy (David Schissel)  Keith Jackson (PyGrid) contacts: Dipti and David. Wrap science app as a web service, all Python.

Collaborations to Consider CS  TOPS, ITAPS, APDEC (Phil Colella), other refurbished round-1 SciDACs  VACET: viz and analytics CET (Wes Bethel). Looking for collaborators (Jeembo and Lee) Fusion-centered projects (but CS part mainly)  FACETS (John Cary)  CPES (Scott Klasky) Portal people will keep him in sight, hailing distance. Need to id which portal tech he is wanting to use/develop, look for overlap.

Collaborations to Consider EU people. (Elan Becoulet, replaced by Par Strand) Similar project but w/o significant CS/Math involvment. (Don and Lee triangulate his old director) Topics mentioned include  XML schema for machine descriptions – seems of interest to SWIM researchers as well  High level run management  Using XML schema for specifying a Plasma State entity (specifies the data entries but not necessarily the actual data object), accessors.  More top-down driven specification