Gordon Erlebacher Florida State University

Slides:



Advertisements
Similar presentations
Common Instrument Middleware Architecture and Federation of Instrument Resources for X-ray Crystallography Rick McMullen Indiana University.
Advertisements

Pulan Yu School of Informatics Indiana University Bloomington Web service based Varuna.Net.
Web Services Darshan R. Kapadia Gregor von Laszewski 1http://grid.rit.edu.
VLAB Framework (afternoon) Gordon Erlebacher. Principal IT Investigators University of Minnesota –Prof. David A. Yuen : Dept. of Geology and Geophysics.
1 OBJECTIVES To generate a web-based system enables to assemble model configurations. to submit these configurations on different.
VLab: Collaborative Grid Services and Portals to Support Computational Material Science Mehmet Nacar, Mehmet Aktas, Marlon Pierce, Zhenyu Lu, Gordon Erlebacher,
Connect. Communicate. Collaborate Click to edit Master title style MODULE 1: perfSONAR TECHNICAL OVERVIEW.
VLab: A Collaborative Cyberinfrastructure for Computations of Materials Properties at High Pressures and Temperatures Cesar R. S. da Silva 1 Pedro R. C.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
What is.NET?. The Clients of.NET a) A new generation of connected application b) Microsoft.NET Framework managed execution c) Allows PCs and other smart.
Extensible Scalable Monitoring for Clusters of Computers Eric Anderson U.C. Berkeley Summer 1997 NOW Retreat.
Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.
Federated Hierarchical Filter Grids STTR-funded project with Indiana, Caltech and Deep Web Technologies A Grid infrastructure for Data Analysis Integrates.
Workload Management Massimo Sgaravatto INFN Padova.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Principles for Collaboration Systems Geoffrey Fox Community Grids Laboratory Indiana University Bloomington IN 47404
Talend 5.4 Architecture Adam Pemble Talend Professional Services.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
QCDgrid Technology James Perry, George Beckett, Lorna Smith EPCC, The University Of Edinburgh.
Grid Computing, B. Wilkinson, a.1 Grid Portals.
Apache Airavata GSOC Knowledge and Expertise Computational Resources Scientific Instruments Algorithms and Models Archived Data and Metadata Advanced.
©Ian Sommerville 2006Software Engineering, 8th edition. Chapter 12 Slide 1 Distributed Systems Architectures.
C Copyright © 2009, Oracle. All rights reserved. Appendix C: Service-Oriented Architectures.
Flexibility and user-friendliness of grid portals: the PROGRESS approach Michal Kosiedowski
GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.
Lecture 15 Introduction to Web Services Web Service Applications.
Grids and Portals for VLAB Marlon Pierce Community Grids Lab Indiana University.
3rd June 2004 CDF Grid SAM:Metadata and Middleware Components Mòrag Burgon-Lyon University of Glasgow.
Crystal-25 April The Rising Power of the Web Browser: Douglas du Boulay, Clinton Chee, Romain Quilici, Peter Turner, Mathew Wyatt. Part of a.
ITPA/IMAGE 7-10 May 2007 Software and Hardware Infrastructure for the ITM B.Guillerminet, on behalf of the ITM & ISIP teams (P Strand, F Imbeaux, G Huysmans,
1 DIRAC – LHCb MC production system A.Tsaregorodtsev, CPPM, Marseille For the LHCb Data Management team CHEP, La Jolla 25 March 2003.
The Grid System Design Liu Xiangrui Beijing Institute of Technology.
Advanced Computer Networks Topic 2: Characterization of Distributed Systems.
Grid Architecture William E. Johnston Lawrence Berkeley National Lab and NASA Ames Research Center (These slides are available at grid.lbl.gov/~wej/Grids)
 Apache Airavata Architecture Overview Shameera Rathnayaka Graduate Assistant Science Gateways Group Indiana University 07/27/2015.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
ICCS WSES BOF Discussion. Possible Topics Scientific workflows and Grid infrastructure Utilization of computing resources in scientific workflows; Virtual.
What is SAM-Grid? Job Handling Data Handling Monitoring and Information.
GRID Overview Internet2 Member Meeting Spring 2003 Sandra Redman Information Technology and Systems Center and Information Technology Research Center National.
Ipgdec5-01 Remarks on Web Services PTLIU Laboratory for Community Grids Geoffrey Fox, Marlon Pierce, Shrideep Pallickara, Choonhan Youn Computer Science,
A Data Access Framework for ESMF Model Outputs Roland Schweitzer Steve Hankin Jonathan Callahan Kevin O’Brien Ansley Manke.
6 February 2009 ©2009 Cesare Pautasso | 1 JOpera and XtremWeb-CH in the Virtual EZ-Grid Cesare Pautasso Faculty of Informatics University.
Introduction to Grids By: Fetahi Z. Wuhib [CSD2004-Team19]
David Adams ATLAS DIAL: Distributed Interactive Analysis of Large datasets David Adams BNL August 5, 2002 BNL OMEGA talk.
A Demonstration of Collaborative Web Services and Peer-to-Peer Grids Minjun Wang Department of Electrical Engineering and Computer Science Syracuse University,
Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing
WebFlow High-Level Programming Environment and Visual Authoring Toolkit for HPDC (desktop access to remote resources) Tomasz Haupt Northeast Parallel Architectures.
Korea Workshop May GAE CMS Analysis (Example) Michael Thomas (on behalf of the GAE group)
Overview of Grid Webservices in Distributed Scientific Applications Dennis Gannon Aleksander Slominski Indiana University Extreme! Lab.
Intro to Web Services Dr. John P. Abraham UTPA. What are Web Services? Applications execute across multiple computers on a network.  The machine on which.
AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.
.NET Mobile Application Development XML Web Services.
Partnerships in Innovation: Serving a Networked Nation Grid Technologies: Foundations for Preservation Environments Portals for managing user interactions.
D.Spiga, L.Servoli, L.Faina INFN & University of Perugia CRAB WorkFlow : CRAB: CMS Remote Analysis Builder A CMS specific tool written in python and developed.
Breaking the frontiers of the Grid R. Graciani EGI TF 2012.
SAM architecture EGEE 07 Service Availability Monitor for the LHC experiments Simone Campana, Alessandro Di Girolamo, Nicolò Magini, Patricia Mendez Lorenzo,
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
AMSA TO 4 Advanced Technology for Sensor Clouds 09 May 2012 Anabas Inc. Indiana University.
The Client/Server Database Environment
#01 Client/Server Computing
VLAB Framework (afternoon)
Federated Hierarchical Filter Grids
Distributed Systems Bina Ramamurthy 4/22/2019 B.Ramamurthy.
Gordon Erlebacher Florida State University
Status of Grids for HEP and HENP
New Tools In Education Minjun Wang
Production Manager Tools (New Architecture)
#01 Client/Server Computing
GGF10 Workflow Workshop Summary
Presentation transcript:

Gordon Erlebacher Florida State University Vlab Framework Gordon Erlebacher Florida State University

Typical workflow workflow job submission Elasticity code input data collection ? data analysis and visualization Defines a workflow Phonon code … more of the same …

VLAB design principles All services are available at multiple locations (redundancy) No single point of failure Automatic resource discovery Development of client interfaces and services are decoupled User interaction is through web browsers (and other specialized client software) Assume the system will be (eventually) be used by hundreds of users on thousands of systems Independence of the languages used to write the services

VLAB requirements Collaboration Ease of use Flexibility Fault tolerance (hardware failures) Diversity of tools analysis, visualization, data reduction, storage, etc .

IT team (US component) Gordon Erlebacher (FSU) Dave Yuen (UMN) (client interfaces) Marlon Pierce (IU) (portals) Bijay Karki (LU) (visualization) Stefano Cozzini (Italy) (scheduling) Kirsten Kleese van Dam (UK) (metadata)

Mantle Convection 5003 grid Ra=1010

Mantle Convection Ra = 5 x 108 Streamlines Isosurface Line Integral Convolution Ra = 5 x 108 Produced with Amira

ElasViz

Visualization Algorithm Challenges Strike balance between High- resolution versus interactive speed Batch versus interactivity Intelligent compression Time-dependent datasets Interactive navigation through a Terabyte dataset across a distributed system? How to share datasets from > 10003 time-dependent simulations

Client devices University of Minnesota

Clients Clustering of earthquake events WEB-IS Mantle convection HTML interface to AMIRA Clients Clustering of earthquake events WEB-IS Mantle convection

VLAB components Portals (user front end) Services (tasks) Schedulers (schedule task execution) Data/metadata transport mechanisms Middleware to connect 1-4 based on NaradaBrokering

Message structure (publish-subscribe) Topic Headers Body: SOAP message

Message Flow with SOAP Client specfies remote commands Service executes remote commands Remote call is encoded in soap message Return values are encoded in soap message

Messaging with SOAP through Naradabrokering Client Service Narada- Brokering SOAP SOAP Wrapped SOAP Wrapped SOAP Client Proxy Service Proxy

Grid – Collaboration NaradaBrokering www.naradabrokering.com

Workflows Services have multiple inputs/outputs Connect services together to form a workflow conditionals loops Elasticity code input job submission data collection data analysis and visualization ? Phonon code

Schedulers Take a high level task definition and break it into subtasks Submit these subtasks to appropriate services Return results to the subscribers (role of HPSEARCH)

Grid – Collaboration NaradaBrokering www.naradabrokering.com

Everything is a service Data input Job submission Analysis tools Visualization tools Data storage Scheduling

Services Can be written in any language with an interface to SOAP Executed via remote calls Defined in terms of an interface (WSDL)

Simulation Services Create “smart” input files Job submissions hide details of computational resources, concentrate on requirements for successful run(s)

Storage Services Retrieve files from storage Supercomputer File servers Archival storage Process file (on one or multiple servers) Extract subset Filter subset Transform subset Send processed file for computing or visualization services

Visualization Services Offscreen, hardware-accelerated, remote visualization Wrap existing visualization software Amira VTK Specialized applications (e.g. elasviz -- Karki 2005) Movie creators

Portals Front end for users Reconfigurable Used for job submission visualization data analysis monitoring collaboration

Summary Challenging goals Potential payback is enormous Accomplishments Initial portals to Espresso input Leveraging of existing software for job submission In the process of converting visualization tools to visualization Initial version of Naradabrokering-based framework is in progress

10 months in the future … Implement test workflow data input and submission of many elastic codes collection and visualization of data basic collaboration between two (or more) individuals