Virtual Geophysics Laboratory Scientific workflows exploiting the cloud Ryan Fraser, Terry Rankine, Lesley Wyborn, Joshua Vote, Ben Evans... Presented.

Slides:

Advertisements

Similar presentations

Andrew McNab - Manchester HEP - 2 May 2002 Testbed and Authorisation EU DataGrid Testbed 1 Job Lifecycle Software releases Authorisation at your site Grid/Web.

Advertisements

Introduction to FAIR-TRADE Todd King 1, Raymond Walker 1,2, Lee Bargatze 1, Deborah McGuinness 3, Everett Toews 4, John Shillington 4, Robert Bentley 5,

ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.

MINERALS DOWN UNDER Using Spatial Data Infrastructure (SDI) to: enable interoperable data exchange & usage to deliver corporate decisions Ryan Fraser 19.

APAC Grid Geosciences Project: Linking Academia, Industry and Government to HPC Robert Woodcock, CSIRO and pmd*CRC Ryan Fraser, IVEC and Curtin.

Ryan Fraser (CSIRO), Lesley Wyborn (GA), Richard Chopping (GA), Terry Rankine (CSIRO), Robert Woodcock (CSIRO) MINERALS DOWN UNDER Virtual Geophysics Laboratory.

Solar and STP Physics with AstroGrid 1. Mullard Space Science Laboratory, University College London. 2. School of Physics and Astronomy, University of.

Management Framework for Amazon EC2 Speaker: Frank Bitzer

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.

FI-WARE – Future Internet Core Platform FI-WARE Cloud Hosting July 2011 High-level description.

Virtual Geophysics Laboratory (VGL) VGL v1.1 Launch Ryan Fraser, Terry Rankine, Joshua Vote, Lesley Wyborn, Ben Evans, Robert Woodcock February 2013 CSIRO.

Virtual Geophysics Laboratory (VGL) VGL v1.2 NeCTAR Project Close R.Fraser, T.Rankine, J.Vote, L.Wyborn, B.Evans, R.Woodcock, C.Kemp July 2013 CSIRO |

The iPlant Collaborative Community Cyberinfrastructure for Life Science Tools and Services Workshop Discovery Environment Overview.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM Australian Earth Science Research Information Infrastructure Dr Robert Woodcock CSIRO.

CCSM Portal/ESG/ESGC Integration (a PY5 GIG project) Lan Zhao, Carol X. Song Rosen Center for Advanced Computing Purdue University With contributions by:

Virtual Geophysics Laboratory Exploiting the Cloud and Empowering Geophysicists Ryan Fraser, Terry Rankine, Lesley Wyborn, Joshua Vote, Ben Evans. Presented.

Ryan Fraser, Nicholas Carr Connecting RDSI, NeCTAR and ANDS via Provenance - VHIRL.

National Earth Science Infrastructure Program AuScope Limited Headquarters School of Earth Sciences University of Melbourne Victoria 3010 Tel

Planning for Arctic GIS and Geographic Information Infrastructure Sponsored by the Arctic Research Support and Logistics Program 30 October 2003 Seattle,

Linking AuScope to the broader minerals industry value chain Jonathan Law, Robert Woodcock, Ryan Fraser, Terry Rankine, Guillaume Duclaux.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM The Spatial Information Services Stack – infrastructure for the AuScope Community Earth.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM Virtual Geophysics Laboratory (VGL): Scientific workflows Exploiting the Cloud Josh.

National Spatial Data Infrastructure The Spatial Information Services Stack Dr Robert Woodcock.

IPlant cyberifrastructure to support ecological modeling Presented at the Species Distribution Modeling Group at the American Museum of Natural History.

Virtual Laboratories VGL and Friends R.Fraser, T.Rankine, J.Vote, R.Woodcock AuScope Grid Roadshow 2014 CSIRO | MINERAL RESOURCES FLAGSHIP.

Future Directions MINERALS DOWN UNDER SISS – Spatial Information Services Stack Ryan Fraser| Project Lead 20 th March 2012.

Supported by EU projects 12/12/2013 Athens, Greece Open Data in Agriculture Hands-on with data infrastructures that can power your agricultural data products.

AuScope Spatial Data Infrastructure Supporting Earth Science Dr Robert Woodcock CSIRO.

Interoperability Grids, Clouds and Collaboratories Ruth Pordes Executive Director Open Science Grid, Fermilab.

Styx Grid Services: Lightweight, easy-to-use middleware for e-Science Jon Blower Keith Haines Reading e-Science Centre, ESSC, University of Reading, RG6.

Data and storage services on the NGS Mike Mineter Training Outreach and Education

Making Geological Map Data for the Earth Accessible OneGeology: assisting Geological Surveys worldwide to interoperate seamlessly on the Next Generation.

Technical Session Virtual Geophysics Laboratory MINERAL RESOURCES Josh Vote | Software Developer September 2014.

Large Scale Nuclear Physics Calculations in a Workflow Environment and Data Provenance Capturing Fang Liu and Masha Sosonkina Scalable Computing Lab, USDOE.

DataONE: Preserving Data and Enabling Data-Intensive Biological and Environmental Research Bob Cook Environmental Sciences Division Oak Ridge National.

Interoperability from the e-Science Perspective Yannis Ioannidis Univ. Of Athens and ATHENA Research Center

Virtual Geophysics Laboratory (VGL) VGL v1.2 NeCTAR Project Close Ryan Fraser, Terry Rankine, Joshua Vote, Lesley Wyborn, Ben Evans, Robert Woodcock July.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM Virtual Geophysics Laboratory (VGL): Scientific workflows Exploiting the Cloud Josh.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM The NCRIS AuScope Community Earth Model Bruce Simons.

| nectar.org.au NECTAR TRAINING Module 2 Virtual Laboratories and eResearch Tools.

AuScope Roadshow 2014 AuScope Andrew Rodger| Geoscience Analytical Platforms September 2014 MINERAL RESOURCES FLAGSHIP (MRF)

AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.

AN ORGANISATION FOR A NATIONAL EARTH SCIENCE INFRASTRUCTURE PROGRAM “Building Clients for the AuScope Spatial Information Services Stack (SiSS)” AuScope.

IPlant Collaborative Tools and Services Workshop iPlant Collaborative Tools and Services Workshop Overview of the iPlant Discovery Environment.

IPlant Collaborative Tools and Services Workshop Overview of the iPlant Discovery Environment Sriram Srinivasan.

The Global Scene Wouter Los University of Amsterdam The Netherlands.

Tutorial on Science Gateways, Roma, Catania Science Gateway Framework Motivations, architecture, features Riccardo Rotondo.

Data Management: Data Processing Types of Data Processing at USGS There are several ways to classify Data Processing activities at USGS, and here are some.

1 This Changes Everything: Accelerating Scientific Discovery through High Performance Digital Infrastructure CANARIE’s Research Software.

Nci.org.au © National Computational Infrastructure 2016 Virtual Laboratories in Australia Lesley Wyborn (NCI) With contributions from:

Enabling Digital Earth by focussing on ‘accessibility’ rather than ‘delivery’. Ryan Fraser CSIRO.

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.

CyVerse Workshop Discovery Environment Overview. Welcome to the Discovery Environment A Simple Interface to Hundreds of Bioinformatics Apps, Powerful.

AMSA TO 4 Advanced Technology for Sensor Clouds 09 May 2012 Anabas Inc. Indiana University.

Making FAAM Flights Discoverable

Accessing the VI-SEEM infrastructure

ISDA + OpenStack Rob Kooper.

Tools and Services Workshop

CyVerse Discovery Environment

Docker Birthday #3.

Workflows in archaeology & heritage sciences

University of Technology

VHIRL: Virtual Hazard Impact and Risk Laboratory – “A reuse story”

LifeWatch Cloud Computing Workshop

Virtual Geophysics Laboratory (VGL): Exploiting the Cloud and HPC

Development Goals for Year 2

Welcome to Microsoft Azure for Research Training!

EOSDIS Approach to Data Services in the Cloud

EOSC-hub Contribution to the EOSC WGs

Roadmap and short term activities on interoperability of Data Cubes

Presentation transcript:

Virtual Geophysics Laboratory Scientific workflows exploiting the cloud Ryan Fraser, Terry Rankine, Lesley Wyborn, Joshua Vote, Ben Evans... Presented by Robert Woodcock October 2012 CSIRO | MINERALS DOWN UNDER FLAGSHIP

Gather data, process it, publish results Simple, isn’t it? bedrock surficial mineral geochemical geochronologic hyrdrogeological Geo-information geophysical knowledge data

Recognise the complete picture....

Virtual Geophysics Laboratory | Robert Woodcock 4 | Introducing The Virtual Geophysics Laboratory

Data discovery Virtual Geophysics Laboratory | Robert Woodcock 5 | Layers discovered via remote registries Layers consist of numerous remote data services

Data discovery Virtual Geophysics Laboratory | Robert Woodcock 6 | Some data services support subsetting Some data services support reformatting e.g. CSV, NetCDF, GeoTIFF

Data discovery Virtual Geophysics Laboratory | Robert Woodcock 7 | Some data is only registered with flat files

Powered by the Spatial Information Services Stack Common Platform MarineEnvironment, Water Groundwater Geology Geophysics

What just happened? Virtual Geophysics Laboratory | Robert Woodcock 9 |

Data processing Virtual Geophysics Laboratory | Robert Woodcock 10 | A variety of different scientific codes are already available in the form of “Toolboxes”

Data processing Virtual Geophysics Laboratory | Robert Woodcock 11 | Further input files can be uploaded. Input files are passed directly into the cloud

Data processing Virtual Geophysics Laboratory | Robert Woodcock 12 | The steps so far have been building an environment to run a processing script...or build from existing templates Either write your own...

What just happened? Virtual Geophysics Laboratory | Robert Woodcock 13 | Processing script/ small input files uploaded Start processing Download big data sets Perform data processing Download Job Script/user input files Upload processing results

Managing results - provenance Presentation title | Presenter name 14 | All of a job’s outputs are also accessible Each job has a lifecycle that can be managed Successful jobs can have their entire process captured in a ISO ‘provenance record’

What just happened? Virtual Geophysics Laboratory | Robert Woodcock 15 | What’s the processing status? What are the job input/outputs? Publish the job’s process and results Cloud storage will persist the final artefacts

Still under construction Virtual Geophysics Laboratory | Robert Woodcock 16 | Courtesy -

What’s left? BYO cloud allocation Users should be able to authorise VGL start jobs using their compute/storage resources. Confidential Data How do you get access to ‘restricted data’ in a secure manner? Where can you store the results? (geographical restrictions) Massive Horizontal Scaling What’s the best way to set up a truly elastic pool of CPU’s for jobs to utilise? A Common Processing Services Platform – SISS like? Virtual Geophysics Laboratory | Robert Woodcock 17 |

Sustainable Resources Policy Societal Need Virtual Solid Earth Sciences Laboratory Environment V. Lab Integrated Virtual Labs Virtual Geophysical Laboratory Virtual Core Laboratory Virtual Geodesy Laboratory Virtual Climate Laboratory Virtual Water Laboratory Virtual Laboratories Geophysics Borehole data Geodesy Climate Modelling Water Monitoring Virtual Libraries Processing Services Data Middleware Processing Services Data Middleware Processing Services Data Middleware Processing Services Data Middleware Processing Services Data Middleware Modelling & analytic tools