Virtual Geophysics Laboratory (VGL) VGL v1.1 Launch Ryan Fraser, Terry Rankine, Joshua Vote, Lesley Wyborn, Ben Evans, Robert Woodcock February 2013 CSIRO | MINERALS DOWN UNDER FLAGSHIP

Gather data, process it, publish results Simple, isn’t it? bedrock surficial mineral geochemical geochronologic hyrdrogeological Geo-information geophysical knowledge data Virtual Geophysics Laboratory v1.1 | Ryan Fraser 2 |

Hardcopy of data SSH Client Maths Package Proprietary Package Let’s map it out for Geophysics … Transform to a regular grid Crop data to area of interest Reformat data for processing Upload data to HPC resource Configure job and start processing Download results Get handed field data Visualise data Virtual Geophysics Laboratory v1.1 | Ryan Fraser 3 |

The Past Compile raw data using proprietary FORTRAN Also use other proprietary software Transform to a regular grid using more software MATLAB, Intrepid, ER Mapper, ESRI ArcGIS, QGIS, etc Crop data spatially eg: everything in Victoria Transform data into a file format that can be read by proprietary scientific code This is usually done with some handwritten python or c There is no version control, code is often rewritten / redone Upload data to HPC Manually enter input parameters/start job Where did my results go, how do I visualise? Virtual Geophysics Laboratory v1.1 | Ryan Fraser 4 |

There seems to be a problem… Reproducibility – there is none What was the input of your model? What transformations occurred? It’s a manual process Time consuming Error prone Expensive Licensing costs Effort/time Virtual Geophysics Laboratory v1.1 | Ryan Fraser 5 |

Scientific workflow – Virtual Geophysics Laboratory (VGL) Automated - Simplify access and integration Reproducible process Open access (less expense) Allow scientists to focus on the science and not the tech Exploiting the Cloud and HPC Operational or education tool Automates and massively expands geophysics computational capacity via the Cloud Virtual Geophysics Laboratory v1.1 | Ryan Fraser 6 |

V(what)GL VEGL – Virtual Exploration Geophysics Laboratory One primary science collaboration One primary workflow One collection of geophysical data sets VGL - Virtual Geophysics Laboratory NeCTAR funded activity Collaboration with multiple partners (CSIRO, NCI, GA, UQ, Monash, ANU) Supporting multiple workflows New data types and collections New Use – Not just exploration. Done Virtual Geophysics Laboratory v1.1 | Ryan Fraser 7 |

Hardcopy of data SSH Client Maths Package Proprietary Package Let’s map it out… Transform to a regular grid Crop data to area of interest Reformat data for processing Upload data to HPC resource Configure job and start processing Download results Get handed field data Visualise data Virtual Geophysics Laboratory v1.1 | Ryan Fraser 8 |

…to this Virtual Geophysics Laboratory Build “science” from existing libraries Run job Collect and publish results Discover raw data Select spatial bounds Virtual Geophysics Laboratory v1.1 | Ryan Fraser 9 |

Virtual Geophysics Laboratory v1.1 | Ryan Fraser 10 | Introducing The Virtual Geophysics Laboratory

Powered by the Spatial Information Services Stack Common Platform MarineEnvironment, Water Groundwater Geology Geophysics Virtual Geophysics Laboratory v1.1 | Ryan Fraser 11 |

VGL – Point-of-View benefits User: Data all accessible in one place Same science but MUCH more efficient Bigger scale, quicker, automated Repeatability Developer/Tech: Reusable for other domains produces actual scientific data products Integrates with any SISS (OGC) data provider Power in the services, accessed using standardised protocols Virtual Geophysics Laboratory v1.1 | Ryan Fraser 12 |

VGL: Opportunities Free for research use Supporting multiple workflows Model Registry (3D) – researcher contributed models New Scientific Codes – Underworld, eScript, UBC, Airborne EM inversion codes + others (relatively “easy” to add) Data sets from GA: National Airborne Geophysical DB including –Gravity, Radiometric, AEM, Magnetics –Data sets from State Surveys Broad application and generic implementation Scalable Infrastructure (Cloud) Virtual Geophysics Laboratory v1.1 | Ryan Fraser 13 |

What’s left? More data, more codes, more workflows! BYO Resource (cloud allocation) Confidential Data Pawsey Centre, NCI, commercial clouds 14 | Virtual Geophysics Laboratory v1.1 | Ryan Fraser

Thank you Ryan Fraser Research Manager t wsiss.auscope.org VGL is Power by: VGL acknowledges funding from the NeCTAR project NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund. VGL is a collaboration between: VGL is supported by: University of Queensland, Monash University and the Australian National University VGL Launch Event is sponsored by:

Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 16 | Introducing The Virtual Geophysics Laboratory

No Installation Necessary! Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 17 |

Don’t Compute on your Laptop Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 18 |

Flexibility What I want to do ≠ Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 19 | Where I want to do itHow I want to do it ≠

Provenance Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 20 | Don’t be this guy...

To Quickly Recap VGL is a different kind of workflow engine... No installation necessary Doesn’t process on your machine Very flexible Records results and the process Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 21 |

Time for a live demo... Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 22 |

VGL is Power by: VGL acknowledges funding from the NeCTAR project NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund. VGL is a collaboration between: VGL is supported by: University of Queensland, Monash University and the Australian National University VGL Launch Event is sponsored by:

Data discovery Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 24 | Layers discovered via remote registries Layers consist of numerous remote data services

Data discovery Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 25 | Some data services support subsetting Some data services support reformatting e.g. CSV, NetCDF, GeoTIFF

Data discovery Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 26 | Some data is only registered with flat files

Data processing Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 27 | A variety of different scientific codes are already available in the form of “Toolboxes”

Data processing Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 28 | Further input files can be uploaded. Input files are passed directly into the cloud

Data processing Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote 29 | The steps so far have been building an environment to run a processing script...or build from existing templates Either write your own...

Managing results - provenance Presentation title | Presenter name 30 | All of a job’s outputs are also accessible Each job has a lifecycle that can be managed Successful jobs can have their entire process captured in a ISO ‘provenance record’

VGL is Power by: VGL acknowledges funding from the NeCTAR project NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund. VGL is a collaboration between: VGL is supported by: University of Queensland, Monash University and the Australian National University VGL Launch Event is sponsored by: