PRODIGUER French Climate Modelling Information Services Platform.

Slides:

Advertisements

Similar presentations

The Quantum Chromodynamics Grid James Perry, Andrew Jackson, Matthew Egbert, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.

Advertisements

Preservation and Long Term Access of Data at the World Data Centre for Climate Frank Toussaint N.P. Drakenberg, H. Höck, M. Lautenschlager, H. Luthardt,

ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.

M. Lautenschlager, H. Ramthun 1 Metafor Review 5 / 2010.

Data Portal for the “Climate changes Spatial Planning” program Henk Klein Baltink (KNMI) Fred Bosveld (KNMI) Hans de Wolf (Dutch Space)

Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.

UbiStream Motivation  Streaming data are abundant in our surroundings: Length of queue at cafeteria If the stadium is crowded or not Course.

Windows.Net Programming Series Preview. Course Schedule CourseDate Microsoft.Net Fundamentals 01/13/2014 Microsoft Windows/Web Fundamentals 01/20/2014.

Scientific Data Infrastructure in CAS Dr. Jianhui Scientific Data Center Computer Network Information Center Chinese Academy of Sciences.

CCSM Portal/ESG/ESGC Integration (a PY5 GIG project) Lan Zhao, Carol X. Song Rosen Center for Advanced Computing Purdue University With contributions by:

 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.

1 Eric Guilyardi and the Metafor team Common Metadata for Climate Modelling Digital Repositories Metafor Dissemination Workshop Abingdon, 14 March 2011.

DISTRIBUTED COMPUTING

Initiatives toward Climate Services in France and in the European Communities C. Déandreis (CNRS/IPSL); M. Plieger and W. Som de Cerff (KNMI); Ph. Dandin,

Grid Architecture William E. Johnston Lawrence Berkeley National Lab and NASA Ames Research Center (These slides are available at grid.lbl.gov/~wej/Grids)

And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR

- EGU 2010 ESSI May Building on the CMIP5 effort to prepare next steps : integrate community related effort in the every day workflow to.

IODE Ocean Data Portal - ODP  The objective of the IODE Ocean Data Portal (ODP) is to facilitate and promote the exchange and dissemination of marine.

1 Earth System Modeling Framework Documenting and comparing models using Earth System Curator Sylvia Murphy: Julien Chastang:

- Vendredi 27 mars PRODIGUER un nœud de distribution des données CMIP5 GIEC/IPCC Sébastien Denvil Pôle de Modélisation, IPSL.

LOGO Development of the distributed computing system for the MPD at the NICA collider, analytical estimations Mathematical Modeling and Computational Physics.

May 6, 2002Earth System Grid - Williams The Earth System Grid Presented by Dean N. Williams PI’s: Ian Foster (ANL); Don Middleton (NCAR); and Dean Williams.

A scalable and flexible platform to run various types of resource intensive applications on clouds ISWG June 2015 Budapest, Hungary Tamas Kiss,

Lightweight construction of rich scientific applications Daniel Harężlak(1), Marek Kasztelnik(1), Maciej Pawlik(1), Bartosz Wilk(1) and Marian Bubak(1,

1 Accomplishments. 2 Overview of Accomplishments  Sustaining the Production Earth System Grid Serving the current needs of the climate modeling community.

Private Cloud Stack Deep Dive Enterprise Cloud Summit.

Earth System Curator and Model Metadata Discovery and Display for CMIP5 Sylvia Murphy and Cecelia Deluca (NOAA/CIRES) Hannah Wilcox (NCAR/CISL) Metafor.

1 GFDL Data Portal Current Status, Achievements and Future Development NOAATECH-2006 K.Dixon, V.Balaji, S.Nikonov GFDL, Princeton.

Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.

AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.

End-to-End Data Services A Few Personal Thoughts Unidata Staff Meeting 2 September 2009.

Global ADC Job Monitoring Laura Sargsyan (YerPhI).

Application of RDF-OWL in the ESG Ontology Sylvia Murphy: Julien Chastang: Luca Cinquini:

Holding slide prior to starting show. Lessons Learned from the GECEM Portal David Walker Cardiff University

INTRODUCTION TO GRID & CLOUD COMPUTING U. Jhashuva 1 Asst. Professor Dept. of CSE.

The Earth System Curator Metadata Infrastructure for Climate Modeling Rocky Dunlap Georgia Tech.

XSEDE GLUE2 Update 1. Current XSEDE Usage Using legacy TeraGrid information services Publishing compute information about clusters – Subset of XSEDE clusters.

PRODIGUER French Climate Modelling Information Services Platform.

Scientific Data Processing Portal and Heterogeneous Computing Resources at NRC “Kurchatov Institute” V. Aulov, D. Drizhuk, A. Klimentov, R. Mashinistov,

OMII-BPEL Grid Services Orchestration using the Business Process Execution Language (BPEL) Liang Chen Bruno Wassermann Project Inspector: Wolfgang Emmerich.

EGI… …is a Federation of over 300 computing and data centres spread across 56 countries in Europe and worldwide …delivers advanced computing.

Metafor Year 3 EU Review CIM Component Services. Of itself an Ontology is an inert artefact, i.e. a dictionary. CON-CIM Conceptual CIM.

Daniele Bonacorsi Andrea Sciabà

Accessing the VI-SEEM infrastructure

Fan Engagement Solution

Simulation Production System

Distributed Collaborations

Volunteer Computing for Science Gateways

Data Bridge Solving diverse data access in scientific applications

The Earth System Grid: A Visualisation Solution

POW MND section.

Onedata Eventually Consistent Virtual Filesystem for Multi-Cloud Infrastructures Michał Orzechowski (CYFRONET AGH)

Cloud Data platform (Cloud Application Development & Deployment)

Grid Computing.

Grid Portal Services IeSE (the Integrated e-Science Environment)

Recap: introduction to e-science

Virtualization in the gLite Grid Middleware software process

University of Technology

Monitoring of the infrastructure from the VO perspective

Johnson & Johnson Virtual Medical Offices and Research Laboratories

CMIP6 / ENES Data TF Meeting: DKRZ

Module 01 ETICS Overview ETICS Online Tutorials

Distributed Systems through Web Services

Large Scale Distributed Computing

Introduction to Cyberspace

How To Integrate an Application on Grid

Computer Network Information Center, Chinese Academy of Sciences

Welcome to (HT)Condor Week #19 (year 34 of our project)

DIBBs Brown Dog BDFiddle

Presentation transcript:

PRODIGUER French Climate Modelling Information Services Platform

COMMUNITY French, European, Global

Community - Problem Space ● E-science collaboration is difficult

Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense

Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors

Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students

Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector

Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector – Politicians / journalists / public

Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector – Politicians / journalists / public ● Data consumers & producers

Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector – Politicians / journalists / public ● Data consumers & producers ● Freedom of information

Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector – Politicians / journalists / public ● Data consumers & producers ● Freedom of information ● Demand for analytical tools

DATA NetCDF, PetaByte, Distributed

Data - Problem Space ● Massive volumes (peta-scale)

Data - Problem Space ● Massive volumes (peta-scale) ● Ever increasing

Data - Problem Space ● Massive volumes (peta-scale) ● Ever increasing ● Widely distributed

Data - Problem Space ● Massive volumes (peta-scale) ● Ever increasing ● Widely distributed ● Difficult to move

Data - Problem Space ● Massive volumes (peta-scale) ● Ever increasing ● Widely distributed ● Difficult to move ● Meta-data publication

META-DATA Publishing & Analysis

Meta-Data - Problem Space ● The black-box problem

Meta-Data - Problem Space ● The black-box problem ● Standards (CF / Metafor)

Meta-Data - Problem Space ● The black-box problem ● Standards (CF / Metafor) ● Extraction tools

Meta-Data - Problem Space ● The black-box problem ● Standards (CF / Metafor) ● Extraction tools ● Publication workflow

Meta-Data - Problem Space ● The black-box problem ● Standards (CF / Metafor) ● Extraction tools ● Publication workflow ● Governance

SOLUTION Data Nodes Web Services Web Portals

DATA NODES CCRT, IDRIS, CERFACS, Meteo-France

COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Data Node

PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Data Node

ESG Data Publishing PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Data Node

ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Data Node

ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Data Node

ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Data Node FIREWALL

ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... STORAGE CLUSTER DISKTAPE PD G HTTPS GRID- FTP HTTPHTTPS WEB SERVICES FIREWALL Data Node

WEB SERVICES ESG, Prodiguer, Metafor, IS-ENES Meta-Data Publication

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)...

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CIMP5)

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CIMP5)OPERATIONAL

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... WEB SERVICES (RESTful, AtomPub) CORE (CIMP5)OPERATIONAL

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY WEB SERVICES (RESTful, AtomPub) OPERATIONALCORE (CIMP5)

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres CORE (CIMP5)OPERATIONAL

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres CORE (CIMP5)OPERATIONAL

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XMLXML XMLXML XML Base64 CORE (CIMP5)OPERATIONAL

Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XMLXML HTTPS / X509 XMLXML XML Base64 CORE (CIMP5)OPERATIONAL

WEB PORTALS ESG, Prodiguer, Metafor, IS-ENES Meta-Data Analysis

Meta-Data Analysis DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

Meta-Data Analysis WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

Meta-Data Analysis ESG CURATOR WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres JSON JSON / XML XML

Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres HTTPS / OpenID JSON JSON / XML XML

Meta-Data Analysis Max Planck Institute University of Cantabria Meteo-France WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres JSON JSON / XML XML HTTPS / OpenID

ESG Portal

Metafor Portal

Prodiguer Portal A VENIR Premier prototype: Fin Mai 2010

PRODIGUER SIMULATION MONITOR Python, Message Queues, Databases

ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... STORAGE CLUSTER DISKTAPE PD G HTTPS GRID- FTP HTTPHTTPS WEB SERVICES FIREWALL Data Node

Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... PD G HTTPS WEB SERVICES FIREWALL Data Node

OBJECTIVE Real time monitoring of all simulations executing within Prodiguer computing centres

USE CASES View simulations by centre; View simulations by month View simulations by failure View simulations by coupling View simulations by coupling and failure View simulation configuration files Display results in tabular and graphical (Atlas) modes.

S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE

S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF Prodiguer Simulation Services Python (Async) Message Queues (RabbitMQ ) Event Monitor Event Publisher SIMULATION MACHINE

S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF Prodiguer Simulation Services Python (Async) Message Queues (RabbitMQ ) Event Monitor Event Publisher SIMULATION MACHINE PRODIGUER DATA NODE FIREWALL Base64

Event Info Python Classes ● EventInfo ● Encapsulates event information (IMPORTANT!) ● SimulationInfo ● Encapsulates information regarding the simulation ● ● MachineInfo* ● Encapsulates information regarding the machine ● UserInfo* ● Encapsulates information regarding the user ● CoreInfo* ● Encapsulates information regarding the cpu

Python Classes ● EventMonitorManager ● Manages collection of event monitors ● Exposes methods for invoking event monitors ● EventMonitor ● Handles events raised from simulation environment ● EventQueueManager ● Manages event queue access ● EventQueue ● Event queue wrapper (RabbitMQ) ● Exposes methods: queue(), dequeue(), requeue() ● EventPublisherManager ● Manages a private collection of event publishers ● Manages interaction with event queues ● EventPublisher ● Publishes event information to Prodiguer data node

PRODIGUER SIMULATION SERVICE FIREWALL Base64

Prodiguer Data Node Services Message Queues (RabbitMQ ) Event Monitor Event Publisher DATA NODE MACHINE PRODIGUER SIMULATION SERVICE FIREWALL Base64

Prodiguer Data Node Services Message Queues (RabbitMQ ) Event Monitor Event Publisher DATA NODE MACHINE PRODIGUER WEB SERVICE HTTPS / X509 PRODIGUER SIMULATION SERVICE FIREWALL Base64

Event Info Python Classes ● ComputationCentreInfo ● Encapsulates information regarding the computing centre

Python Classes ● EventMonitor ● Handles events raised from simulation environment. ● EventMonitorManager ● Manages a private collection of event monitors ● Exposes methods for invoking event monitors ● ● EventPublisher ● Publishes event information to Prodiguer web service ● EventPublisherManager ● Manages a collection of event publishers ● Manages interaction with event queues ● EventQueue ● Wrapper around an event queue (RabbitMQ) ● Exposes methods: queue(), dequeue(), requeue() ● EventQueueManager ● Manages access to a private collection of event queues

HTTPS / X509 PRODIGUER DATA NODE Base64

Message Queues (RabbitMQ) Event Monitor Web Service Event Processors Prodiguer Web Server HTTPS / X509 PRODIGUER DATA NODE Base64

Message Queues (RabbitMQ) Event Monitor Web Service Event Processors Prodiguer Web Server HTTPS / X509 PRODIGUER DATA NODE Base64 PRODIGUER DATABASE(S) SqlAlchemy

Python Classes ● EventMonitorController ● HTTP controller handling events raised from data node ● Validates incoming http messages ● Deserializes incoming http messages ● ● EventQueue ● Event queue wrapper (RabbitMQ) ● Exposes methods: queue(), dequeue(), requeue() ● EventQueueManager ● Manages event queue access ● EventProcessorManager ● Manages a private collection of event publishers ● Manages interaction with event queues ● EventProcessor ● Publishes event information to Prodiguer web service

Prodiguer Web Portal PRODIGUER DATABASE(S)

Simulation Monitor Web Application Prodiguer Web Portal PRODIGUER DATABASE(S) SqlAlchemy

Simulation Monitor Web Application Prodiguer Web Portal PRODIGUER DATABASE(S) PRODIGUER COMMUNITY Researchers, Scientists, Students, Impacts, Private Sector, Politicians, Journalists, Public HTTPS / OpenID JSON SqlAlchemy

COMMUNITY French, European, Global

DATA NetCDF, PetaByte, Distributed

META-DATA Publishing & Analysis

SOLUTION Data Nodes Web Services Web Portals