Presentation is loading. Please wait.

Presentation is loading. Please wait.

PRODIGUER French Climate Modelling Information Services Platform.

Published byEzra Henry Modified over 8 years ago

Similar presentations

Presentation on theme: "PRODIGUER French Climate Modelling Information Services Platform."— Presentation transcript:

1 PRODIGUER French Climate Modelling Information Services Platform

2 COMMUNITY French, European, Global

3 Community - Problem Space ● E-science collaboration is difficult

4 Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense

5 Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors

6 Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students

7 Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector

8 Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector – Politicians / journalists / public

9 Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector – Politicians / journalists / public ● Data consumers & producers

10 Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector – Politicians / journalists / public ● Data consumers & producers ● Freedom of information

11 Community - Problem Space ● E-science collaboration is difficult ● Political debate is intense ● Many different actors – Scientists / researchers / students – Impacts community / private sector – Politicians / journalists / public ● Data consumers & producers ● Freedom of information ● Demand for analytical tools

12 DATA NetCDF, PetaByte, Distributed

13 Data - Problem Space ● Massive volumes (peta-scale)

14 Data - Problem Space ● Massive volumes (peta-scale) ● Ever increasing

15 Data - Problem Space ● Massive volumes (peta-scale) ● Ever increasing ● Widely distributed

16 Data - Problem Space ● Massive volumes (peta-scale) ● Ever increasing ● Widely distributed ● Difficult to move

17 Data - Problem Space ● Massive volumes (peta-scale) ● Ever increasing ● Widely distributed ● Difficult to move ● Meta-data publication

18 META-DATA Publishing & Analysis

19 Meta-Data - Problem Space ● The black-box problem

20 Meta-Data - Problem Space ● The black-box problem ● Standards (CF / Metafor)

21 Meta-Data - Problem Space ● The black-box problem ● Standards (CF / Metafor) ● Extraction tools

22 Meta-Data - Problem Space ● The black-box problem ● Standards (CF / Metafor) ● Extraction tools ● Publication workflow

23 Meta-Data - Problem Space ● The black-box problem ● Standards (CF / Metafor) ● Extraction tools ● Publication workflow ● Governance

24 SOLUTION Data Nodes Web Services Web Portals

25 DATA NODES CCRT, IDRIS, CERFACS, Meteo-France

26 COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Data Node

27 PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Data Node

28 ESG Data Publishing PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Data Node

29 ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Data Node

30 ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Data Node

31 ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Data Node FIREWALL

32 ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... STORAGE CLUSTER DISKTAPE PD G HTTPS GRID- FTP HTTPHTTPS WEB SERVICES FIREWALL Data Node

33 WEB SERVICES ESG, Prodiguer, Metafor, IS-ENES Meta-Data Publication

34 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)...

35 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CIMP5)

36 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CIMP5)OPERATIONAL

37 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... WEB SERVICES (RESTful, AtomPub) CORE (CIMP5)OPERATIONAL

38 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY WEB SERVICES (RESTful, AtomPub) OPERATIONALCORE (CIMP5)

39 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres CORE (CIMP5)OPERATIONAL

40 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres CORE (CIMP5)OPERATIONAL

41 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XMLXML XMLXML XML Base64 CORE (CIMP5)OPERATIONAL

42 Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XMLXML HTTPS / X509 XMLXML XML Base64 CORE (CIMP5)OPERATIONAL

43 WEB PORTALS ESG, Prodiguer, Metafor, IS-ENES Meta-Data Analysis

44 Meta-Data Analysis DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

45 Meta-Data Analysis WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

46 Meta-Data Analysis ESG CURATOR WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

47 Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

48 Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

49 Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

50 Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres JSON JSON / XML XML

51 Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres HTTPS / OpenID JSON JSON / XML XML

52 Meta-Data Analysis Max Planck Institute University of Cantabria Meteo-France WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres JSON JSON / XML XML HTTPS / OpenID

53 ESG Portal

54 Metafor Portal

55 Prodiguer Portal A VENIR Premier prototype: Fin Mai 2010

56

57 PRODIGUER SIMULATION MONITOR Python, Message Queues, Databases

58 ESG Data Publishing Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... STORAGE CLUSTER DISKTAPE PD G HTTPS GRID- FTP HTTPHTTPS WEB SERVICES FIREWALL Data Node

59 Prodiguer Simulation Monitoring PRODIGUER SERVER COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... PD G HTTPS WEB SERVICES FIREWALL Data Node

60 OBJECTIVE Real time monitoring of all simulations executing within Prodiguer computing centres

61 USE CASES View simulations by centre; View simulations by month View simulations by failure View simulations by coupling View simulations by coupling and failure View simulation configuration files Display results in tabular and graphical (Atlas) modes.

62 S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE

63 S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF Prodiguer Simulation Services Python (Async) Message Queues (RabbitMQ ) Event Monitor Event Publisher SIMULATION MACHINE

64 S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF Prodiguer Simulation Services Python (Async) Message Queues (RabbitMQ ) Event Monitor Event Publisher SIMULATION MACHINE PRODIGUER DATA NODE FIREWALL Base64

65 Event Info Python Classes ● EventInfo ● Encapsulates event information (IMPORTANT!) ● SimulationInfo ● Encapsulates information regarding the simulation ● ● MachineInfo* ● Encapsulates information regarding the machine ● UserInfo* ● Encapsulates information regarding the user ● CoreInfo* ● Encapsulates information regarding the cpu

66 Python Classes ● EventMonitorManager ● Manages collection of event monitors ● Exposes methods for invoking event monitors ● EventMonitor ● Handles events raised from simulation environment ● EventQueueManager ● Manages event queue access ● EventQueue ● Event queue wrapper (RabbitMQ) ● Exposes methods: queue(), dequeue(), requeue() ● EventPublisherManager ● Manages a private collection of event publishers ● Manages interaction with event queues ● EventPublisher ● Publishes event information to Prodiguer data node

67 PRODIGUER SIMULATION SERVICE FIREWALL Base64

68 Prodiguer Data Node Services Message Queues (RabbitMQ ) Event Monitor Event Publisher DATA NODE MACHINE PRODIGUER SIMULATION SERVICE FIREWALL Base64

69 Prodiguer Data Node Services Message Queues (RabbitMQ ) Event Monitor Event Publisher DATA NODE MACHINE PRODIGUER WEB SERVICE HTTPS / X509 PRODIGUER SIMULATION SERVICE FIREWALL Base64

70 Event Info Python Classes ● ComputationCentreInfo ● Encapsulates information regarding the computing centre

71 Python Classes ● EventMonitor ● Handles events raised from simulation environment. ● EventMonitorManager ● Manages a private collection of event monitors ● Exposes methods for invoking event monitors ● ● EventPublisher ● Publishes event information to Prodiguer web service ● EventPublisherManager ● Manages a collection of event publishers ● Manages interaction with event queues ● EventQueue ● Wrapper around an event queue (RabbitMQ) ● Exposes methods: queue(), dequeue(), requeue() ● EventQueueManager ● Manages access to a private collection of event queues

72 HTTPS / X509 PRODIGUER DATA NODE Base64

73 Message Queues (RabbitMQ) Event Monitor Web Service Event Processors Prodiguer Web Server HTTPS / X509 PRODIGUER DATA NODE Base64

74 Message Queues (RabbitMQ) Event Monitor Web Service Event Processors Prodiguer Web Server HTTPS / X509 PRODIGUER DATA NODE Base64 PRODIGUER DATABASE(S) SqlAlchemy

75 Python Classes ● EventMonitorController ● HTTP controller handling events raised from data node ● Validates incoming http messages ● Deserializes incoming http messages ● ● EventQueue ● Event queue wrapper (RabbitMQ) ● Exposes methods: queue(), dequeue(), requeue() ● EventQueueManager ● Manages event queue access ● EventProcessorManager ● Manages a private collection of event publishers ● Manages interaction with event queues ● EventProcessor ● Publishes event information to Prodiguer web service

76 Prodiguer Web Portal PRODIGUER DATABASE(S)

77 Simulation Monitor Web Application Prodiguer Web Portal PRODIGUER DATABASE(S) SqlAlchemy

78 Simulation Monitor Web Application Prodiguer Web Portal PRODIGUER DATABASE(S) PRODIGUER COMMUNITY Researchers, Scientists, Students, Impacts, Private Sector, Politicians, Journalists, Public HTTPS / OpenID JSON SqlAlchemy

79 COMMUNITY French, European, Global

80 DATA NetCDF, PetaByte, Distributed

81 META-DATA Publishing & Analysis

82 SOLUTION Data Nodes Web Services Web Portals

Download ppt "PRODIGUER French Climate Modelling Information Services Platform."

Similar presentations

The Quantum Chromodynamics Grid James Perry, Andrew Jackson, Matthew Egbert, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.

The Quantum Chromodynamics Grid James Perry, Andrew Jackson, Matthew Egbert, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.
Preservation and Long Term Access of Data at the World Data Centre for Climate Frank Toussaint N.P. Drakenberg, H. Höck, M. Lautenschlager, H. Luthardt,

Preservation and Long Term Access of Data at the World Data Centre for Climate Frank Toussaint N.P. Drakenberg, H. Höck, M. Lautenschlager, H. Luthardt,
ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.

ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.
M. Lautenschlager, H. Ramthun 1 Metafor Review 5 / 2010.

M. Lautenschlager, H. Ramthun 1 Metafor Review 5 / 2010.
Data Portal for the “Climate changes Spatial Planning” program Henk Klein Baltink (KNMI) Fred Bosveld (KNMI) Hans de Wolf (Dutch Space)

Data Portal for the “Climate changes Spatial Planning” program Henk Klein Baltink (KNMI) Fred Bosveld (KNMI) Hans de Wolf (Dutch Space)
Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.

Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.
UbiStream 2006.10.16. Motivation  Streaming data are abundant in our surroundings: Length of queue at cafeteria If the stadium is crowded or not Course.

UbiStream Motivation  Streaming data are abundant in our surroundings: Length of queue at cafeteria If the stadium is crowded or not Course.
Windows.Net Programming Series Preview. Course Schedule - 2014 CourseDate Microsoft.Net Fundamentals 01/13/2014 Microsoft Windows/Web Fundamentals 01/20/2014.

Windows.Net Programming Series Preview. Course Schedule CourseDate Microsoft.Net Fundamentals 01/13/2014 Microsoft Windows/Web Fundamentals 01/20/2014.
Scientific Data Infrastructure in CAS Dr. Jianhui Scientific Data Center Computer Network Information Center Chinese Academy of Sciences.

Scientific Data Infrastructure in CAS Dr. Jianhui Scientific Data Center Computer Network Information Center Chinese Academy of Sciences.
CCSM Portal/ESG/ESGC Integration (a PY5 GIG project) Lan Zhao, Carol X. Song Rosen Center for Advanced Computing Purdue University With contributions by:

CCSM Portal/ESG/ESGC Integration (a PY5 GIG project) Lan Zhao, Carol X. Song Rosen Center for Advanced Computing Purdue University With contributions by:
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.

 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
1 Eric Guilyardi and the Metafor team Common Metadata for Climate Modelling Digital Repositories Metafor Dissemination Workshop Abingdon, 14 March 2011.

1 Eric Guilyardi and the Metafor team Common Metadata for Climate Modelling Digital Repositories Metafor Dissemination Workshop Abingdon, 14 March 2011.
DISTRIBUTED COMPUTING

DISTRIBUTED COMPUTING
Initiatives toward Climate Services in France and in the European Communities C. Déandreis (CNRS/IPSL); M. Plieger and W. Som de Cerff (KNMI); Ph. Dandin,

Initiatives toward Climate Services in France and in the European Communities C. Déandreis (CNRS/IPSL); M. Plieger and W. Som de Cerff (KNMI); Ph. Dandin,
Grid Architecture William E. Johnston Lawrence Berkeley National Lab and NASA Ames Research Center (These slides are available at grid.lbl.gov/~wej/Grids)

Grid Architecture William E. Johnston Lawrence Berkeley National Lab and NASA Ames Research Center (These slides are available at grid.lbl.gov/~wej/Grids)
And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR 17.05.2012.

And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR
- EGU 2010 ESSI5 - 07 May 2010 - 1 Building on the CMIP5 effort to prepare next steps : integrate community related effort in the every day workflow to.

- EGU 2010 ESSI May Building on the CMIP5 effort to prepare next steps : integrate community related effort in the every day workflow to.
IODE Ocean Data Portal - ODP  The objective of the IODE Ocean Data Portal (ODP) is to facilitate and promote the exchange and dissemination of marine.

IODE Ocean Data Portal - ODP  The objective of the IODE Ocean Data Portal (ODP) is to facilitate and promote the exchange and dissemination of marine.
1 Earth System Modeling Framework Documenting and comparing models using Earth System Curator Sylvia Murphy: Julien Chastang:

1 Earth System Modeling Framework Documenting and comparing models using Earth System Curator Sylvia Murphy: Julien Chastang:
- Vendredi 27 mars 2009 1 PRODIGUER un nœud de distribution des données CMIP5 GIEC/IPCC Sébastien Denvil Pôle de Modélisation, IPSL.

- Vendredi 27 mars PRODIGUER un nœud de distribution des données CMIP5 GIEC/IPCC Sébastien Denvil Pôle de Modélisation, IPSL.

Similar presentations

Ads by Google