1. PRODIGUER French Climate Modelling Information Services Platform

2. Core Partners IPSL – (Paris) CERFACS – (Toulouse) Meteo-France (Toulouse)

3. Core Team Sebastien Denvil (IPSL) Ashish Bhardwaj (IPSL) Mark Morgan (IPSL)

4. Core Objectives Collaborative eScience Accessible Data Real-time simulation monitoring Meta-data publishing

5. PART 1 The Big Picture

6. CCRT IDRIS CERFAC S Meteo France Core Computational Centres

7. SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CMIP5+)OPERATIONAL (R&D) PRODIGUER WEB SERVERS (IPSL) ESG GATEWAY (PCMDI) METAFOR (BADC) IS-ENES (DKRZ) PRODIGUER COMMUNITY (Scientists, Researchers, Students) HTTPS / X509 HTTPS / OpenID HTTPS / X509 HTTPS / OpenID

8. 3 Step Transition 1- Data Nodes 2 – Web Services 3 – Web Portals

9. STEP 1 DATA NODES CCRT, IDRIS, CERFACS, Meteo-France

10. COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Computational Centre

11. PRODIGUER DATA- NODE COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Computational Centre

12. ESG Data Publishing PRODIGUER DATA- NODE COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE Computational Centre

13. ESG Data Publishing Prodiguer Messaging PRODIGUER DATA- NODE COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Computational Centre

14. ESG Data Publishing Prodiguer Messaging PRODIGUER DATA- NODE COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Computational Centre

15. ESG Data Publishing Prodiguer Messaging PRODIGUER DATA- NODE COMPUTATION CLUSTER SM 1SM 2 SM N... STORAGE CLUSTER DISKTAPE PD G Computational Centre FIREWALL

16. WEB SERVICES ESG Publishing Prodiguer Messaging PRODIGUER DATA- NODE COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... STORAGE CLUSTER DISKTAPE PD G HTTPS GRID- FTP HTTPHTTPS FIREWALL Computational Centre

17. STEP 2 WEB SERVICES ESG, Prodiguer, Metafor, IS-ENES Meta-Data Publication

18. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)...

19. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CIMP5)

20. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CIMP5)OPERATIONAL

21. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... WEB SERVICES (RESTful, AtomPub) CORE (CIMP5)OPERATIONAL

22. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY WEB SERVICES (RESTful, AtomPub) OPERATIONALCORE (CIMP5)

23. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres CORE (CIMP5)OPERATIONAL

24. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres CORE (CIMP5)OPERATIONAL

25. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XMLXML XMLXML XML Base64 CORE (CIMP5)OPERATIONAL

26. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XMLXML HTTPS / X509 XMLXML XML Base64 CORE (CIMP5)OPERATIONAL

27. Prodiguer Web Services Apache X509 Python Pylons

28. STEP 3 WEB PORTALS ESG, Prodiguer, Metafor, IS-ENES Meta-Data Analysis

29. Meta-Data Analysis DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

30. Meta-Data Analysis WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

31. Meta-Data Analysis ESG CURATOR WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

32. Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

33. Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

34. Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres

35. Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres JSON JSON / XML XML

36. Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres HTTPS / OpenID JSON JSON / XML XML

37. Meta-Data Analysis Max Planck Institute University of Cantabria Meteo-France WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres JSON JSON / XML XML HTTPS / OpenID

38. ESG Web Portal

39. Metafor Web Portal

40. Prodiguer Web Portal Apache OpenID Python Pylons HTML Javascript Jquery JSON In Progress Version 1.0: July 2010

41. PART 2 Bottom-Up View

42. PRODIGUER Messaging Infrastructure Python, Message Queues, Databases

43. WEB SERVICES ESG Publishing Prodiguer Messaging PRODIGUER DATA- NODE COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... STORAGE CLUSTER DISKTAPE PD G HTTPS GRID- FTP HTTPHTTPS FIREWALL Computational Centre

44. WEB SERVICES Prodiguer Messaging PRODIGUER DATA- NODE COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... STORAGE CLUSTER DISKTAPE PD G HTTPS FIREWALL Computational Centre

45. OBJECTIVE Real time simulation monitoring across Prodiguer enabled computation centres

46. USE CASES View simulations by centre View simulations by time-period View Simulations by user View simulations by failure View simulations by coupling View simulations by coupling and failure View simulation configuration files Display results in tabular and graphical (Atlas) modes.

47. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE

48. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE Prodiguer Simulation Monitor

49. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE Prodiguer Simulation Monitor Prodiguer Data Node FIREWALL

50. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE Prodiguer Simulation Monitor Prodiguer Data NodeProdiguer Web Services FIREWALL HTTPS / X509

51. System Constraints Secure Fault Tolerant Distributed Reliable Extensible Loosely Coupled Python Implemented

52. What If....

53. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE Prodiguer Simulation Monitor Prodiguer Data NodeProdiguer Web Services FIREWALL HTTPS / X509

54. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE Prodiguer Simulation Monitor Prodiguer Data NodeProdiguer Web Services FIREWALL HTTPS / X509

55. Solution = Durable Message Queues

56. AMPQ Advanced Message Queue Protocol 1. A standard ala HTTP 2. Supported Message Patterns: Reliable Fire & Forget; Broadcast; File Transfer; 3. Supported by industry players

58. RabbitMQ AMQP Implementation 1. Erlang – i.e. parallel & scalable; 2. Persistence – i.e. durable; 3. Multi-OS – i.e. ubiquitous; 4. Python – py-amqlib / txAMPQ

59. RabbitMQ Python Libraries (py-amqlib / txAMPQ) # Importing client package from amqplib import client_0_8 as amqp # Opening comms channel to RabbitMQ server conn = amqp.Connection(host="localhost:5672 ", userid="guest", password="guest", virtual_host="/", insist=False) chan = conn.channel() # Declaring queue chan.queue_declare(queue="po_box", durable=True, auto_delete=False) # Writing to queue msg = amqp.Message("Test message!") chan.basic_publish(msg,exchange="sorting_room",routing_key="jason") # Reading from queue chan.queue_bind(queue="po_box", exchange="sorting_room", routing_key="EGU") msg = chan.basic_get("po_box")

60. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE Prodiguer Simulation Monitor Prodiguer Data NodeProdiguer Web Services FIREWALL HTTPS / X509

61. MessageInfo Python Base Classe MessageInfo MachineInfo* UserInfo* ProcessInfo* * = derived QueueInfo IdInfo (GUID, Timestamp, TypeCode)

62. ● MessageQueueManager ● Manages all message queue access ● MessageQueue ● Message queue wrapper (RabbitMQ) ● Exposes methods: queue(), dequeue(), requeue() ● MessageDispatcherManager ● Manages a private collection of message dispatchers ● Manages interaction with event queues ● MessageDispatcher ● Dispatches messages to next comms node Message Handling Python Classes

63. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE Prodiguer Simulation Monitor Event Queue(s) (RabbitMQ) Event Monitor Event Publisher Prodiguer Data Node

64. Event Info Python Classes EventInfo : MessageInfo SimulationInfo (Status, ID) * = derived

65. ● EventMonitorManager ● Manages collection of event monitors ● Exposes methods for invoking event listeners ● EventMonitor ● Handles events raised from simulation environment ● EventPublisherManager ● Manages a private collection of event publishers ● Manages interaction with event queues ● EventPublisher ● Publishes event information to Prodiguer data node Event Handling Python Classes

66. Prodiguer Data Node Prodiguer Simulation Monitor Event Queue(s) (RabbitMQ) Event Monitor Event Publisher Prodiguer Web Services

67. Event Info Python Classes * = derived SimulationInfo EventInfo : MessageInfo SimulationInfo (Status, ID) ComputingCentreInfo* (Code, Timestamp)

68. ● EventMonitorManager ● Manages collection of event monitors ● Exposes methods for invoking event listeners ● EventMonitor ● Handles events raised from simulation environment ● EventPublisherManager ● Manages a private collection of event publishers ● Manages interaction with event queues ● EventPublisher ● Publishes event information to Prodiguer data node Event Handling Python Classes

69. Prodiguer Web Services Prodiguer Simulation Monitor Event Queue(s) (RabbitMQ) Event Monitor Event Processors

70. Prodiguer Web Services Prodiguer Simulation Monitor Event Queue(s) (RabbitMQ) Event Monitor Event Processors PRODIGUER DATABASE(S) SqlAlchemy

71. ● EventMonitorManager ● Manages collection of event monitors ● Exposes methods for invoking event listeners ● EventMonitor ● Handles events raised from simulation environment ● EventProcessorManager ● Manages a private collection of event processors ● Manages interaction with event queues ● EventProcessor ● Processes event information, e.g. persist to database Event Handling Python Classes

72. Simulation Monitor Web Application Prodiguer Web Portal PRODIGUER DATABASE(S) PRODIGUER COMMUNITY Researchers, Scientists, Students, Impacts, Private Sector, Politicians, Journalists, Public HTTPS / OpenID JSON SqlAlchemy

73. System Constraints Secure Fault Tolerant Distributed Reliable Extensible Loosely Coupled Python Implemented

74. FUTURE Intra-Cluster Messaging ???

75. SUMMARY

76. PRODIGUER French Climate Modelling Information Services Platform

77. Core Partners IPSL – (Paris) CERFACS – (Toulouse) Meteo-France (Toulouse)

78. Core Team Sebastien Denvil (IPSL) Ashish Bhardwaj (IPSL) Mark Morgan (IPSL)

79. Core Objectives Collaborative eScience Accessible Data Real-time simulation monitoring Meta-data publishing

80. SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CMIP5+)OPERATIONAL (R&D) PRODIGUER WEB SERVERS (IPSL) ESG GATEWAY (PCMDI) METAFOR (BADC) IS-ENES (DKRZ) PRODIGUER COMMUNITY (Scientists, Researchers, Students) HTTPS / X509 HTTPS / OpenID HTTPS / X509 HTTPS / OpenID

81. ESG Publishing Prodiguer Messaging PRODIGUER DATA- NODE COMPUTATION CLUSTER SIM 1 SIM 2 SIM N... STORAGE CLUSTER DISKTAPE PD G HTTPS GRID- FTP HTTPHTTPS WEB SERVICES FIREWALL Computational Centre

82. Meta-Data Publication FRENCH SCIENTIFIC COMPUTING CENTRES (DN = DATA NODE) DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XMLXML HTTPS / X509 XMLXML XML Base64 CORE (CIMP5)OPERATIONAL

83. Meta-Data Analysis ESG CURATOR METAFOR / IS-ENES PRODIGUER WEB PORTALS (HTML / Javascript) DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres HTTPS / OpenID JSON JSON / XML XML

84. S 1... S N S 3 S 2 Simulation Execution Environment (e.g. IGCM) Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE Prodiguer Simulation Monitor Prodiguer Data NodeProdiguer Web Services FIREWALL HTTPS / X509

85. Durable Message Queues AMPQ RabbitMQ