Presentation is loading. Please wait.

Presentation is loading. Please wait.

Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 OpenSource GeoSpatial Catalogue Platform-as-a-Service Salvatore Pinto Cloud.

Similar presentations


Presentation on theme: "Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 OpenSource GeoSpatial Catalogue Platform-as-a-Service Salvatore Pinto Cloud."— Presentation transcript:

1 www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 OpenSource GeoSpatial Catalogue Platform-as-a-Service Salvatore Pinto Cloud Technologist (EGI.eu) 1

2 www.egi.eu EGI-InSPIRE RI-261323 Outline EGI & EGI Federated Cloud Data Access and Dissemination PaaS service OpenSource GeoSpatial Catalogue 2

3 www.egi.eu EGI-InSPIRE RI-261323 What is EGI European –Over 35 countries Grid –Secure sharing Infrastructure –Computers –Cloud –Data –Applications Enabling more than 230 virtual organisations (>20K users) to perform digital research by pooling together their distributed resources Coordinated by EGI.eu Legal organisation based in Amsterdam

4 www.egi.eu EGI-InSPIRE RI-261323 EGI Solutions Portfolio Federated Infrastructure Operations Federated High- Throughput Data Analysis Federated Infrastructure as a Service Cloud Community Networks and Support Community Driven Innovations More info: http://www.egi.eu/solutions/

5 www.egi.eu EGI-InSPIRE RI-261323 What is the EGI Federated Cloud Standards and validation: federation is based on common Open-Standards – OCCI, CDMI, OVF, GLUE, etc... Heterogeneous implementation: no mandate on the cloud technology, the only condition is to expose the chosen interfaces and services. The EGI Federated Cloud is federation of institutional private Clouds, offering Cloud Infrastructure as a Service to scientists in Europe and worldwide. OCCI CDMI GLUE SAM GSI APEL GOCDB OVFHEPiX

6 www.egi.eu EGI-InSPIRE RI-261323 What is the EGI Federated Cloud Standards and validation: federation is based on common Open-Standards – OCCI, CDMI, OVF, GLUE, etc... Heterogeneous implementation: no mandate on the cloud technology, the only condition is to expose the chosen interfaces and services. The EGI Federated Cloud is federation of institutional private Clouds, offering Cloud Infrastructure as a Service to scientists in Europe and worldwide.

7 www.egi.eu EGI-InSPIRE RI-261323 Members ~70 individuals ~35 institutions >13 countries Members ~70 individuals ~35 institutions >13 countries Stakeholders 23 Resource Providers 10 production 10 Technology Providers 8 User Communities 4 Liaisons Technologies OpenNebula. StratusLab. OpenStack. Synnefo. WNoDeS. PERUN SlipStream Technologies OpenNebula. StratusLab. OpenStack. Synnefo. WNoDeS. PERUN SlipStream BSC CNRS LMU OeRC Masaryk IFAE CyfronetCESNET RADICAL SRCE DANTE FZJ GRNET GWDG STFC SARA KTH INFN- CNAF FCTSG EGI.eu Imperial CESGA CETA IFCA IGI IPHC IN2P3 SZTAKI IISAS SixSq 100% IT CSC INFN-Bari EGI Federated Cloud – September 2013

8 www.egi.eu EGI-InSPIRE RI-261323 Outline EGI & EGI Federated Cloud Data Access and Dissemination PaaS service OpenSource GeoSpatial Catalogue 8

9 www.egi.eu EGI-InSPIRE RI-261323 Data Access and Dissemination Service The problem: Provide community with a ready-to-use service for collecting, cataloguing, searching and disseminating of huge amount of scientific data. Generic and customizable service (need to adapt to different scientific user communities) The solution: Platform-as-a-Service built on top of Cloud IaaS and STaaS Service integrated with Cloud Storage, Long-term-data storage (ex. EUDAT safe replication) or local private storage. Based on the ENVRI project tools (OSGC catalogue) Pilot use case in collaboration with EISCAT 3D 9

10 www.egi.eu EGI-InSPIRE RI-261323 10 Open Source Geospatial Catalogue (OSGC) CESNET site (CZ) OpenNebula Catalogue EISCAT archive Cloud Storage Juelich site (DE) OpenStack SWIFT CDMI with HTTP export ENVRI-EISCAT 3D Pilot (concept with EGI solutions) EGI Federated Cloud Drop box tool to upload data on- demand from client side Near Real Time tool to import data automatically from receiving stations Admin tools Scientific users Data administrators Web browser wget 5m files, ~1TB in total On-site Off-site Phase 1: In ENVRI Phase 2: In a H2020 project Metadata generator service 1 Metadata generator service N... Processing / visulation service 1 Processing / visulation service N... Further metadata have to be extracted from the data E.g. Number of spikes, type of spikes Complex search based on data patterns E.g. Particular shapes of the data FFT, etc… Processing services (for metadata extraction and visualization) Integrated analysis tools on the data (eg. With GUISDAP) exposed to OSGC as services Further metadata have to be extracted from the data E.g. Number of spikes, type of spikes Complex search based on data patterns E.g. Particular shapes of the data FFT, etc… Processing services (for metadata extraction and visualization) Integrated analysis tools on the data (eg. With GUISDAP) exposed to OSGC as services

11 www.egi.eu EGI-InSPIRE RI-261323 11 Open Source Geospatial Catalogue (OSGC) CESNET site (CZ) OpenNebula Catalogue EISCAT archive Cloud Storage Juelich site (DE) OpenStack SWIFT CDMI with HTTP export ENVRI-EISCAT 3D Pilot (concept with EGI solutions) EGI Federated Cloud Drop box tool to upload data on- demand from client side Scientific users Data administrators Web browser wget 5m files, ~1TB in total On-site Off-site

12 www.egi.eu EGI-InSPIRE RI-261323 12 Open Source Geospatial Catalogue (OSGC) CESNET site (CZ) OpenNebula Catalogue EISCAT archive B2SAFE (safe data replication) Juelich site (DE) CSC site (SW) iRODS ENVRI-EISCAT 3D Pilot (integrating with EUDAT storage) EGI Federated Cloud Drop box tool to upload data on- demand from client side Scientific users Data administrators Web browser B2STAGE (data staging) On-site Off-site EUDAT

13 www.egi.eu EGI-InSPIRE RI-261323 13 Open Source Geospatial Catalogue (OSGC) CESNET site (CZ) OpenNebula Catalogue EISCAT archive Cloud Storage Juelich site (DE) OpenStack SWIFT CDMI with HTTP export ENVRI-EISCAT 3D Pilot (future work) EGI Federated Cloud Drop box tool to upload data on- demand from client side Near Real Time tool to import data automatically from receiving stations Admin tools Scientific users Data administrators Web browser wget 5m files, ~1TB in total On-site Off-site Phase 1: In ENVRI Phase 2: In a H2020 project Metadata generator service 1 Metadata generator service N... Processing / visulation service 1 Processing / visulation service N... Further metadata have to be extracted from the data E.g. Number of spikes, type of spikes Complex search based on data patterns E.g. Particular shapes of the data FFT, etc… Processing services (for metadata extraction and visualization) Integrated analysis tools on the data (eg. With GUISDAP) exposed to OSGC as services Further metadata have to be extracted from the data E.g. Number of spikes, type of spikes Complex search based on data patterns E.g. Particular shapes of the data FFT, etc… Processing services (for metadata extraction and visualization) Integrated analysis tools on the data (eg. With GUISDAP) exposed to OSGC as services

14 www.egi.eu EGI-InSPIRE RI-261323 Outline EGI & EGI Federated Cloud Data Access and Dissemination PaaS service OpenSource GeoSpatial Catalogue 14

15 www.egi.eu EGI-InSPIRE RI-261323 15 OpenSource Geospatial Catalogue Based on OpenSearch and OGC OpenSearch GeoSpatial Extension Released under GPLv3 on SourceForge (https://sourceforge.net/projects/osgcat/)https://sourceforge.net/projects/osgcat/ Built in PHP/PostgreSQL Timeline: –September 2013 : Defined requirements and started development –December 2013 : First Beta –January 2014 : Release Candidate –Febraury 2014 : Start PoC with EISCAT 3D –June 2014 : Final release

16 www.egi.eu EGI-InSPIRE RI-261323 16 Web Client Interface OpenSearch Catalogue Engine OpenSource GeoSpatial Catalogue Drop box tool to upload data on-demand from client side Near Real Time tool to import data automatically from receiving stations External Storage Scientific users Data administrators Admin Web Interface Platform Administrator OpenSearch RESTful API Storage module

17 www.egi.eu EGI-InSPIRE RI-261323 OpenSearch Catalogue Engine 17

18 www.egi.eu EGI-InSPIRE RI-261323 OpenSearch Catalogue Engine 18 Features: OGC 10-32r3 Compliant Custom product and series metadata definition (custom metadata schema per collection) Custom output format (Search output can be presented in custom formats, ex. RDF, Atom, KML) Custom Search Fields (OpenSearch search fields and rules are customizable) REST Interface (RESTful APIs, with GET, PUT, DELETE support)

19 www.egi.eu EGI-InSPIRE RI-261323 Dropbox 19

20 www.egi.eu EGI-InSPIRE RI-261323 Dropbox 20 Features: Simple catalogue ingestion (automatize metadata extraction, according to custom metadata extractor) Simple storage ingestion (automatize data storage) Simple APIs (RESTful APIs: PUT for file upload, DELETE for file removal) Support for NRTService metadata extractor

21 www.egi.eu EGI-InSPIRE RI-261323 Web Client 21

22 www.egi.eu EGI-InSPIRE RI-261323 Web Client 22 Features: Easy to use (users needs just a browser to query the catalogue) Customizable (you can customize query GUI and output metadata) OGC OpenSearch support (can query any catalogue, supposing that it exposes an OpenSearch RDF interface) Integration into virtual laboratories (can be integrated as standalone application to provide data access tools inside a virtual laboratory)

23 www.egi.eu EGI-InSPIRE RI-261323 Admin Web Interface 23

24 www.egi.eu EGI-InSPIRE RI-261323 Admin Web Interface 24 Features: Full customization from the web, with the possibility to setup and customize: Metadata parsers OpenSearch query fields and rules Search output formats Input metadata formats Integration with storage

25 www.egi.eu EGI-InSPIRE RI-261323 www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Thank you 25 Contacts: Salvatore Pinto (salvatore.pinto@egi.eu)

26 www.egi.eu EGI-InSPIRE RI-261323 26 Client Interface (EISCAT-3D proposal)

27 www.egi.eu EGI-InSPIRE RI-261323 27 Client Interface (EISCAT-3D proposal)


Download ppt "Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 OpenSource GeoSpatial Catalogue Platform-as-a-Service Salvatore Pinto Cloud."

Similar presentations


Ads by Google