Presentation is loading. Please wait.

Presentation is loading. Please wait.

18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center.

Published byNora Barritt Modified over 10 years ago

Similar presentations

Presentation on theme: "18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center."— Presentation transcript:

1 18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center – DKRZ * Collaborative Climate Community Grid Project (Part of D-Grid Initiative)

2 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 2  C3Grid Overview: Architecture, Partners, Goals..  C3Grid Federation System Components:  C3Grid ISO Discovery Metadata and Metadata Catalog A short interop. study: C3Grid ISO Metadata / Geonetwork  Data Access and Preprocessing  C3Grid Security Overview  C3Grid / IPCC ?

3 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 3 C3Grid Data and Job Management Middleware D-Grid (SRM, d-cache,..) D-Grid (SRM, d-cache,..) C3Grid: Overview World Data Centers Research Institutes ClimateMareRSATPIKGKSSAWIMPI-M Universities FU Berlin Uni Köln Data Access Interface DWD ISO Discovery Metadata Data + Metadata Workflow Data + Metadata Grid Data / Job Interface ISO 19139 Discovery Catalog Result Data Products + Metadata C3Grid Data Providers Collaborative Grid Workspace (A)(B) ? ! IFM- Geomar DKRZ Portal C3RC

4 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 4 (A) Metadata for Data Discovery: Design and Implementation Data Access InterfaceISO Discovery Metadata ISO 19139 Discovery Catalog C3Grid Data Providers (A) ?

5 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 5 (A) Metadata – harvesting and lookup components Fast Range Queries Java API + Web Service Interface made available on sourceforge.net see also: http://www.panfmp.org Technology  ISO 19115/19139 metadata profile  OAI-PMH harvesting  catalogue  lucene based catalogue search  GridSphere based portal

6 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 6 (A) C3Grid ISO 19139 profile Design criteria:  no schema extensions, profiling by restriction  restriction using schematron constraints  „the granularity of the discovery metadata should reflect the logical organization of the data repository at a sufficiently coarse grained level“ (1)  CF based content description  Link to resource metadata infrastructure (GT4-MDS based) (1) Inspire: DT Metadata – Draft Implementing Rules for Metadata (version 2, 02/02/2007)

7 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 7 (A) C3Grid ISO Profile Description at aggregate level (e.g. experiment)  Aggregate extent description with multiple verticalExtent sections  Sub-selection in data request

8 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 8 (A) C3Grid ISO Profile: CF usage air_temperature K sea_surface_temperature K Reference to vertical CRS Content description based on (extended) CF names  Link to corresponding vertical CRS

9 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 9 (A) C3Grid ISO profile Data Distributor Info: reference to C3Grid resource metadata catalog (MDS) (names  service endpoints) (optional: service endpoints)

10 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 10 (A) C3Grid ISO profile Data provenance description: by now (data staging output): simple sequence of ProcessStep descriptions later (c3grid processed data): combined Source/ProcessStep blocks + external data provenance store

11 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 11

12 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 12 C3Grid ISO Profile: A short geonetwork experiment Federation building: OAI-PMH, WebDAV, Z39.50, geonet Full ISO metadata support (ISO19139/19119) OGC CSW 2.0 reference impl. RSS and GeoRSS newsfeeds SKOS based thesauri adaptable to new schema`s schematron constraint checking On roadmap: flexible ISO profile support shibboleth integration

13 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 13 C3Grid ISO Profile: A short geonetwork experiment

14 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 14 Building complex metadata federations … Harvesting via: CSW OAI-PMH Geneonet Web-Dav

15 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 15 C3Grid ISO Profile: A short geonetwork experiment Import / Edit / Search: ok Missing: content (CF) search vertical search temporal BBox search data staging

16 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 16

17 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 17

18 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 18  complete portal protoype to seach, access (pre-process) data described by C3Grid ISO profile in 3 weeks based on geonetwork open source solution..

19 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 19

20 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 20 (B) Data Access and Preprocessing World Data CentersResearch InstitutesUniversity Partners Data Access InterfaceISO Discovery Metadata Data + Metadata Data Analysis Workflow Data + Metadata ISO Discovery Catalog Result Data Products + Metadata C3Grid Data Providers Collaborative Grid Workspace (A) (B) ? !

21 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 21 Data Staging Request Data IDs Output Properties Selection: lon, lat, alt time content: CF Data Staging Web Service DB Flat File Archive Distributed C3Grid Work Space (B) Data Access and Pre-Processing: Implementation Offer Time / resource estimation skeleton impl status.. Provider staging jars Provider staging scripts MD DB WS GRAM JSDL based description Processing jobs Local resource manager C3Grid Generation 1: secured plain web services (status) C3Grid Generation 2: WSRF service interfaces (scheduled november 08) Generation 2+: full PKI/SAML security stack

22 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 22 C3Grid Middleware Components Scheduler: Globus WSRF based, accepts WSL workflow description: compute tasks + data staging tasks Datamanagement: Globus WSRF based, offer negotiation with scheduler, consistent view to distributed data, (later: replica management, caching) Globus MDS Resource Metadata Catalog: service registry, resource status  Dependency on Globus SW stack, no high level impl. support tools, impl. Globus 4.1.x migration ??, problems with delegation impl. (insufficient docu. and guidance)

23 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 23 C3Grid Workflow Analysis workflow-related task-related analysis and preparation of workflows monitoring and management of workflow execution (individual) scheduling strategy to optimize the management Handler to facade single/ specific Tasks interaction an moitoring via WS Notification standard

24 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 24 (C) Security Infrastructure Identity Provider Home Organisation Attribute Provider Virtual Organisation MyProxy Delegation Service Grid Service Grid Resource GRAM / DataRAM C3Grid Middleware GridShib SAML tools wflow client SLCS (CA) SLCS (CA) X509 Grid- proxy GridShib for GT policy Portal <..SAML Assertions..> SAML Personal / Group Account „Home attributes + VO attributes“ DFN Browser Webstart app Shibb. login

25 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 25 (C) Security Infrastructure Status: Shibb IdP`s running at core C3Grid partners Online CA for short-lived credentials tested, set up & operated by DFN (the German NREN) Online CA (DFN-SLCS) accreditation process with EUGridPMA started SLC contain campus attributes as SAML assertion Java Webstart app to bootstrap SLCS in development at DFN GridShib SAML Tools (v0.6.0) tested Prototype of shibbolized GridSphere portal tested open issues with GT4 proxy-delegation implementation Next: Integration of components Virtual home organization for C3 users without a Shibboleth IdP Integration of VO attributes (shibbolized VOMS)

26 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 26 C3Grid / IPCC Use Case (0) IPCC Metadata harvested / mirrored in CERA DB (WDCC) (1) Metadata visible in C3 Portal (2) User issues IPCC data import from external repository (3) User  OpenID IdP / + IPCC_Access role  external repos (4) Download  ??  C3 Repository (5) C3Grid grants access to users with IPCC_Access role ‘grant procedure ?’: before each wflow exec. contact to IdP/AttributeService ?? or more offline method ? C3RC / C3 Workspace IPCC data import Wflow result publication Analysis wflow

27 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 27 Appendix

28 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 28 C3Grid Content Info (Version 2) CF_name_with_attribute <MD_CoverageContentTypeCode codeList="http://wis.wmo.int/2006/catalogues/cf-standard-name-table.xml""http://wis.wmo.int/2006/catalogues/cf-standard-name-table.xml" codeListValue="air_temperature"> air_temperature with a cell_methods attribute including time:mean (interval: 1 day) K

29 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 29 Security Aspect: C3Grid step 0  step 1

30 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 30

31 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 31

32 18.09.08 / S. Kindermann / DKRZ GO-ESSP 2008 32 Lucene+ Index (C) Data Reuse of Analysis Results: Metadata Generation OAI-Harvester WS Interface C3Grid Workspace wflow m_tool OAI-PMH Server Portal p_data parent process step source collection Time stamp Description Citation info Description * + + 0..* 0..1 is_part_of has_parent is_generated_by has_input “quality check” API Prototype (Python) Context description of Analysis Data: Aggregation Processing history

Download ppt "18.09.08 / S. Kindermann / DKRZ GO-ESSP 20081 C3-Grid * Federation System for Climate Data Handling Stephan Kindermann German Climate Computing Center."

Similar presentations

Building Portals to access Grid Middleware National Technical University of Athens Konstantinos Dolkas, On behalf of Andreas Menychtas.

Building Portals to access Grid Middleware National Technical University of Athens Konstantinos Dolkas, On behalf of Andreas Menychtas.
LEAD Portal: a TeraGrid Gateway and Application Service Architecture Marcus Christie and Suresh Marru Indiana University LEAD Project (http://lead.ou.edu)

LEAD Portal: a TeraGrid Gateway and Application Service Architecture Marcus Christie and Suresh Marru Indiana University LEAD Project (
MyProxy Jim Basney Senior Research Scientist NCSA

MyProxy Jim Basney Senior Research Scientist NCSA
© 2006 Open Grid Forum GGF18, 13th September 2006 OGSA Data Architecture Scenarios Dave Berry & Stephen Davey.

© 2006 Open Grid Forum GGF18, 13th September 2006 OGSA Data Architecture Scenarios Dave Berry & Stephen Davey.
UKOLN is supported by: JISC Information Environment update Repositories and Preservation Programme meeting, October 24-25, 2006 Rachel Heery UKOLN www.ukoln.ac.uk.

UKOLN is supported by: JISC Information Environment update Repositories and Preservation Programme meeting, October 24-25, 2006 Rachel Heery UKOLN
Data Management Expert Panel - WP2. WP2 Overview.

Data Management Expert Panel - WP2. WP2 Overview.
FP7-INFRA-222667 Enabling Grids for E-sciencE www.eu-egee.org EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.

FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.
Contrail and Federated Identity Management

Contrail and Federated Identity Management
C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Project: C3-Grid Collaborative Climate Community Data and Processing Grid Scheduling.

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Project: C3-Grid Collaborative Climate Community Data and Processing Grid Scheduling.
MTA SZTAKI Hungarian Academy of Sciences Grid Computing Course Porto, 22-24 January 2007. Introduction to Grid portals Gergely Sipos

MTA SZTAKI Hungarian Academy of Sciences Grid Computing Course Porto, January Introduction to Grid portals Gergely Sipos
Globus Toolkit 4 hands-on Gergely Sipos, Gábor Kecskeméti MTA SZTAKI

Globus Toolkit 4 hands-on Gergely Sipos, Gábor Kecskeméti MTA SZTAKI
NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.

NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.
M.Lautenschlager (WDCC / MPI-M) / 15.06.06 / 1 GO-ESSP at LLNL Livermore, June 19th – 21st, 2006 World Data Center Climate: Status and Portal Integration.

M.Lautenschlager (WDCC / MPI-M) / / 1 GO-ESSP at LLNL Livermore, June 19th – 21st, 2006 World Data Center Climate: Status and Portal Integration.
Globus 4 Guy Warner NeSC Training.

Globus 4 Guy Warner NeSC Training.
Kate Keahey Argonne National Laboratory University of Chicago Globus Toolkit® 4: from common Grid protocols to virtualization.

Kate Keahey Argonne National Laboratory University of Chicago Globus Toolkit® 4: from common Grid protocols to virtualization.
Java Services in Apache Axis Plus GT Libraries and Handlers Your Python Service Flac WS RFT GRAM Delegation Index Trigger Python WS Core Your C Service.

Java Services in Apache Axis Plus GT Libraries and Handlers Your Python Service Flac WS RFT GRAM Delegation Index Trigger Python WS Core Your C Service.
INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Intelligent Distributed Data Management in Earth system science K. Ronneberger, DKRZ, Germany.

INFSO-RI Enabling Grids for E-sciencE Intelligent Distributed Data Management in Earth system science K. Ronneberger, DKRZ, Germany.
Data Integration in Service Oriented Architectures Rahul Patel Sr. Director R & D, BEA Systems Liquid Data – XML-based data access and integration for.

Data Integration in Service Oriented Architectures Rahul Patel Sr. Director R & D, BEA Systems Liquid Data – XML-based data access and integration for.
QCDgrid Technology James Perry, George Beckett, Lorna Smith EPCC, The University Of Edinburgh.

QCDgrid Technology James Perry, George Beckett, Lorna Smith EPCC, The University Of Edinburgh.
GRAPPA Part of Active Notebook Science Portal project A “notebook” like GRAPPA consists of –Set of ordinary web pages, viewable from any browser –Editable.

GRAPPA Part of Active Notebook Science Portal project A “notebook” like GRAPPA consists of –Set of ordinary web pages, viewable from any browser –Editable.

Similar presentations

Ads by Google