Download presentation
Presentation is loading. Please wait.
Published byBaldric McGee Modified over 9 years ago
1
Deutscher Wetterdienst DWD GISC Implementation GISC Development Team
2
Page: 2GISC Audit 17.09.2010 Agenda Welcome Architecture –High level Architecture –Hardware Performance Tests International Collaborations and Co-operations Contributions to WIS Q&A
3
Page: 3GISC Audit 17.09.2010 Design Principles of DWD GISC Reuse of existing well established software and implementation methodology Use of established Interfaces – in particular to „critical components“ to allow later exchange if necessary –Metadata Providing –Authentication and Authorization Modular concept to address performance and scalability requirements Clear separation of responsibilities Keep things simple Describe external interfaces
4
Page: 4GISC Audit 17.09.2010 DWD GISC - Development In house development team and support of business units Professional Services –Fraunhofer – research facility –Con terra – experts on Metadata Co-operation with companies –ask Contributes to the development with 2-3 persons Supports WMO events (EC Demo) –IBL Long term partner Supports WMO events (EC Demo) Face lift of DWD GISC Offers professional services (integration and maintenance) for DWD GISC
5
Page: 5GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database Subscription Management Web Interfaces –Customer –Operator –Administrator Data & Product Import Management & GISC Synchroniser DAR Components – Metadata Harvesting – Metadata Providing Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases DAR Generator /Importer SRU Search and Retrieval by URL
6
Page: 6GISC Audit 17.09.2010 Timeline Project Start Concept Phase 11/08 01/09 03/09 05/0907/09 09/09 11/0901/10 03/10 05/10 VGISC Workshop G2G com. Workshop Impl. Start: Web Interface Subscription DAR Metadata Generator DAR Catalogue Perf. Study G2G com. Tests G2G com. Workshop MoU CMA VGISC Backup WS OAI Infrastructure SRU OAI & WIS Workshop Data & Product Management Test 06/10 EC Demo 08/10 OAI Tests 09/10 Metadata Study
7
Page: 7GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database Subscription Management Data & Product Import Management & GISC Synchroniser DAR Components – Metadata Harvesting – Metadata Providing Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases DAR Generator /Importer SRU Search and Retrieval by URL Web Interfaces –Customer –Operator –Administrator
8
Page: 8GISC Audit 17.09.2010 High Level Architecture: Web Interfaces GISC –Redesign of WebWerdis –Navigation structure –Search functions Simple Search Extended Search (e.g. Full Text Search) Expert Search –Internationalization (I18N) –Integrate logging framework (necessary for monitoring) –Code clean-up –Entry points for External Application (e.g. Online Resource) AUST –Redesign of AUST
9
Page: 9GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database Subscription Management Data & Product Import Management & GISC Synchroniser Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases SRU Search and Retrieval by URL Web Interfaces –Customer –Operator –Administrator DAR Components – Metadata Harvesting – Metadata Providing DAR Generator /Importer
10
Page: 10GISC Audit 17.09.2010 High Level Architecture: DAR Generator, Importer and Components DAR Metadata Provider –jOAI (also used by NCAR/UCAR – reference site) DAR Metadata Harvester –OAIHarvester2 –Modified Source Code with CMA together –In house development of a scheduler and workflow component (done by ask) DAR Metadata Generator –In house development by DWD shared with CMA, Hong Kong, WMO, Brazil, Norway, Australia, South Africa, … DAR Metadata Importer –In house development by DWD
11
Page: 11GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database DAR Components – Metadata Harvesting – Metadata Providing DAR Generator /Importer SRU Search and Retrieval by URL Web Interfaces –Customer –Operator –Administrator Subscription Management Data & Product Import Management & GISC Synchroniser Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases
12
Page: 12GISC Audit 17.09.2010 High Level Architecture: Delivery Components Subscription Management –Controller: time and event triggered management AFD –Workhorse in the background –GISC-to-GISC communication –File distribution –White List Message Switching System –REST Web-Service to manage and control Moving Weather –First proposal for generic Interface
13
Page: 13GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database Subscription Management Data & Product Import Management & GISC Synchroniser DAR Components – Metadata Harvesting – Metadata Providing Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases DAR Generator /Importer Web Interfaces –Customer –Operator –Administrator SRU Search and Retrieval by URL
14
Page: 14GISC Audit 17.09.2010 High Level Architecture: SRU SRU –WMO SRU2JDBC implementation Cooperation between WMO and DWD available as Open Source –In house development by DWD: Mapping of CQL (Contextual Query Language) to SQL (done by ask) –External configuration file for Mapping It is easy to add new indices Database specific feature (e.g. functions, special operators) are kept outside of the code –All Queries and Responses are encoded according to the SRU-Standard allowing for both interactive as well as automated usage – e.g. by catalogue systems
15
Page: 15GISC Audit 17.09.2010 Hardware Architecture –Infrastructure DMZDMZ INTRANETINTRANET Web Cache OAI Infrastructure Application Server Adapter & Scheduler Operations Application Server Adapter & Scheduler OAI Infrastructure Integration System
16
Page: 16GISC Audit 17.09.2010 Performance 05/2009 Performance Study of DAR Metadata Catalogues – Both Systems terraCatalog 2.3 and Geonetworks 2.2 do not fulfill all Performance Requirements 09/2009 CMA and DWD did performance study of GISC-to-GISC communication solutions – JMA Blog solution and AFD 03/2010 CMA and DWD did performance tests of OAI Provider solution 04/2010 INMET and DWD did performance tests of GISC-to-GISC communication solutions – AFD and Moving Weather (IBL) 07/2010 Internal performance tests of DAR Metadata Catalogue and SRU – Lessons learned –Database is hot spot, especially when we talk about Full Text Search –Oracle standard reports –Limit the result set – configuration parameter –Optimize SQL Statements Explain plan Rewrite SQL Statements Indices –Improved responses for most critical queries by an order of magnitude
17
Page: 17GISC Audit 17.09.2010 International collaboration: CMA (GISC) GISC-to-GISC communication –Synchronization of 24 hour cache Extensively tests of JMAs Blog solution Extensively tests of AFD GISC-to-GISC communication –Description for GISC-to-GISC backup based on XML Schema
18
Page: 18GISC Audit 17.09.2010 International collaboration: CMA (GISC) DAR Metadata –DAR Metadata Generator –Metadata Template –Request for Clarification paper 11/2009 (CMA, JMA and DWD) Metadata Providing / Harvesting –Functional tests of standard OAI provider –Common development of OAI Harvester –Extensively performance tests for DAR Metadata synchronization SRU –Mapping indices to DAR Metadata elements Workshops: –GISC-to-GISC communication workshop 04/2009 at DWD –GISC-to-GISC communication workshop 09/2009 at CMA –OAI workshop 03/2010 at CMA
19
Page: 19GISC Audit 17.09.2010 International co-operation: JMA (GISC) DAR Metadata –Design of DAR Metadata –Request for Clarification paper 11/2009 (CMA, JMA and DWD) Metadata Providing / Harvesting –Synchronization tests based on OAI GISC-to-GISC communication –Knowledge exchange JMA Blog solution –JMA will take XML Schema (based on a proposal from CMA and DWD) as working draft for GISC-to-GISC backup Workshop –WIS Workshop at JMA 03/2010
20
Page: 20GISC Audit 17.09.2010 International co-operation: Brazil (GISC) DAR Metadata –DAR Metadata Generator Metadata Providing / Harvesting –Synchronization tests based on OAI GISC-to-GISC communication –Moving Weather –AFD
21
Page: 21GISC Audit 17.09.2010 International collaboration: EUMETSAT (DCPC) DAR Metadata –Conceptual design of DAR Metadata –Knowledge exchange DAR catalogue –Extensively knowledge exchange VGISC ITT –Attended as expert EUMETSAT is connected as DCPC since 23th of August 2010 –Test Phase 06/07 2010
22
Page: 22GISC Audit 17.09.2010 International co-operation: Hong Kong (DCPC) DAR Metadata –DAR Metadata Generator Metadata Providing / Harvesting –Synchronization tests based on OAI International co-operation: NCAR/UCAR (DCPC) DAR Metadata –Knowledge exchange Metadata Providing / Harvesting –Synchronization tests based on OAI
23
Page: 23GISC Audit 17.09.2010 International co-operation: VGISC Since 2002 VGISC – Météo France, UK MetOffice, DWD, met.no, ECWMF and EUMETSAT SIMDAT VGISC ITT Several workshops regarding Backup and Recovery
24
Page: 24GISC Audit 17.09.2010 Contributions to WIS DWD participates several ETs, ICG, … Released Metadata catalogue performance study to community Geonetwork improvements Shared results of DAR Metadata analysis with WIS community DAR Metadata Generator Released SRU2JDBC in co-operation with WMO as Open Source http://sru2jdbc.svn.sourceforge.net/ http://sru2jdbc.svn.sourceforge.net/ Active member of jOAI community Contribute experts for WIS Jump Start Offer
25
Page: 25GISC Audit 17.09.2010 Q&A
26
Page: 26GISC Audit 17.09.2010 Live Demo http://gisc.dwd.de/
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.