Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction to the Aggregation Database Søren Roug, IT Project manager.

Published byAbner Sutton Modified over 8 years ago

Similar presentations

Presentation on theme: "Introduction to the Aggregation Database Søren Roug, IT Project manager."— Presentation transcript:

1 Introduction to the Aggregation Database Søren Roug, IT Project manager

2 What is the problem? How to discover the datasets How to search for the datasets How to track updates to the datasets How to bookmark found datasets How to merge datasets How to trust the dataset How to trust the trust The Aggregation Database is being developed to solve some new problems occuring when data is distributed in SEIS: The system is going to be an extension to EEA’s Reportnet system

3 Discovering the data 1.A collaborating node provides a manifest of the relevant files existing on its own website 2.Users register the files at a new website called QAW along the principles of del.icio.us and digg.com Datasets tend to be only numbers. Unlike Google, that mainly harvests webpages, a SEIS search engine can’t get information from the content of the files. The aggregation database will use two mechanisms to discover relevant datasets.

4 Discovering with the manifest file The manifest lists location and any metadata the provider has on each file The format can handle any type of metadata property Bathing water report, 2006 2007-11-15T16:17:27Z 2006

5 Registering a SEIS dataset Discovered via manifest files and manual registration

6 Adding metadata

7 Bookmarking and searching the dataset

8 Working with files vs. records Now we know where the files are in the SEIS universe But we can do more: We can read the content of XML files Example of an XML snippet: <stations xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xsi:noNamespaceSchemaLocation="http://water.eionet.europa.eu/stations.xsd"> 32301 St. Pölten 15.63166 48.21139 270 Industrial urban...

9 Merging principles Station structure as a table (austria.xml) Identifierlocal_codename... #3230132301St. Pölten... #3230232302Linz... Quadruple structure SubjectPredicateObjectSource #32301typeRiver Stationaustria.xml #32301local_code32301austria.xml #32301nameSt. Pöltenaustria.xml #32302typeRiver Stationaustria.xml #32302local_code32302austria.xml #32302nameLinzaustria.xml

10 Merging the datasets Austria Stations.xml Belgium Stations.xml Germany Stations.xml Aggregation Database XSL Transformation to quadruples SubjectPredicateObjectSource #32301nameSt. PöltenAu..xml #30299nameGentBe..xml #42882nameKölnGe..xml

11 Merging the datasets (with later updates) Austria Stations.xml Austria update1.xml Aggregation Database XSL Transformation SubjectPredicateObjectSource #32301nameSt. PöltenAu..xml #32301date2005-10-8Au..xml #32301nameSpratzernAu..update1.xml #32301date2008-6-18Au..update1.xml

12 Searching To find all river stations in Europe you search for subjects with the type=”River Station” The query will format it as a table for you Obviously you get duplicates because 32301 has been updated IdentifierLocal_codeNameDateLongitude #3230132301St. Pölten2005-10-815.63166 #32301Spratzern2008-6-18 #3029930299Gent2004-11-123.733333 #4288242882Köln2001-4-146.966667

13 QA work Let’s first colour the cells by their source IdentifierLocal_codeNameDateLongitude #3230132301St. Pölten2005-10-815.63166 #32301Spratzern2008-6-18 #3029930299Gent2004-11-123.733333 #4288242882Köln2001-4-146.966667

14 QA work Then we merge by letting the newer sources overwrite the older: IdentifierLocal_codeNameDateLongitude #3230132301Spratzern2008-6-1815.63166 #3029930299Gent2004-11-123.733333 #4288242882Köln2001-4-146.966667

15 QA work Don’t trust one source? Turn it off before you merge IdentifierLocal_codeNameDateLongitude #3230132301St. Pölten2005-10-815.63166 #32301Spratzern2008-6-18 #3029930299Gent2004-11-123.733333 #4288242882Köln2001-4-146.966667

16 QA work Then we merge IdentifierLocal_codeNameDateLongitude #3230132301St. Pölten2005-10-815.63166 #3029930299Gent2004-11-123.733333 #4288242882Köln2001-4-146.966667

17 QA work Gapfilling? Add your own source as a layer The layer is stored on QAW IdentifierLocal_codeNameDateLongitude #3230132301St. Pölten2005-10-815.63166 #32301Spratzern2008-6-18 #3029930299Gent2004-11-123.733333 #4288242882Köln2001-4-146.966667 #323012008-11-2715.65000 Hermann’s gapfilling layer created 2008-11-27

18 QA work Then we merge IdentifierLocal_codeNameDateLongitude #3230132301Spratzern2008-11-2715.65000 #3029930299Gent2004-11-123.733333 #4288242882Köln2001-4-146.966667 And we export to our working database for production...

19 Trusting the dataset and trusting trust Datasets and values can be evaluated by looking at the source Is the source URL from a reliable organisation/person? Is the methodology described? Are there reviews on QAW? Who wrote the reviews? Are there others who have used the data? Who are they?

20 Summary These new tools intend to solve the use of the Reportnet deliveries: Aggregation/Merging Manual QA and gap-filling Traceability to the sources Noticing when the source has been updated/deleted Review of the source for inclusion That was no problem before because only authorised parties could upload to CDR With SEIS now anyone can participate

Download ppt "Introduction to the Aggregation Database Søren Roug, IT Project manager."

Similar presentations

European Topic Centre on Biological Diversity Reporting Tool workshop March 2012.

European Topic Centre on Biological Diversity Reporting Tool workshop March 2012.
1 Reportnet – an introduction Reportnet – an introduction Presentation for the Meeting on Noise Copenhagen, 12 November 2008 Søren Roug, EEA 2008-11-12.

1 Reportnet – an introduction Reportnet – an introduction Presentation for the Meeting on Noise Copenhagen, 12 November 2008 Søren Roug, EEA
Linked Environment Data and how we are implementing SEIS Søren Roug.

Linked Environment Data and how we are implementing SEIS Søren Roug.
Effective management Accurate tracking Easier automation.

Effective management Accurate tracking Easier automation.
19 th Advanced Summer School in Regional Science An introduction to GIS using ArcGIS.

19 th Advanced Summer School in Regional Science An introduction to GIS using ArcGIS.
Federated Searching Pre-Conference Workshop - The federated searching cookbook Qin Zhu HP Labs Research Library February 18, 2007.

Federated Searching Pre-Conference Workshop - The federated searching cookbook Qin Zhu HP Labs Research Library February 18, 2007.
1 Introducing Reportnet Miruna Badescu. 2 A linear view of Reportnet process.

1 Introducing Reportnet Miruna Badescu. 2 A linear view of Reportnet process.
System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.

System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.
New “Collaborate” Button Integrate UI directly into the browser. Preferred target: Firefox Easiest browser to extend in terms of UI.

New “Collaborate” Button Integrate UI directly into the browser. Preferred target: Firefox Easiest browser to extend in terms of UI.
1 User Manual C Manufacturer & Association Registrations See user manual A for free and skydiver registrations and user manual B for business and rigger.

1 User Manual C Manufacturer & Association Registrations See user manual A for free and skydiver registrations and user manual B for business and rigger.
An Overview of Social Bookmarking By: Aimee Martin Technology Resource Teacher

An Overview of Social Bookmarking By: Aimee Martin Technology Resource Teacher
Free e-Sources for English Language Teachers by Wallace Barboza Carolina TESOL December 6th, 2008 Charleston, SC.

Free e-Sources for English Language Teachers by Wallace Barboza Carolina TESOL December 6th, 2008 Charleston, SC.
Status of upgrading CDI service (user interface, harvesting via GeoNetwork, CDI interoperability options following SeaDataNet D8.7) By Dick M.A. Schaap.

Status of upgrading CDI service (user interface, harvesting via GeoNetwork, CDI interoperability options following SeaDataNet D8.7) By Dick M.A. Schaap.
Reportnet standards and next steps Søren Roug, Information and Data Services (IDS)

Reportnet standards and next steps Søren Roug, Information and Data Services (IDS)
Chapter 7 Web Content Mining Xxxxxx. Introduction Web-content mining techniques are used to discover useful information from content on the web – textual.

Chapter 7 Web Content Mining Xxxxxx. Introduction Web-content mining techniques are used to discover useful information from content on the web – textual.
Event-Based Model for Reconciling Digital Entries Thesis Proposal Ahmet Fatih Mustacoglu 10/3/20151Ahmet.

Event-Based Model for Reconciling Digital Entries Thesis Proposal Ahmet Fatih Mustacoglu 10/3/20151Ahmet.
ITIS 1210 Introduction to Web-Based Information Systems Chapter 27 How Internet Searching Works.

ITIS 1210 Introduction to Web-Based Information Systems Chapter 27 How Internet Searching Works.
Julie Hannaford Director, Information Resources & Services OISE, University of Toronto Image credit to:

Julie Hannaford Director, Information Resources & Services OISE, University of Toronto Image credit to:
1 Reportnet for beginners Hermann Peifer, EEA. 2 What is Reportnet? The EEA’s data collection machinery.

1 Reportnet for beginners Hermann Peifer, EEA. 2 What is Reportnet? The EEA’s data collection machinery.
© 2008 The McGraw-Hill Companies, Inc. All rights reserved. ACCESS 2007 M I C R O S O F T ® THE PROFESSIONAL APPROACH S E R I E S Lesson 13 – Advanced.

© 2008 The McGraw-Hill Companies, Inc. All rights reserved. ACCESS 2007 M I C R O S O F T ® THE PROFESSIONAL APPROACH S E R I E S Lesson 13 – Advanced.

Similar presentations

Ads by Google