The GAVO Cross-Matcher Application Hans-Martin Adorf, Gerard Lemson, Wolfgang Voges GAVO, Max-Planck-Institut für extraterrestrische Physik, Garching b.

Slides:



Advertisements
Similar presentations
Lehrstuhl Informatik III: Datenbanksysteme Grid-based Data Stream Processing in e-Science 1 Richard Kuntschke 1, Tobias Scholl 1, Sebastian Huber 1, Alfons.
Advertisements

Advanced SQL (part 1) CS263 Lecture 7.
Database Management Using Microsoft Access Xinhua Chen, Ph.D. Chinese Association of Professionals in Science and Technology March 23, 2003.
Application Graphic design / svetagraphics.com 01 FRAMEWORK data service.
1Key – Report Creation with DB2. DB2 Databases Create Domain for DB2 Test Demo.
MAST-VizieR/NED cross correlation tutorial 1. Introduction Figure 1: Screenshot of the MAST VizieR Catalog Search Form. or enter here as object class:
Technical BI Project Lifecycle
Guide to Oracle10G1 Introduction To Forms Builder Chapter 5.
XP Chapter 3 Succeeding in Business with Microsoft Office Access 2003: A Problem-Solving Approach 1 Analyzing Data For Effective Decision Making.
ADASS, London Sep 23-26, 2007 VOExplorer : Visualising Data Discovery in the Virtual Observatory Jonathan Tedds University of Leicester/AstroGrid/ EuroVO.
Chapter 7 Managing Data Sources. ASP.NET 2.0, Third Edition2.
WorkPad 4 Quick Start WorkPad 4 Quick Start  Business Optix brings the rigor and discipline of business modelling and design into.
EU 2nd Year Review – Jan – WP9 WP9 Earth Observation Applications Demonstration Pedro Goncalves :
1 Web Developer & Design Foundations with XHTML Chapter 6 Key Concepts.
Module 3: Table Selection
Ihr Logo Data Explorer - A data profiling tool. Your Logo Agenda  Introduction  Existing System  Limitations of Existing System  Proposed Solution.
S. Derriere et al., ESSW03 Budapest, 2003 May 20 UCDs - metadata for astronomy Sébastien Derriere François Ochsenbein Thomas Boch CDS, Observatoire astronomique.
Attribute Data in GIS Data in GIS are stored as features AND tabular info Tabular information can be associated with features OR Tabular data may NOT be.
Copyright © 2003 by Prentice Hall Module 4 Database Management Systems 1.What is a database? Data hierarchy and data organization Field, record, file,
Copyright © 2003 by Prentice Hall Computers: Tools for an Information Age Chapter 13 Database Management Systems: Getting Data Together.
Database Systems: Design, Implementation, and Management Eighth Edition Chapter 10 Database Performance Tuning and Query Optimization.
DAY 14: ACCESS CHAPTER 1 Tazin Afrin October 03,
GAVO – after one year H.-M. Adorf & the GAVO team.
Theory in the German Astrophysical VO Summary: We show results of efforts done within the German Astrophysical Virtual Observatory (GAVO). GAVO has paid.
Functions and Demo of Astrogrid 1.1 China-VO Haijun Tian.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
Flexibility and user-friendliness of grid portals: the PROGRESS approach Michal Kosiedowski
Analyzing Data For Effective Decision Making Chapter 3.
Prototype system of the Japanese Virtual Observatory The Japanese Virtual Observatory (JVO) aims at providing easy access to federated astronomical databases.
Strasbourg astronomical Data Centre (DS) Françoise GENOVA.
Lecture2: Database Environment Prepared by L. Nouf Almujally & Aisha AlArfaj 1 Ref. Chapter2 College of Computer and Information Sciences - Information.
Introduction to the ESA Planetary Science Archive  Jose Luis Vázquez (ESAC/ESA)  Dave Heather (ESTEC/ESA)  Joe Zender (ESTEC/ESA)
Data Management Console Synonym Editor
Searching Business Data with MOSS 2007 Enterprise Search Presenter: Corey Roth Enterprise Consultant Stonebridge Blog:
Key Applications Module Lesson 21 — Access Essentials
Federation and Fusion of astronomical information Daniel Egret & Françoise Genova, CDS, Strasbourg Standards and tools for the Virtual Observatories.
Wiss. Beirat AIP, ClusterFinder & VO-Methods H. Enke German Astrophysical Virtual Observatory ClusterFinder VO Methods for Astronomical Applications.
XP Chapter 3 Succeeding in Business with Microsoft Office Access 2003: A Problem-Solving Approach 1 Analyzing Data For Effective Decision Making Chapter.
Workshop Garching, June 27 – July Statistical Cross-Matching Across Distributed Archives H.-M. Adorf & GAVO Team MPI f. extraterrestrische Physik.
Access 2007 ® Use Databases How can Microsoft Access 2007 help you structure your database?
1 Introduction to Oracle Chapter 1. 2 Before Databases Information was kept in files: Each field describes one piece of information about student Fields.
Access Review. Access Access is a database application A database is a collection of records and files organized for a particular purpose Access supports.
Common Archive Observation Model (CAOM) What is it and why does JWST care?
1 Biometric Databases. 2 Overview Problems associated with Biometric databases Some practical solutions Some existing DBMS.
Japanese Virtual Observatory Project Abstract : The National Astronomical Observatory of Japan (NAOJ) started the Japanese Virtual Observatory (JVO) project.
CMU-CS lunch talk, Gerard Lemson1 Computational and statistical problems for the Virtual Observatory With contributions from/thanks to: GAVO.
Graphical Enablement In this presentation… –What is graphical enablement? –Introduction to newlook dialogs and tools used to graphical enable System i.
The International Virtual Observatory Alliance (IVOA) interoperability in action.
1 Chapter 4: Creating Simple Queries 4.1 Introduction to the Query Task 4.2 Selecting Columns and Filtering Rows 4.3 Creating New Columns with an Expression.
German Astrophysical Virtual Observatory Overview and Results So Far W. Voges, G. Lemson, H.-M. Adorf.
Japanese Virtual Observatory (JVO) National Astronomical Observatory of Japan (NAOJ) Contact Address: Demo 1: GRID-based database federation.
Performance Testing Test Complete. Performance testing and its sub categories Performance testing is performed, to determine how fast some aspect of a.
1 Chapter 3: Getting Started with Tasks 3.1 Introduction to Task Dialogs 3.2 Creating a Listing Report 3.3 Creating a Frequency Report 3.4 Creating a Two-Way.
Your university or experiment logo here Performance Monitoring Gidon Moont e-Science, HEP, Imperial College London Talk to JRA1.
In this session, you will learn to: Query data by using joins Query data by using subqueries Objectives.
Description and exemplification use of a Data Dictionary. A data dictionary is a catalogue of all data items in a system. The data dictionary stores details.
Oracle Business Intelligence Foundation – Testing and Deploying OBI Repository.
1 Working with MS SQL Server Beginning ASP.NET in C# and VB Chapter 12.
Database Systems, 8 th Edition SQL Performance Tuning Evaluated from client perspective –Most current relational DBMSs perform automatic query optimization.
Simulation Production System Science Advisory Committee Meeting UW-Madison March 1 st -2 nd 2007 Juan Carlos Díaz Vélez.
Physical Layer of a Repository. March 6, 2009 Agenda – What is a Repository? –What is meant by Physical Layer? –Data Source, Connection Pool, Tables and.
The AstroGrid-D Information Service Stellaris A central grid component to store, manage and transform metadata - and connect to the VO!
Some of the utilities associated with the development of programs. These program development tools allow users to write and construct programs that the.
InSilicoLab – Grid Environment for Supporting Numerical Experiments in Chemistry Joanna Kocot, Daniel Harężlak, Klemens Noga, Mariusz Sterzel, Tomasz Szepieniec.
Data Virtualization Tutorial: Custom Functions
Data Virtualization Tutorial: JSON_TABLE Queries
Database Performance Tuning and Query Optimization
Chapter 11 Database Performance Tuning and Query Optimization
Google Sky.
Datamining VizieR and SDSS
Presentation transcript:

The GAVO Cross-Matcher Application Hans-Martin Adorf, Gerard Lemson, Wolfgang Voges GAVO, Max-Planck-Institut für extraterrestrische Physik, Garching b. München, Germany Abstract: Catalogue matching is a key application of a virtual observatory. Its main purpose is object identification, classification and SED-construction, but it can also be used for cross-validating catalogues. The GAVO cross-matcher is a multi-archive, multi-server, multi-catalogue, statistical matcher, operating on astrometric coordinates and individual associated uncertainties. Based on a “private catalogue” of sky-positions, the matcher queries one or more distributed archives (currently SDSS and VizieR). From the primary match- results hypothetical combinations of counterparts, the “match-candidates”, are formed. The statistical properties of each match-candidate are computed, and a reduced chi-squared criterion is employed for discriminating against unlikely counterpart combinations. For each surviving match-candidate, the cross-matcher assembles astrometric, photometric and auxiliary information in CSV- or VOTable-formatted files. The GAVO cross-matcher application serves as a test-bed for exploring and refining the process of statistical catalogue- matching. The application is in beta-test status, and has already successfully been used in a scientific project (Komossa et al. 2005, submitted). Operations The operation of any matcher requires three major phases: 1.Specification of the match request: the user provides a list of input positions, along with a selection of the archives, archive servers, catalogues to query, and columns to retrieve from each catalogue. The specification also requires some parameters such as catalogue-specific search radii, and a discrimination threshold. 2.Execution of the match request: the matcher starts a thread for each query. When all queries have returned a result, the matcher carries out a deterministic match (left outer join in a database) followed by a statistical discrimination. The matcher produces a comprehensive match table and different views: a summary view, an astrometry view, and a photometry view. Each row is labeled by an ID uniquely identifying a match candidate. 3.Assessment of the match results: the user inspects the number of counterparts returned from each catalogue, investigates match ambiguities, views images and maps, and possibly modifies the match parameters. Features The GAVO matcher (Fig. 1) uses a “private” list of input positions, works with existing archives (currently VizieR and SDSS) that are accessed via HTTP, uses a local metadata file for archive, server, and catalogue description (Fig. 2), Offers a GUI editor supporting the formulation of a match request (Fig. 3), queries multiple catalogue servers in parallel (Fig. 4) using catalogue-specific search radii, uses individual astrometric uncertainties for statistical matching, and automatically generates several table views (Fig.5) from the comprehensive match result. Fig. 1: The upper half of the diagram shows the dataflow through the GAVO multi-catalogue matcher which takes an input list of sky positions, queries multiple catalogues from the VizieR and SDSS archives, and produces a comprehensive match result. Intermediate results are stored in an in- memory database, which is also used for deterministic matching. The lower half of the diagram shows the foreseen integration of the matcher with a future SED-assembly module. Summary Scientific catalogue matching is a complex process. It is important to involve astronomers with scientific matching projects as early as possible. A rich client GUI application (Figs. 2 & 3) facilitates the matching process. Multi-catalogue matching generates tables with numerous columns – typically 200 or more. Users are easily overwhelmed by the amount of data generated. Different tabular views (Fig. 5) support the assessment of match candidates. Results are available in different formats including VOTable, directly viewable via VOPlot. An augmented visualization tool is needed in order to assess match-candidates, particularly when counterpart ambiguities arise. A separate detailed technical report is available containing our observations and recommendations concerning catalogue matching. Demo The GAVO catalogue cross-matcher is being demonstrated at this conference. Fig. 2: The manager GUI facilitates the specification and control of the matching process. The left panel allows access to the local file system containing match configuration and result files. The upper right panel allows viewing of input and output data. (Here a portion of the archive metadata file is shown, specifying details of servers and catalogues.) The lower right panel shows logger information. Fig. 3: A GUI editor supports the specification of a complex match request. The editor reads the archive metadata from a local multi-archive.xml file. The user may select catalogues from archives, and select table columns from catalogues. The editor generates a multi-archive match query file. Fig. 4: A monitor shows how the matching process progresses. Catalogue information is displayed along with server description and run-time statistics. A failed query will automatically be re-submitted to a different server. Fig. 5: The summary view of the comprehensive match result. Each row contains a different match- candidate and is labeled with a unique ID. Several match candidates can be identified that are related to the same input position (matching ambiguity). A reduced chi-squared metric permits a statistical discrimination against the inclusion of unlikely counterparts into a match candidate.