Workshop Garching, June 27 – July Statistical Cross-Matching Across Distributed Archives H.-M. Adorf & GAVO Team MPI f. extraterrestrische Physik

Hans-Martin Adorf, GAVOMatcher Demo, Page 2 Statistical cross-matching  Cross-matching of astrometric and photometric catalogues –core functionality of a virtual observatory  Operational modes –on an area of the sky –using an input catalogue (GAVO matcher)

Hans-Martin Adorf, GAVOMatcher Demo, Page 3 Philosophy  Build a cross-matcher application that –should be usable by scientists and help producing science results –uses what’s there and what works now –doesn’t get stopped by a missing standard  Support the VO process by –helping to generate appropriate VO-standards –adopting new VO-standards whenever feasible

Hans-Martin Adorf, GAVOMatcher Demo, Page 4 Querying remote archives  Movie

Hans-Martin Adorf, GAVOMatcher Demo, Page 5 Querying remote archives  Movie  Using up to 10 servers –distributed around the world –operating in parallel  Sneak preview of grid computing –Locally specify your tasks –Execute them remotely at the data centers –Receive results locally for final combination

Hans-Martin Adorf, GAVOMatcher Demo, Page 6 Software demo (#1)  Input list –67 galaxies from FIRST radio catalogue  Query –2 remote archives: SDSS, VizieR –20 catalogues: radio, infrared, optical, X-ray  Task –get counterparts for each input coordinate –gather counterparts to form reasonable matches

Hans-Martin Adorf, GAVOMatcher Demo, Page 7 The matching problem (#1)

Hans-Martin Adorf, GAVOMatcher Demo, Page 8 The matching problem (#2)

Hans-Martin Adorf, GAVOMatcher Demo, Page 9 Matcher workflow

Hans-Martin Adorf, GAVOMatcher Demo, Page 10 Metadata  Querying and cross-matching requires metadata about catalogues & archives –astrometric fields and associated uncertainties –photometric fields and associated uncertainties –some metadata …  … are locally generated and stored  … are retrieved from archives in real-time

Hans-Martin Adorf, GAVOMatcher Demo, Page 11 Software demo (#2)  Issue: false alarms –matching is non-unique –input: 67 sources –output: almost 500 match candidates –many of these match candidates are “false alarms”

Hans-Martin Adorf, GAVOMatcher Demo, Page 12 Issue: false alarms (#3)  Two fundamental, independent probabilities –Hit probability: p(c|C) –False alarm probability:p(c|not C)  Goal –keep the hit probability high (completeness) –while keeping the false alarm probability low –goodness depends on S/N ratio in the data

Hans-Martin Adorf, GAVOMatcher Demo, Page 13 Issue: false alarms (#4)  Solution: use statistics (``fuzzy’’ matching) –compute statistical (Mahalanobis) distance between counterparts and center position –Compute reliability measure for match candidate (reduced chi-squared)

Hans-Martin Adorf, GAVOMatcher Demo, Page 14 Software demo (#3)  Lower reduced chi-squared from 10,000 to 3

Hans-Martin Adorf, GAVOMatcher Demo, Page 15 Software demo (#3)  Lower reduced chi-squared from 10,000 to 3  Result –Hit-rate is still pretty high –False-alarm rate is dramatically reduced

Hans-Martin Adorf, GAVOMatcher Demo, Page 16 Issue: server reliability  An archive server –may be down (easy to detect) –may be slow today (more difficult to detect) –may deliver wrong results (spoils the science)

Hans-Martin Adorf, GAVOMatcher Demo, Page 17 VO Standards  Status –Input  CSV files for data  XML files for query & match process description –Sending plain HTTP/HTML to archive servers –Receiving  CSV file from SDSS SkyServer  VOTable from VizieR (VO-Std) –Output  VOTable with complete match result (VO-Std) - VOPlot  various CSV files

Hans-Martin Adorf, GAVOMatcher Demo, Page 18 Software demo (#4)  VOPlot

Hans-Martin Adorf, GAVOMatcher Demo, Page 19 Plans & Ideas  GUI for newcomers –Facilitates selection of catalogues, astrometric & photometric columns, etc. –Generates configuration file  for query including server selection  for core cross-matcher, including chi-squared limit  Automatic monitoring of server response and reliability  Improved matching algorithm  GUI panel for match candidate visualization

Hans-Martin Adorf, GAVOMatcher Demo, Page 20 Summary  Shown a working cross-matcher application –Operates with distributed archives queried in parallel  Demonstrated that –fuzzy matching is needed –reduced chi-squared is a powerful statistical discriminator  High hit-probability, low false-alarm probability  GAVO cross-matcher currently being used in a first science application

Hans-Martin Adorf, GAVOMatcher Demo, Page 21 Thanks  Particularly to the folks –from SkyServer/SDSS, and –from CDS mirror sites, who, with their services, have enabled the cross- matcher

Hans-Martin Adorf, GAVOMatcher Demo, Page 22 The end

Hans-Martin Adorf, GAVOMatcher Demo, Page 23 Issue: false alarms (#5)

Hans-Martin Adorf, GAVOMatcher Demo, Page 24 Issue: false alarms (#6)

Hans-Martin Adorf, GAVOMatcher Demo, Page 25 GAVO  GAVO I –Funded by BMBF –Started end of 2002 –Ended end of March 2005  GAVO interim –Fundend  50% by Leibniz-prize money  50% by BMBF

Hans-Martin Adorf, GAVOMatcher Demo, Page 26 The matching problem (#3)