1. 2004-06-03CMU-CS lunch talk, Gerard Lemson1 Computational and statistical problems for the Virtual Observatory With contributions from/thanks to: GAVO team: Wolfgang Voges, Matthias Steinmetz, Harry Enke, Hans-Martin Adorf Joerg Colberg (NVO@UPitt), Pat Dowler (CVO), Tony Banday (MPA), Class X team

2. 2004-06-03CMU-CS lunch talk, Gerard Lemson2 Overview Intro to VO IVOA standards process Some concrete examples, demos Scenarios, science cases Interesting problems

3. 2004-06-03CMU-CS lunch talk, Gerard Lemson3 Intro to VO Very large data sets Multi-wavelength astronomy made easy Federation of distributed archives. Publication of expert services. New software developments. Why contribute ? Too easy to do bad science ?

4. 2004-06-03CMU-CS lunch talk, Gerard Lemson4

5. 2004-06-03CMU-CS lunch talk, Gerard Lemson5 IVOA standards and specifications Collaboration of national VOs Develop standards for interoperability –publication (registry) –description (dm, ucd) –query (dal, voql) –data transfer (votable) –services (grid/web services) Interest groups: –architecture –applications –theory

6. 2004-06-03CMU-CS lunch talk, Gerard Lemson6 Babylonian confusion

7. 2004-06-03CMU-CS lunch talk, Gerard Lemson7 VO domain model as Esperanto

8. 2004-06-03CMU-CS lunch talk, Gerard Lemson8

9. 2004-06-03CMU-CS lunch talk, Gerard Lemson9

10. 2004-06-03CMU-CS lunch talk, Gerard Lemson10 Protocols VOTable + UCD  DM based XML + XSLT SCS/SIAP/SSAP  ADQL  VOQL SkyNode Registry resource model and harvesting interface

11. 2004-06-03CMU-CS lunch talk, Gerard Lemson11 Data models Targeted “small” data models –Quantity –Observation –Simulation Domain model as ontology Meta-data repository Bindings Representations, views, transformations

12. 2004-06-03CMU-CS lunch talk, Gerard Lemson12

13. 2004-06-03CMU-CS lunch talk, Gerard Lemson13

14. 2004-06-03CMU-CS lunch talk, Gerard Lemson14 Theory in the VO With Joerg Colberg http://ivoa.net/pub/papers/TheoryInTheVO.pdfhttp://ivoa.net/pub/papers/TheoryInTheVO.pdf Spatial query protocols irrelevant No object-based federation New phenomena/observables. Different kind of provenance. Model dependency. Theoretical archives rather unstructured. Theory/observational interface.

15. 2004-06-03CMU-CS lunch talk, Gerard Lemson15 Observed Simulated Thanks to Alexis Finoguenov, Ulrich Briel, Peter Schuecker, MPE) Thanks to Volker Springel

16. 2004-06-03CMU-CS lunch talk, Gerard Lemson16 Some concrete efforts NVO (USA): Registry (DIS), ADQL, SkyNode, data mining (UPitt+CMU)DIS AstroGrid (UK): grid/web services, work flows AVO (ESO, CDS, AstroGrid): Aladin visualization tool, science demosAladin visualization tool CVO (Canada): archive federation France VO: GalICSGalICS GAVO (Germany): data publication (RASS photons), application prototypes, data mining, theoryGAVO RASS photonsdata mining theory

17. 2004-06-03CMU-CS lunch talk, Gerard Lemson17 Scenarios, use cases, results Registry based data discovery and retrieval (GAVO, DIS) Class X classifier and generalizations X-Ray cluster analysis using simulations Cluster detection by combining SDSS and RASS catalogues (Schuecker et al, astro- ph/0403116) Discovery of obscured quasars using VO tools (Padovani et al, astro-ph/0406056)

18. 2004-06-03CMU-CS lunch talk, Gerard Lemson18 Typical workflow

19. 2004-06-03CMU-CS lunch talk, Gerard Lemson19 Download manager

20. 2004-06-03CMU-CS lunch talk, Gerard Lemson20 ClassXClassX@GAVO

21. 2004-06-03CMU-CS lunch talk, Gerard Lemson21 Theory/observational interface: X-Ray clusters Goal: interpret observations of X-Ray cluster using results of hydro simulations: 1.Extract parameters from the observation (services) that can be queried directly (dm, ucd). 2.Find simulations that may be relevant, that are “ similar ” to observation by searching registry for hydro simulations of clusters (registry, voql). Requires simulation results to be published and described in sufficient detail (dm, ucd). 3.Observe simulations using “ virtual telescope ” (application, grid/webservices) configured according to telescope configuration extracted from observation (dm). 4.Compare real with virtual observation (services). 5.For interesting simulation, extract full simulation result (dal) for further analysis, 6.or analyse the simulation using services (grid-services) provided by the archive or some other service provider

22. 2004-06-03CMU-CS lunch talk, Gerard Lemson22

23. 2004-06-03CMU-CS lunch talk, Gerard Lemson23 Computational, statistical and astronomical challenges I Data models Data modeling Data model transformations, views Archive structure Database tuning Querying, matching Distributed query algorithms Probabilistic matchers, systematic errors, identification of moving sources Improve identification using full point process information Add physical properties, not just position, to identification Complex, frequency dependent source definition Characterization of complex results in "few" parameters for discovery (PCA (after transformation)? 3D->2D ?) Comparison of real and virtual observations

24. 2004-06-03CMU-CS lunch talk, Gerard Lemson24 Usage Complex model Simplify using view concept Example from RDB XSLT for translation between domain XSD and application-specific derived schemas.

25. 2004-06-03CMU-CS lunch talk, Gerard Lemson25 CREATE VIEW SEXTRACTOR_GALAXIES AS SELECT S.RA AS _RAJ2000, S.DEC AS _DECJ2000, -2.5 * LOG(S.FLUX) AS M_APP, S.CLASSIFICATION, I.STORAGE_URL AS IMAGE FROM SOURCE S, SOURCE_CATALOGUE SC, IMAGE I, SOURCE_EXTRACTOR AS SE WHERE S.CLASS = ‘GALAXY’ AND S.FLUX < 15 AND S.CATALOGUE_ID = SC.ID AND IMAGE.ID = SC.IMAGE_ID AND SC.EXTRACTED_WITH = SOURCE_EXTRACTOR.ID AND SE.IDENTIFIER = ‘SExtractor’

26. 2004-06-03CMU-CS lunch talk, Gerard Lemson26 Probabilistic cross matching

27. 2004-06-03CMU-CS lunch talk, Gerard Lemson27 Computational, statistical and astronomical challenges II Data mining Algorithms for analyzing generic SEDs (classifiers ? visualization ? incorrect identification ?) Source extraction using multiple images, at very different wavelengths, how to take into account different physics/images of same source at different wavelengths ? Cluster finders using multiple catalogues Publish sophisticated statistical analysis algorithms Implementation Efficient implementation virtual telescopes (parallel, distributed, grid based, data structures)