1. AstroWise((G)A)VO Meeting 6 May 2004 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO Edwin A. Valentijn

2. AstroWise((G)A)VO Meeting 6 May 2004 ((G)A)VO meeting Goal: Connect raw data to end user in VO environment –Exchange views on design level –3 years after –Design – bound / unbound –Review the layers Can we build a working near-VO system? –OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO Goal: Connect raw data to end user in VO environment –Exchange views on design level –3 years after –Design – bound / unbound –Review the layers Can we build a working near-VO system? –OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO

3. AstroWise((G)A)VO Meeting 6 May 2004 Programme – 2 rounds AstroWise –Vision - Edwin –Implementation – Object maker– Danny –Prototype – demo - Ewout –Object viewer - Michiel AstroGrid AVO Sextractor - AVO GAVO AstroWise –Vision - Edwin –Implementation – Object maker– Danny –Prototype – demo - Ewout –Object viewer - Michiel AstroGrid AVO Sextractor - AVO GAVO

4. AstroWise((G)A)VO Meeting 6 May 2004 VST

5. AstroWise((G)A)VO Meeting 6 May 2004 Paranal Mid 2005 + 3 year

6. AstroWise((G)A)VO Meeting 6 May 2004 Lofar IBM- Blue Gene/L

7. AstroWise((G)A)VO Meeting 6 May 2004 Large Data Volume Wide-field imaging instruments, vast amounts of data –VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data Wide-field imaging instruments, vast amounts of data –VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data Handling of the data is non-trivial –Pipeline data reduction –Calibration and re-calibration –Image comparisons and combinations –Working with source lists –Visualization Science can only be archive based Handling of the data is non-trivial –Pipeline data reduction –Calibration and re-calibration –Image comparisons and combinations –Working with source lists –Visualization Science can only be archive based ESO compliant ESO compliant } }

8. AstroWise((G)A)VO Meeting 6 May 2004 ASTRO-WISE Mission Virtual Survey System Environment that provides systematic and controlled –Access to all raw and calibration data –Execution and modification image/calibration pipelines –Execution of source extraction algorithms- catalogues –Archiving or regenerate dynamically –federated to link different data centers Environment that provides systematic and controlled –Access to all raw and calibration data –Execution and modification image/calibration pipelines –Execution of source extraction algorithms- catalogues –Archiving or regenerate dynamically –federated to link different data centers Dynamical archive continuously grows, can be used for –small or large science projects –generating and checking calibration data –exchanging methods, scripts and configuration Dynamical archive continuously grows, can be used for –small or large science projects –generating and checking calibration data –exchanging methods, scripts and configuration raw pixel data  pipelines/cal files  catalogues

9. AstroWise((G)A)VO Meeting 6 May 2004 Keys -Solution Procedurizing –Data taking at telescope for both science and calibration data Observing Modes: — Stare — Jitter — Dither — SSO Observing Strategies: — Stan — Deep — Freq — Mosaic –Full integration with data reduction –Design- ADD –Data model (classes) defined for data reduction and calibration –View pipeline as an administrative problem Procedurizing –Data taking at telescope for both science and calibration data Observing Modes: — Stare — Jitter — Dither — SSO Observing Strategies: — Stan — Deep — Freq — Mosaic –Full integration with data reduction –Design- ADD –Data model (classes) defined for data reduction and calibration –View pipeline as an administrative problem

10. AstroWise((G)A)VO Meeting 6 May 2004 Monitoring Photometric Calibration

11. AstroWise((G)A)VO Meeting 6 May 2004 Calibration procedures Sanity checks Quality control Calibration procedures Calibration procedures Image pipeline Source pipeline

12. AstroWise((G)A)VO Meeting 6 May 2004 AstroWise Pipelines Photometric pipeline Bias pipeline Flatfield pipeline Image pipeline Source pipeline

13. Tbyte source lists brains make the associations Link -lists as fast as possible

14. AstroWise((G)A)VO Meeting 6 May 2004 DB – engine Oracle contract: – reference – licenses - consultancy Oracle 9i -> 10g: -federation through “Advanced replication” to evolve into cross site links (=references= pointers) -Python I/F SQL – OCI- Oracle db -Full oo support + Python to db persistency -Terabyte scalability through “partitioned tables” -Administration tools –Interoperability: STREAMS connects to others: SyBase, MySQL Oracle contract: – reference – licenses - consultancy Oracle 9i -> 10g: -federation through “Advanced replication” to evolve into cross site links (=references= pointers) -Python I/F SQL – OCI- Oracle db -Full oo support + Python to db persistency -Terabyte scalability through “partitioned tables” -Administration tools –Interoperability: STREAMS connects to others: SyBase, MySQL

15. AstroWise((G)A)VO Meeting 6 May 2004 Share the load AstroWise Survey System Processing –Hardware Beowulf processors – 32 (most cases) Multi Terabyte disks (10 – 100) –Data reduction Derive calibration Run image pipeline (1 Mpx/s) Processing –Hardware Beowulf processors – 32 (most cases) Multi Terabyte disks (10 – 100) –Data reduction Derive calibration Run image pipeline (1 Mpx/s) Archiving –Storage Images (100’s Tbyte), Calibration files (10 Tbyte) Source parameters (1-10 Tbyte) –Federate (network speed) 5 Mb/s (24 hours/day) full replication (MegaCAM 10 Mb/s) 200 Mb/s no replication, on-the-fly retrieval

16. AstroWise((G)A)VO Meeting 6 May 2004 Contents of federation Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Coadded images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Coadded images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects

17. AstroWise((G)A)VO Meeting 6 May 2004 Concepts of federation Federation maintained by a single database- Oracle9i Full history tracking –of all input that went into result –providing on-the fly reprocessing Dynamical archive - Context as object attributes –Project: Calibration, Science, Survey, Personal –Owner: Pipeline, Developer, User –Strategy: — Standard — Deep — Freq — Mosaïc –Mode: — Stare — Jitter — Dither — SSO –Time: Time stamping VO interface Software standards –Classes/data model/procedures –00 – inheritance/ persistency –Python scripts/ c-libraries USER Python Federation maintained by a single database- Oracle9i Full history tracking –of all input that went into result –providing on-the fly reprocessing Dynamical archive - Context as object attributes –Project: Calibration, Science, Survey, Personal –Owner: Pipeline, Developer, User –Strategy: — Standard — Deep — Freq — Mosaïc –Mode: — Stare — Jitter — Dither — SSO –Time: Time stamping VO interface Software standards –Classes/data model/procedures –00 – inheritance/ persistency –Python scripts/ c-libraries USER Python

18. AstroWise((G)A)VO Meeting 6 May 2004 How to use the system? UI: Ingest raw data UI: Run standard pipelines – GUI UI: Modify pipeline/modules UI: Make/associate source lists 5LS: Query – whatever UI: Ingest raw data UI: Run standard pipelines – GUI UI: Modify pipeline/modules UI: Make/associate source lists 5LS: Query – whatever

19. AstroWise((G)A)VO Meeting 6 May 2004 VST - Virtual Survey Telescope

20. AstroWise((G)A)VO Meeting 6 May 2004 What ? science ? Collaborate - EU-wide -Projects - user defined qualifications -Surveys = projects -Archive studies = Surveys =projects Collaborate - EU-wide -Projects - user defined qualifications -Surveys = projects -Archive studies = Surveys =projects

21. AstroWise((G)A)VO Meeting 6 May 2004 New components Python binding to Oracle – 5LS ۷ Fileserver ۷ Parallelisation code ۷ Catalogue class/ associating HTM ۷ Federating with Oracle Streams User interfaces –Tell me everything tool on the fly reprocessing Python binding to Oracle – 5LS ۷ Fileserver ۷ Parallelisation code ۷ Catalogue class/ associating HTM ۷ Federating with Oracle Streams User interfaces –Tell me everything tool on the fly reprocessing

22. AstroWise((G)A)VO Meeting 6 May 2004 Schedule Hardware –OmegaCEN: 6Tbyte + servers + 8 node cluster operational ۷ –OAC, USM, Terapix + CVS federation ۷ Software –Design – review Q2 2002- ۷ –Design – formalized July 2003 ۷ –WFI +INT Prototype operational Q2 2003 ۷ –Basic operations – Q4 2003 ۷ –Evaluate and prepare for mass production 2004 –Qualify and populate 2005 –Deliver survey system – satellites Hardware –OmegaCEN: 6Tbyte + servers + 8 node cluster operational ۷ –OAC, USM, Terapix + CVS federation ۷ Software –Design – review Q2 2002- ۷ –Design – formalized July 2003 ۷ –WFI +INT Prototype operational Q2 2003 ۷ –Basic operations – Q4 2003 ۷ –Evaluate and prepare for mass production 2004 –Qualify and populate 2005 –Deliver survey system – satellites