AstroWise((G)A)VO Meeting 6 May 2004 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO Edwin A. Valentijn

AstroWise((G)A)VO Meeting 6 May 2004 ((G)A)VO meeting Goal: Connect raw data to end user in VO environment –Exchange views on design level –3 years after –Design – bound / unbound –Review the layers Can we build a working near-VO system? –OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO Goal: Connect raw data to end user in VO environment –Exchange views on design level –3 years after –Design – bound / unbound –Review the layers Can we build a working near-VO system? –OmegaCAM VISTA Lofar SExtractor VOT GAVO AVO

AstroWise((G)A)VO Meeting 6 May 2004 Programme – 2 rounds AstroWise –Vision - Edwin –Implementation – Object maker– Danny –Prototype – demo - Ewout –Object viewer - Michiel AstroGrid AVO Sextractor - AVO GAVO AstroWise –Vision - Edwin –Implementation – Object maker– Danny –Prototype – demo - Ewout –Object viewer - Michiel AstroGrid AVO Sextractor - AVO GAVO

AstroWise((G)A)VO Meeting 6 May 2004 VST

AstroWise((G)A)VO Meeting 6 May 2004 Paranal Mid year

AstroWise((G)A)VO Meeting 6 May 2004 Lofar IBM- Blue Gene/L

AstroWise((G)A)VO Meeting 6 May 2004 Large Data Volume Wide-field imaging instruments, vast amounts of data –VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data Wide-field imaging instruments, vast amounts of data –VST sees equivalent of Southern sky 30 min exp, 300 nights/y in 3 years. 100 Tbyte of image data and Tbytes of source list data Handling of the data is non-trivial –Pipeline data reduction –Calibration and re-calibration –Image comparisons and combinations –Working with source lists –Visualization Science can only be archive based Handling of the data is non-trivial –Pipeline data reduction –Calibration and re-calibration –Image comparisons and combinations –Working with source lists –Visualization Science can only be archive based ESO compliant ESO compliant } }

AstroWise((G)A)VO Meeting 6 May 2004 ASTRO-WISE Mission Virtual Survey System Environment that provides systematic and controlled –Access to all raw and calibration data –Execution and modification image/calibration pipelines –Execution of source extraction algorithms- catalogues –Archiving or regenerate dynamically –federated to link different data centers Environment that provides systematic and controlled –Access to all raw and calibration data –Execution and modification image/calibration pipelines –Execution of source extraction algorithms- catalogues –Archiving or regenerate dynamically –federated to link different data centers Dynamical archive continuously grows, can be used for –small or large science projects –generating and checking calibration data –exchanging methods, scripts and configuration Dynamical archive continuously grows, can be used for –small or large science projects –generating and checking calibration data –exchanging methods, scripts and configuration raw pixel data  pipelines/cal files  catalogues

AstroWise((G)A)VO Meeting 6 May 2004 Keys -Solution Procedurizing –Data taking at telescope for both science and calibration data Observing Modes: — Stare — Jitter — Dither — SSO Observing Strategies: — Stan — Deep — Freq — Mosaic –Full integration with data reduction –Design- ADD –Data model (classes) defined for data reduction and calibration –View pipeline as an administrative problem Procedurizing –Data taking at telescope for both science and calibration data Observing Modes: — Stare — Jitter — Dither — SSO Observing Strategies: — Stan — Deep — Freq — Mosaic –Full integration with data reduction –Design- ADD –Data model (classes) defined for data reduction and calibration –View pipeline as an administrative problem

AstroWise((G)A)VO Meeting 6 May 2004 Monitoring Photometric Calibration

AstroWise((G)A)VO Meeting 6 May 2004 Calibration procedures Sanity checks Quality control Calibration procedures Calibration procedures Image pipeline Source pipeline

AstroWise((G)A)VO Meeting 6 May 2004 AstroWise Pipelines Photometric pipeline Bias pipeline Flatfield pipeline Image pipeline Source pipeline

Tbyte source lists brains make the associations Link -lists as fast as possible

AstroWise((G)A)VO Meeting 6 May 2004 DB – engine Oracle contract: – reference – licenses - consultancy Oracle 9i -> 10g: -federation through “Advanced replication” to evolve into cross site links (=references= pointers) -Python I/F SQL – OCI- Oracle db -Full oo support + Python to db persistency -Terabyte scalability through “partitioned tables” -Administration tools –Interoperability: STREAMS connects to others: SyBase, MySQL Oracle contract: – reference – licenses - consultancy Oracle 9i -> 10g: -federation through “Advanced replication” to evolve into cross site links (=references= pointers) -Python I/F SQL – OCI- Oracle db -Full oo support + Python to db persistency -Terabyte scalability through “partitioned tables” -Administration tools –Interoperability: STREAMS connects to others: SyBase, MySQL

AstroWise((G)A)VO Meeting 6 May 2004 Share the load AstroWise Survey System Processing –Hardware Beowulf processors – 32 (most cases) Multi Terabyte disks (10 – 100) –Data reduction Derive calibration Run image pipeline (1 Mpx/s) Processing –Hardware Beowulf processors – 32 (most cases) Multi Terabyte disks (10 – 100) –Data reduction Derive calibration Run image pipeline (1 Mpx/s) Archiving –Storage Images (100’s Tbyte), Calibration files (10 Tbyte) Source parameters (1-10 Tbyte) –Federate (network speed) 5 Mb/s (24 hours/day) full replication (MegaCAM 10 Mb/s) 200 Mb/s no replication, on-the-fly retrieval

AstroWise((G)A)VO Meeting 6 May 2004 Contents of federation Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Coadded images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Coadded images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects

AstroWise((G)A)VO Meeting 6 May 2004 Concepts of federation Federation maintained by a single database- Oracle9i Full history tracking –of all input that went into result –providing on-the fly reprocessing Dynamical archive - Context as object attributes –Project: Calibration, Science, Survey, Personal –Owner: Pipeline, Developer, User –Strategy: — Standard — Deep — Freq — Mosaïc –Mode: — Stare — Jitter — Dither — SSO –Time: Time stamping VO interface Software standards –Classes/data model/procedures –00 – inheritance/ persistency –Python scripts/ c-libraries USER Python Federation maintained by a single database- Oracle9i Full history tracking –of all input that went into result –providing on-the fly reprocessing Dynamical archive - Context as object attributes –Project: Calibration, Science, Survey, Personal –Owner: Pipeline, Developer, User –Strategy: — Standard — Deep — Freq — Mosaïc –Mode: — Stare — Jitter — Dither — SSO –Time: Time stamping VO interface Software standards –Classes/data model/procedures –00 – inheritance/ persistency –Python scripts/ c-libraries USER Python

AstroWise((G)A)VO Meeting 6 May 2004 How to use the system? UI: Ingest raw data UI: Run standard pipelines – GUI UI: Modify pipeline/modules UI: Make/associate source lists 5LS: Query – whatever UI: Ingest raw data UI: Run standard pipelines – GUI UI: Modify pipeline/modules UI: Make/associate source lists 5LS: Query – whatever

AstroWise((G)A)VO Meeting 6 May 2004 VST - Virtual Survey Telescope

AstroWise((G)A)VO Meeting 6 May 2004 What ? science ? Collaborate - EU-wide -Projects - user defined qualifications -Surveys = projects -Archive studies = Surveys =projects Collaborate - EU-wide -Projects - user defined qualifications -Surveys = projects -Archive studies = Surveys =projects

AstroWise((G)A)VO Meeting 6 May 2004 New components Python binding to Oracle – 5LS ۷ Fileserver ۷ Parallelisation code ۷ Catalogue class/ associating HTM ۷ Federating with Oracle Streams User interfaces –Tell me everything tool on the fly reprocessing Python binding to Oracle – 5LS ۷ Fileserver ۷ Parallelisation code ۷ Catalogue class/ associating HTM ۷ Federating with Oracle Streams User interfaces –Tell me everything tool on the fly reprocessing

AstroWise((G)A)VO Meeting 6 May 2004 Schedule Hardware –OmegaCEN: 6Tbyte + servers + 8 node cluster operational ۷ –OAC, USM, Terapix + CVS federation ۷ Software –Design – review Q ۷ –Design – formalized July 2003 ۷ –WFI +INT Prototype operational Q ۷ –Basic operations – Q ۷ –Evaluate and prepare for mass production 2004 –Qualify and populate 2005 –Deliver survey system – satellites Hardware –OmegaCEN: 6Tbyte + servers + 8 node cluster operational ۷ –OAC, USM, Terapix + CVS federation ۷ Software –Design – review Q ۷ –Design – formalized July 2003 ۷ –WFI +INT Prototype operational Q ۷ –Basic operations – Q ۷ –Evaluate and prepare for mass production 2004 –Qualify and populate 2005 –Deliver survey system – satellites