1. 10 June 2002Towards an International VO - Garching bei Munchen 1 ASTRO-WISE An Astronomical Wide-field Imaging System for Europe Konrad Kuijken, Edwin Valentijn Kapteyn Institute, Groningen

2. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 2 VST + OmegaCAM 6x WFI 1 o x 1 o 16k x 16k pixels

3. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 3 The Problem Many wide-field imaging instruments, vast amounts of data –E.g.: VST = Southern sky (30 min exp, 300 nights/y) in 3 years. Vast amount of data! 100 Tbyte Science on these data should be archive-based Handling of the data is non-trivial –Reduction –Calibration and re-calibration –Image comparisons and combinations –Working with source lists –Visualization ASTRO-WISE aim: –provide a system for maximal data handling flexibility

4. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 4 Concept solution Environment which –Has access to all raw and calibration data –Can run and modify reduction/calibration pipelines –Can run source extraction algorithms –Archives reduced data and source lists, or regenerates these dynamically –Can be federated to link different data centers This dynamical archive continuously grows as more raw data enter the system, and can be used for –`small’ or large science projects –generating and checking calibration data –exchanging methods and scripts A key functionality is the link back from source data to the original raw pixel data and calibration files

5. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 5 Example uses Deep multi-colour fields –No need to take all data in one campaign –Combine data of particular quality, assess results –Select sources, visualize interesting ones, … 1-in-1,000,000 events spurious or not? Large homogeneous surveys –Eg weak lensing maps, cluster searches, star counts Variability (source list- or pixel based) –Proper motions (asteroids, nearby stars) –Flux variations

6. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 6 Example uses Monitor instrument (calibration files) Planning observations –View quality of existing data –Build on what already exists, add more filters, more exposure time, better seeing, … All this can be done on the continuously growing archive

7. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 7 Philosophy System is not geared to a single data product, but to being a flexible tool. Not focused on making a single, homogeneous survey with static data release(s) - but could be used for this Optimise interaction between users and their data  Dynamic archive Geared to optical (IR) wide-field image data

8. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 8 Partners Distribute the load/environment over National centers, each serving their community Partners: –NL - Groningen (+ Leiden) –ESO – EIS –I – Napoli + –F - Terapix –D – USM + –UK - VISTA Edinburgh/Cambridge 1.5 MEuro grant from EU RTD programme

9. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 9 How? Procedurizing data taking at telescope for both science and calibration data Full integration with data reduction OmegaCAM FDR Data model (classes) defined for data reduction and calibration Design View pipeline as an administrative problem

10. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 10

11. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 11

12. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 12 Tbyte source lists- brains make the associations Link -lists as fast as possible

13. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 13 Share the load –Hardware: Beowulf processors – 32 (most cases) –Hardware: multi Terabyte disks (10 – 100) –Data reduction: derive calibration –Data reduction: run image pipeline (1 Mpx/s) –Archive: storage of images (100’s Tbyte) –Archive: storage of source parameters (1-10 Tbyte) Network speed free parameter: - from 5 Mb/s (24 hours/day provides full replication of all OmegaCAM data) - to 200 Mb/s no replication

14. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 14 Federate Calibration results Methods (pipelines) for processing calibration Raw data Reduced images Source lists - catalogues

15. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 15 Federation maintained by a single db Full history tracking of all input that went into result providing on-the fly reprocessing – tell me everything tools Dynamical archive - Context areas in object attributes –Project: eg Cal, Science, Survey, Personal –Owner: eg pipeline, developer, user –Strategy: Standard, Deep, Freq (monitoring) –Mode: Stare, Jitter, Dither –Time: time stamping Standards -classes/data model/ procedures -00 – inheritance/ persistency -Python scripts/ libraries Concepts Partitions Link to AVO VO table

16. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 16 DB – engine Oracle contract: – reference – licenses - consultancy Oracle 9i: -federation through “Advanced replication” to evolve into cross site links (=references= pointers) -Python I/F SQL – OCI- Oracle db -Full oo support + Python to db persistency -Terabyte scalability through “partitioned tables” -Administration tools –Interoperability: STREAMS connects to others: SyBase, MySQL

17. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 17 Intra-operability peer to peer CVS “Advanced Replication” evolving to pointers

18. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 18 WP s WP 1 – pipelines / same as ESO-DMD »Implement and operate WP 2 – visualization tools » Panorapix – fast Gbyte image display + true colour »Graphic cards – OpenGL - WP3 – federated db’s »CVS »Oracle WP4 – processing power 4 Beowulf centers WP5 – direct access 100 Tbyte storage WP6 – coordinate – AVO – CERN-HEP- Datagrid ASTRO-GRID

19. Astro-Wise Wide field imaging system for Europe 10 June 2002Towards an International VO - Garching bei Munchen 19 Schedule ADD – review Q2 2002- OK Basic operations – Q4 2003 Evaluate and prepare for mass production 2004 Qualify and populate 2005 Deliver survey system – satellites Links to LOFAR, JIVE,WHIRCAM, GAIA Personal Observatory