1. OmegaCAM: The 16k x 16k Survey Camera for the VST Calibration, Data Analysis Strategy and Software Calibration, Data Analysis Strategy and Software Erik R. Deul Konrad Kuijken Edwin A. Valentijn Erik R. Deul Konrad Kuijken Edwin A. Valentijn

2. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 2 People involved The Netherlands Kapteyn Institute: J-W. Pel, K. Begeman, D.R. Boxhoorn, E. Valentijn, K. Kuijken Sterrewacht Leiden: R. Rengelink, E.R. Deul Germany Universitäts-Sternwarte München: R. Bender, L. Greggio, R. Häfner, U. Hopp, H. Kravkar, W. Mitsch, B. Muschielok, M. Neeser, R. Saglia Universitäts-Sternwarte Göttingen: R. Harke, H. Nicklas, W. Wellem Sternwarte der Universität Bonn: K. Reif Italy Astronomical Observatory of Capodimonte - Napoli: E. Cascone Osservatorio Astronomico di Padova: A. Baruffolo, E. Cappellaro, E. V. Held, H. Nazaryan, G. Piotto, H. Navarsadyan, L. Rizzi ESO D. Baade, A. Balestra, J-L. Beckers, C. Cavadore, C. Cumani, F. Christen, S. D'Odorico, S. Deiries, N. Devillard, C. Geimer, N. Haddad, G. Hess, J. Hess, O. Iwert, H. Kotzlowski, J-L Lizon, A. Longinotti, W. Nees, A. Renzini, J. Reyes Moreno, G. Sikkema, M. Tarenghi The Netherlands Kapteyn Institute: J-W. Pel, K. Begeman, D.R. Boxhoorn, E. Valentijn, K. Kuijken Sterrewacht Leiden: R. Rengelink, E.R. Deul Germany Universitäts-Sternwarte München: R. Bender, L. Greggio, R. Häfner, U. Hopp, H. Kravkar, W. Mitsch, B. Muschielok, M. Neeser, R. Saglia Universitäts-Sternwarte Göttingen: R. Harke, H. Nicklas, W. Wellem Sternwarte der Universität Bonn: K. Reif Italy Astronomical Observatory of Capodimonte - Napoli: E. Cascone Osservatorio Astronomico di Padova: A. Baruffolo, E. Cappellaro, E. V. Held, H. Nazaryan, G. Piotto, H. Navarsadyan, L. Rizzi ESO D. Baade, A. Balestra, J-L. Beckers, C. Cavadore, C. Cumani, F. Christen, S. D'Odorico, S. Deiries, N. Devillard, C. Geimer, N. Haddad, G. Hess, J. Hess, O. Iwert, H. Kotzlowski, J-L Lizon, A. Longinotti, W. Nees, A. Renzini, J. Reyes Moreno, G. Sikkema, M. Tarenghi

3. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 3 Detectors Science array 1 x 1 degree, 32 CCDs –15 m pixels – 0.21 arcsec/pixel –Marconi (former EEV) 2k x 4k –16k x 16k pixels Auxiliary CCD’s – 4 CCDs –For guiding –Image analysis Science array 1 x 1 degree, 32 CCDs –15 m pixels – 0.21 arcsec/pixel –Marconi (former EEV) 2k x 4k –16k x 16k pixels Auxiliary CCD’s – 4 CCDs –For guiding –Image analysis

4. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 4 Filters Primary set –Sloan u’, g’, r’, i’, z’ –Johnson B, V –Narrow-band e.g. H up to 8000 km/s –Composite u’,B,V,i’ in four quadrants User filter Primary set –Sloan u’, g’, r’, i’, z’ –Johnson B, V –Narrow-band e.g. H up to 8000 km/s –Composite u’,B,V,i’ in four quadrants User filter More details see Harald Nicklas [4836-34]

5. 28/08/20025 VST construction see [4836-09] Mancini Details instrument control see [4848-10] Baruffolo VST construction see [4836-09] Mancini Details instrument control see [4848-10] Baruffolo

6. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 6 Wide Field Imaging Science Provide targets for VLT 2/3 of time through ESO’s OPC Individual programs –Supernovae, Lensing, Kuiper belt objects, Gamma ray, bursts, Microlensing, Brown dwarfs, High proper motion objects, Galactic halo objects, Quasars, AGNs Sky Surveys Long term archival research (10 yr mission) Science Cases –Finding exceptional single, rare objects –Statistics on large samples of objects Provide targets for VLT 2/3 of time through ESO’s OPC Individual programs –Supernovae, Lensing, Kuiper belt objects, Gamma ray, bursts, Microlensing, Brown dwarfs, High proper motion objects, Galactic halo objects, Quasars, AGNs Sky Surveys Long term archival research (10 yr mission) Science Cases –Finding exceptional single, rare objects –Statistics on large samples of objects

7. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 7 Large Data Volume Wide-field imaging instruments, vast amounts of data –E.g.: VST = Southern sky (30 min exp, 300 nights/y) in 3 years. Large amount of data! 100 Tbyte Wide-field imaging instruments, vast amounts of data –E.g.: VST = Southern sky (30 min exp, 300 nights/y) in 3 years. Large amount of data! 100 Tbyte Science can only be archive-based Handling of the data is non-trivial –Pipeline data reduction –Calibration and re-calibration –Image comparisons and combinations –Working with source lists –Visualization Handling of the data is non-trivial –Pipeline data reduction –Calibration and re-calibration –Image comparisons and combinations –Working with source lists –Visualization ESO compliant ESO compliant } }

8. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 8 Concepts for solution Environment that provides systematic and controlled –Access to all raw and calibration data –Execution and modification reduction/calibration pipelines –Execution of source extraction algorithms –Archiving reduced data and source lists, or regenerates these dynamically –Can be federated to link different data centers Environment that provides systematic and controlled –Access to all raw and calibration data –Execution and modification reduction/calibration pipelines –Execution of source extraction algorithms –Archiving reduced data and source lists, or regenerates these dynamically –Can be federated to link different data centers Dynamical archive continuously grows, can be used for –small or large science projects –generating and checking calibration data –exchanging methods, scripts and configuration Dynamical archive continuously grows, can be used for –small or large science projects –generating and checking calibration data –exchanging methods, scripts and configuration Key functionality –Link back from source data to the original raw pixel data and calibration files Key functionality –Link back from source data to the original raw pixel data and calibration files

9. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 9 How to use this Deep multi-color fields –No need to take all data in one campaign –Combine data of particular quality, assess results –Select sources, visualize interesting ones, … 1-in-1,000,000 events spurious or not? Deep multi-color fields –No need to take all data in one campaign –Combine data of particular quality, assess results –Select sources, visualize interesting ones, … 1-in-1,000,000 events spurious or not? Large homogeneous surveys –E.g. weak lensing maps, cluster searches, star counts Large homogeneous surveys –E.g. weak lensing maps, cluster searches, star counts Variability (source list - or pixel based) –Proper motions (asteroids, nearby stars) –Flux variations Variability (source list - or pixel based) –Proper motions (asteroids, nearby stars) –Flux variations Monitor instrument (calibration files) Planning observations –View quality of existing data –Build on what already exists, add more filters, more exposure time, better seeing, … Planning observations –View quality of existing data –Build on what already exists, add more filters, more exposure time, better seeing, …

10. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 10 Solution Procedurizing –Data taking at telescope for both science and calibration data –Full integration with data reduction –Design –Data model (classes) defined for data reduction and calibration –View pipeline as an administrative problem Procedurizing –Data taking at telescope for both science and calibration data –Full integration with data reduction –Design –Data model (classes) defined for data reduction and calibration –View pipeline as an administrative problem

11. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 11 Observing Modes Dither matching max. gap between arrays ~400 pixels –N pointings (N=5 is standard) –nearly cover all gaps in focal plane and maximizes sky coverage –the context map will be very complex –couple the photometry among individual CCDs. Dither matching max. gap between arrays ~400 pixels –N pointings (N=5 is standard) –nearly cover all gaps in focal plane and maximizes sky coverage –the context map will be very complex –couple the photometry among individual CCDs. Jitter matching the smallest gaps in CCDs ~5 pixels –optimizes for maximum homogeneity of the context map –observations for which the wide CCD gaps are not critical –all data from single sky pixel originates from single chip Jitter matching the smallest gaps in CCDs ~5 pixels –optimizes for maximum homogeneity of the context map –observations for which the wide CCD gaps are not critical –all data from single sky pixel originates from single chip Stare reobserving fixed pointing positions multiple times –main workhorse monitoring instrument and optical transients. Stare reobserving fixed pointing positions multiple times –main workhorse monitoring instrument and optical transients. SSO observing Solar System objects –non-siderial tracking and the auto guiding switched off. SSO observing Solar System objects –non-siderial tracking and the auto guiding switched off.

12. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 12 Observing Strategies Standard –Single observations (one observing block) Deep –Long, multiple integrations –Selected atmospheric conditions –Several nights Frequent –Monitors same field –Timescales from minutes to months (overriding) Mosaïc –Maps areas of sky > 1 o Standard –Single observations (one observing block) Deep –Long, multiple integrations –Selected atmospheric conditions –Several nights Frequent –Monitors same field –Timescales from minutes to months (overriding) Mosaïc –Maps areas of sky > 1 o

13. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 13 Calibration procedures Sanity checks Quality control Calibration procedures Calibration procedures Image pipeline Source pipeline

14. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 14 Science Observations Photometric pipeline Bias pipeline Flatfield pipeline Image pipeline Source pipeline

15. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 15 Monitoring Photometric Calibration

16. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 16 Share the load Processing –Hardware Beowulf processors – 32 (most cases) Multi Terabyte disks (10 – 100) –Data reduction Derive calibration Run image pipeline (1 Mpx/s) Processing –Hardware Beowulf processors – 32 (most cases) Multi Terabyte disks (10 – 100) –Data reduction Derive calibration Run image pipeline (1 Mpx/s) Archiving –Storage Images (100’s Tbyte), Calibration files (10 Tbyte) Source parameters (1-10 Tbyte) –Federate (network speed) 5 Mb/s (24 hours/day) full replication 200 Mb/s no replication, on-the-fly retrieval

17. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 17 Contents of federation Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Coadded images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects Raw data –Observed images –Ancillary information Calibration results –Calibration files time stamped Reduced images –Single observation –Coadded images Software –Methods (pipelines) for processing calibration –Configuration files Source lists – catalogues –Extracted source information –Associated among different data objects

18. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 18 Concepts of federation Federation maintained by a single database Full history tracking –of all input that went into result –providing on-the fly reprocessing Dynamical archive - Context as object attributes –Project: Calibration, Science, Survey, Personal –Owner: Pipeline, Developer, User –Strategy:Standard, Deep, Freq (monitoring), Mosaïc –Mode: Stare, Jitter, Dither, SSO –Time: Time stamping Software standards –Classes/data model/procedures –00 – inheritance/ persistency –Python scripts/ c-libraries Federation maintained by a single database Full history tracking –of all input that went into result –providing on-the fly reprocessing Dynamical archive - Context as object attributes –Project: Calibration, Science, Survey, Personal –Owner: Pipeline, Developer, User –Strategy:Standard, Deep, Freq (monitoring), Mosaïc –Mode: Stare, Jitter, Dither, SSO –Time: Time stamping Software standards –Classes/data model/procedures –00 – inheritance/ persistency –Python scripts/ c-libraries

19. 28/08/2002SPIE Conference [4836] Survey and Other Telescopes Technologies and Discoveries 19 Schedule Hardware –Dome/Telescope erected at location –Camera on telescope Q1 2004 –First run: Jan 2004 –Second run: Mar 2004 Software –Design – review Q2 2002- Done –Basic operations – Q4 2003 –Evaluate and prepare for mass production 2004 –Qualify and populate 2005 –Deliver survey system – satellites Hardware –Dome/Telescope erected at location –Camera on telescope Q1 2004 –First run: Jan 2004 –Second run: Mar 2004 Software –Design – review Q2 2002- Done –Basic operations – Q4 2003 –Evaluate and prepare for mass production 2004 –Qualify and populate 2005 –Deliver survey system – satellites