1. 1 The EIS Experience: Lessons Learned May 12, 2005

2. 2 EIS 1996-2004 Original mission: –Limited-term experimental project –Preparation of data sets for VLT commissioning and early operation –Coordination with ESO community (SWG, visitors) –Train & disseminate tools as possible In 1999 mission expanded –Support long-term public surveys (1997-present; 8 years) –Develop infrastructure for public surveys –Develop image processing engine –Develop survey software system –Ramp-up to VST and VISTA Background Mission: R&D & operation; science & software; but...

3. 3 Community Participation Community Participation OPC (approval) Survey Working Group (design, oversight) –29 participants (2 committees) –24 institutes – 7 member states Visitor Program (operation/development) –43 participants –15 institutes –7 member states Full & broad community involvement in every aspect of the project

4. 4 EIS Team responsibilities: Observations (preparation/observing) Software Development: –Infrastructure (GUIs, database, architecture) –Image processing engine –Scientific algorithms –System integration & tests Hardware procurement Preparation, verification & delivery of survey products WEB development & maintenance Publications & reports Recruitment of visitors & administration of intense visitor program Composition: –5-6 FTE/year (astronomers & software developers)

5. 5 EIS Surveys

6. 6 Summary of surveys (07/97- ) surveys 9 surveys in < 6 years (FORS, FLAMES & VIMOS) 13 strategies 3 Telescopes (NTT, VLT, 2.2m) 5 imagers (EMMI, SUSI2, SOFI, ISAAC, WFI) 22 filters 240 nights in visitor mode alone done by the team 73,000 frames of raw data; 37,000 science exposures –WFI 7,000 science; 23,000 total –SOFI 18,200 science; 31,000 total –ISAAC 11,800 science; 21,000 total 27 public releases (several data products, software, zeropoints)public releases Observations still ongoing (SOFI 09.04; WFI: 02.05)

7. 7 Phase-1 ( 03.97-10.99 ) Best-effort Simple image reduction pipeline using –Available packages (IRAF, Eclipse) –Tools (Drizzle, LDAC, SExtractor) –Wrapper: shell scripts Limited software development (adaptations, bug-fixing, small-scale new developments) Help from experts in-house and in the community

8. 8 Limitations of best-effort approach: Problems with the data: –Calibration (loss of flux in 1998 release associated to Jitter) –Serious problems with the astrometric calibration of WFI –Rapid increase ( > 6 x) in data volume in 1999 –Reductions mostly manual; no history Over-reliance on a few people making operation vulnerable & fatigue Departure of key team members => 6 months interruption in reductions & development Different environments for optical and infrared Unsuitable hardware & software

9. 9 April ‘99April ‘02April ‘04 SOFTWARE Phase-2 Phase-1 EIS Data volume: time evolution Project split into 2 phases

10. 10 Lessons learned Best-effort approach one-off (not 24 times in 6 years)  unsustainable in the long-term  error-prone, hard to recover  disruptive, leaves no legacy  impedes progress of development (developers become operators) Need of framework and stable core group to preserve know-how and inherit code Resource-limited operation requires large degree of automation Handling large amount of data/information big challenge in survey context (differs from data-in/data-out problem) Requirement: 1) develop new image processing code 2) develop integrated reduction system

11. 11 Procedures standardized and accessible via GUIs Access to data/information transparent to user (SE/DAL) Integrated environment (CVS, ARS, Database, Web) Common optical/IR image processing engine (90,000 lines of C-code) System Wrapper (Python > 400,000 lines of code) XML technology (configuration; logs; Web; database contents) Self-describing products with quality parameters Uniqueness, versioning and history of products EIS data reduction system (06.00-09.04) Medium-size project (by industrial standards) done by non-specialists

12. 12

13. 13 Releases Maximum interval between end of survey and release < 3 years Products –Night (XMM, DPS, PF) –Stacks –Mosaics –Source catalogs (SExtractor, DAOPHOT) –Catalogs of clusters of galaxies, quasars, low-mass stars –WFI zero-points for 150 nights over 5 years –WFI data covering 29 square degrees Highlights 2004 –Release of EIS/MVM code –9 releases in 4 months; 11 releases in 2004 (1 release/month) –Data releases with product logs and READMES –7000 ISAAC frames; 3000 WFI frames

14. 14 Raw data –Requests: 1367 –Products: 84589 –Volume: 6.1Tb Survey products –Requests: 672 –Products: 9292 –Volume: 0.93 Tb Software: ~ 62 users Total of last 46 months –Requests: 2039 (44/mo) –Products: 93851 (2040/mo) –Volume: 7 Tb Over project lifetime –27 releases –40 requests/mo –Products: > 100,000 –12,500 prod/year; 34 prod/day Data Request Statistics

15. 15 Project Legacy High-performance, instrument-independent image processing pipeline (tested for all ESO imagers; publicly available) Integrated, end-to-end data reduction system to monitor & reduce multiple surveys from a single desktop Survey infra-structure (WG, database, Web interface, release) Blueprint for a modern data reduction & analysis system System easily adaptable for different applications

16. 16 Summary Over 25 man-years of development System supports –optical/infrared, single/multi-chip instruments –configurable workflows for un-supervised operations System consists of several pipelines –Image processing –Photometric calibration –Stack & Mosaics –Catalog (DAOPHOT, SExtractor) –Science applications (plug-ins) Mature, extensively-tested system before UKIDSS, VST and VISTA commissioning Work in progress EIS Data Reduction System