18 April 2007 Second Generation VLT Instruments 1 VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit

18 April 2007Second Generation VLT Instruments2 Introduction to CASU ► Small group within the Institute of Astronomy, specialising in survey astronomy. ► Pipeline reduction of imaging data  APM (Schmidt Plates 1 x 40k x 40k)  INT Wide Field Camera (4 x 2k x 4k)  ESO WFI on 2.2m at La Silla (8 x 2k x 4k)  MOSAIC-1 on KPNO 4m (8 x 2k x 4k)  MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k)  AAO WFI on AAT (8 x 2k x 4k)  CIRSI on INT (4 x 1k x 1k)  INGRID on WHT (1k x 1k)  UFTI on UKIRT (1k x 1k)  WFCAM on UKIRT (4 x 2k x 2k)

18 April 2007Second Generation VLT Instruments3 Introduction to VISTA and VDFS ► VISTA: 4m survey telescope on Paranal ► VIRCAM: The IR camera for VISTA  16 Non-buttable 2k x 2k detectors ► VDFS: PPARC funded facility to provide an end-to-end data-flow system for VISTA and WFCAM. ► Quality control and calibration pipelines  Paranal and Garching ► Science pipeline for full calibration of science data.  Cambridge ► Science archive acts as the point of access of the reduced data. Plus some further processing.  WFAU, Edinburgh

18 April 2007Second Generation VLT Instruments4 Data Flow ► Raw telescope data is assessed by the summit pipeline (QC1) ► Shipped to Garching (discs) ► Shipped to CASU for science reduction and calibration. (discs) ► Calibrated data shipped to Edinburgh for archiving (ftp)

18 April 2007Second Generation VLT Instruments5 IR Data Reduction Worries ► IR detectors are currently inherently more unstable than optical CCDs.  Some odd electronic effects  Much poorer cosmetic quality than CCDs ► Sky emission > 100x brighter than most objects  And it’s variable both spatially and temporally! ► Exposure times are short, so data rates are very high.  Gb/night expected for VISTA public surveys  Rice tile compression can save factors of 3-4 in 32 bit integer data

18 April 2007Second Generation VLT Instruments6 VDFS Pipeline Recipes ► Create master calibration frames (dark, twilight flats, confidence maps, etc) ► Linearity analysis ► Detector noise & dark current properties ► Persistence and crosstalk analysis ► Illumination correction analysis ► Full reduction recipes for standard star and programme fields

18 April 2007Second Generation VLT Instruments7 Summit & Garching Pipelines ► QC1 parameters  e.g. photometric zeropoints, astrometric fit quality ► Written using ESO qfits/CPL infrastructure  Both use the same software modules ► Amount of processing can be scaled down ► Calibration images (flats etc) from a master library.

18 April 2007Second Generation VLT Instruments8 Cambridge Pipeline ► Full reduction  e.g. better sky correction ► Results shipped to WFAU ► Some of the same CPL based modules reused. Some additional modules required.

18 April 2007Second Generation VLT Instruments9 Current Status of VISTA Pipelines ► Version 0.5 released to ESO in March ► Full compliment of recipes (almost!) ► PAE

18 April 2007Second Generation VLT Instruments10 The Decision To Use CPL (1) ► We were given the choice. ► We already had advanced pipeline software in our own environment for WFCAM and other IR instruments. ► First view of CPL and qfits was not favourable.

18 April 2007Second Generation VLT Instruments11 The Decision To Use CPL (2) ► CPL v2.0 was a great improvement ► CASU has an ‘ESO workstation’ to generate ‘data’ using the VIRCAM acquisition software ► Integration into Paranal and Garching systems greatly simplified by using CPL

18 April 2007 Second Generation VLT Instruments 12 Things I wish I had known…

18 April 2007Second Generation VLT Instruments13 My First Attempt At A Recipe ► INPUT:  FITS image of WFCAM data  Binary FITS table of astronomical objects  Binary FITS tables containing the 2MASS catalogue ► TASK:  Work out RA, Dec coverage of image using header FITS WCS keys  Extract 2MASS standards  Cross match standards to object catalogue  Fit a WCS  Write resulting WCS to FITS image

18 April 2007Second Generation VLT Instruments14 World Coordinate Systems ► CPL has no WCS facility  But it soon will

18 April 2007Second Generation VLT Instruments15 Input File Access ► CPL doesn’t give write access to input files.  cpl_frame, cpl_image  Data abstraction is not perfect  cpl recipes can’t be written to update a file

18 April 2007Second Generation VLT Instruments16 CPL vs FITS tables ► CPL does not allow for direct manipulation of FITS tables  cpl_table structures can be created from a FITS table.  But one must read the WHOLE table even though you might only be interested in a few rows or columns. ► Tables with many rows take too long to read  Break them up ► Tables with many character type entries cause the system to run out of memory pointers  Remove unnecessary character columns

18 April 2007Second Generation VLT Instruments17 User Documentation ► Reference manual, users guide, software developers kit ► User’s guide and SDK are helpful but limited in scope  Very few examples  Get a copy of a working CPL pipeline

18 April 2007Second Generation VLT Instruments18 QC ► Garching and Summit pipelines generate QC parameters  Provides data for QC trending to be done both on site and in Garching  The ‘ESO System’ is really two systems ► QC passed through FITS headers (Garching) ► QC passed through PAF files (Chile) ► Neither system appears to be documented. ► PAF files contents are not documented in CPL users or reference manuals. ► CPL has no standard class for passing QC parameters

18 April 2007Second Generation VLT Instruments19 File Tags ► DPR TYPE  OBSTYPE ► PRO CATG  Product type  No two recipes can produce the same value ► DO CATG  File type for files used as input to recipes  Raw frame DO CATG values must be unique to each recipe.  There are some rules on the values here.

18 April 2007Second Generation VLT Instruments20 Output File Names ► Recipe creates and names output files.  File names must be hard coded!  Several files of the same PRO CATG must be indexed in a predictable way. ► esorex can change the file name

18 April 2007Second Generation VLT Instruments21 Headers ► CPL cannot directly manipulate FITS headers  These must be read into a cpl_propertylist structure.