SDD/PSD P.Ballester INS Software Workshop - 10 Oct Data Processing Day 9:00 Data Flow System Deliverables and Integration 10:00 Coffee Break 10:15 The ESO Common Pipeline Library 11:00 The ESO-Reflex Environment 12:00 Lunch break 13:00 CPL Tutorial 15:00 Coffee Break / End of Workshop

SDD/PSD P.Ballester INS Software Workshop - 10 Oct SDD / Pipeline Systems Department P.Ballester K.Banse S.Castro L. de Bilbao A. Gabasch E. Garcia C. Izzo Y. Jung J. Larsen H. Lorch L. Lundin A. Modigliani R. Palsa D. Petry K. Shabun J. Vinther

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Data Flow System Deliverables and Integration

SDD/PSD P.Ballester INS Software Workshop - 10 Oct DFS Deliverables (1): Data Reduction Library PSD Paranal Science Operations INS Commissioning Team Data Flow Operations DICB/Archive Public Release User Community

SDD/PSD P.Ballester INS Software Workshop - 10 Oct DFS Deliverables (2): ETC/ Observation Preparation Tools

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Talk Outline Data Reduction Library Calibration cascade Modular design Data Products Science-grade data products Simulations and validation Project Organisation Project phases and timeline DFS Integration Acceptance tests DFS infrastructure

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Data Reduction Library Calibration Cascade Modular Design

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Data Reduction Correct for detector and instrument effects Correct for atmospheric effects Separate science data from noise and background Astronomical Calibrators (Position, Spectral flux, Diameter, etc..) Instrumental Calibrators (Internal sources)

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Calibration Cascade Detector Bias Instrument and Detector Sensitivity Photometry Standard Star

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Shift & Add: HAWK-I Modular Design -Independent modular recipes -Integrated pipeline recipe

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Shift & Add: HAWK-I Reflex Workflow

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Shift & Add: Demonstration 1. Raw image (sky = 10,000 star = 1) 2. Dome flat 3. Estimating the sky from N jittered exposures 4. Subtracting the sky and correcting for flat-field 5. Co-adding the sky corrected images 6. Astrometry + Photometry !!

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Science-Grade Data Products Accuracy Recipes Design Robustness, Fault tolerance Validation

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Science-Grade Data Products Calibrated in physical units with error estimates No residual systematic error S/N close to the achievable optimum (e.g. ETC prediction) OB combination E.g. fully calibrated and mosaiced images with error bars Standard data formats SGDPs are independent of the science goals

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Understanding the Instrument Signature (1) UVES Adaptive Optimal Extraction UVES archive data reprocessing: Robust to CCD defects S/N adaptive optimal extraction Analytic profile for low S/N Non-parametric profile for high S/N Ripple scale QC parameters

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Understanding the Instrument Signature (2) MIDI Visibility Spectra

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Designing Pipeline Recipes Image Model Optical/Detector model Assumptions on stability/reproducibility Input Data Signal-to-noise Missing data Contamination Stability (Atm., Ins.) Multiplex/Volume Reduced Data Random error estimates Systematic errors Inverse Solution Objective function No assumptions on scientific program Optimization strategy Computational complexity Robustness, Fault tolerance Pipeline Recipe Default parameters Fixed calibration data Accuracy Throughput Robustness

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Ar lamp - 1” slit - from P. Spano et al. in Merate, July 2007 First light in the lab (VIS arm)

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Using Simulations Ar lamp - 1” slit - simulated by P. Bristow (ESO)

SDD/PSD P.Ballester INS Software Workshop - 10 Oct State-of-the-art DRS pipeline Simulated image, multi-pinhole VIS: curved orders, line tilt varying along order

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Robustness, Fault Tolerance Physical Model Pattern Matching

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Project Organisation Scheduling Validation and Acceptance

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Phases Phase A to FDR Documentation/ Prototyping2-3 years FDR to PAE Development/ Simulation2-3 years PAE to SV and SOP Deployment 6-12 months Operational PhaseMaintenance10 years

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Document and Technical Documentation 1618 Document (currently v.2.0) CPL Documentation DICB / VLTI DICD Gasgano Reflex

SDD/PSD P.Ballester INS Software Workshop - 10 Oct DFS Deliverables: 1618 Template Schedule

SDD/PSD P.Ballester INS Software Workshop - 10 Oct DFS Deliverables: 1618 Template Schedule (cont’d)

SDD/PSD P.Ballester INS Software Workshop - 10 Oct From Phase A to FDR Phase A and PDR Preparation Reuse existing documents as templates Identify extra needs: observation preparation, visualisation, link to the data analysis? Learning CPL before the FDR First recipes in CPL, coding standards, memory management, … Review existing pipelines, develop prototypes Coding starts officially only after the FDR FDR Preparation Prototype data reduction algorithms (test and simulated data) New document: Validation and Test Plan Observatory Pipeline is template based —Further processing requirements ? —Data combination, interactive processing

SDD/PSD P.Ballester INS Software Workshop - 10 Oct From FDR to Science Verification FDR to PAE Plan more import validation effort for the first recipes, In general, plan enough time for testing and finalising Regular intermediate software releases (3 to 6 months), including test reports Keep improving simulated or adapted data, in sync with instrument schedule PAE Plan for the complete set of recipes to be ready at PAE Validated data reduction algorithms (using laboratory and simulated data) Have ready a few alternative calibration/reduction methods Commissioning to Science Verification Test and validate on instrument and sky data Identify and solve the unexpected problems

SDD/PSD P.Ballester INS Software Workshop - 10 Oct DICB Approval DPR Keyword Values Ideally, classification rules are only based on DPR keywords Valid values are listed in the DICB 4.0 documents Submit proposal to DICB by FDR before using new values DICB Validation No redefinition of existing keywords (database at Keyword names should not be too long Avoid underscores and special characters Multi-HDU files Always the same structure of data for a given data type Data extensions come first, auxiliary and optional tables afterward

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Acceptance Tests Usage of CPL recipe template Following CPL coding standard Usage of external libraries Namespace protection Execution Tests Completeness of the set of recipes and DRL functions Availability/representativity of test data Proper execution of recipes Generation of products Memory leaks Unit tests Documentation Detailed Validation Correctness of results Validation of input FITS compliance User-friendly documentation Data reduction cascade Unit tests Performance and Portability Execution speed Standard platforms: Scientific Linux

SDD/PSD P.Ballester INS Software Workshop - 10 Oct DFS Integration Observatory Pipeline Quality Control Pipeline Desktop science-grade data reduction Maintenance

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Data reduction environments Observatory Pipeline On-the-fly data processing (event driven) Template-based processing Static calibration database (only certified products are used) Quality Control Pipeline Batch processing of complete data sets (all science and calibration data produced by one ESO instrument in one night) Best available calibrations are used => data must be organized according to the Calibration Cascade Desktop science-grade data reduction Modular and additional recipes are avilable Several front-ends for scripting (esorex), browsing (Gasgano), interactive (Reflex)

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Observatory Pipeline Instrument Raw Data Archive PIPELINEOFFLINE FurtherAnalysis Raw data Processed data On-Line Archive Sytem Shipping Shipping

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Observatory Pipeline: Workstation Pipeline workstation Data Organizer Reduction Block Scheduler Configuration files Calibration database Data Reduction System PRODUCTS Instrument Package Common Pipeline Library Archive Quality control Data arriving from the instrument Reduction Block  List of the raw frames  List of the calibration data  Name of the products  DRS recipe to apply

SDD/PSD P.Ballester INS Software Workshop - 10 Oct OCA Rules if DPR.CATG=="CALIB" and DPR.TYPE=="LAMP,WAVE" then { DO.CLASS = "ARC_SPECTRUM"; RAW.TYPE = "WAVE"; } 1.C LASSIFICATION 2.O RGANIZATION select execute(GI_WAVE_CALIBRATION) from inputFiles where RAW.TYPE=="WAVE“ group by TPL.START 3.A SSOCIATION action GI_WAVE_CALIBRATION { select file as MASTER_BIAS from calibFiles where PRO.CATG=="MASTER_BIAS" and inputFile.DET.WIN1.BINX==DET.WIN1.BINX; select file as GRATING_DATA from calibFiles where PRO.CATG=="GRATING_DATA" and inputFile.INS.GRAT.NAME==INS.GRAT.NAME; recipe giwavecalibration { } product mflat { PRO.CATG="MASTER_FLAT"; } }

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Quality Control Log Files 12:21:51>-START GROUP / Start [AMBER] 12:21:51> ARCFILE = 'AMBER T12:16: fits' [AMBER] 12:21:51> TELESCOP = 'NOT_SPECIFIED' / Telescope [AMBER] 12:21:51> INSTRUME = 'AMBER' / Instrument name [AMBER] 12:21:51> OBSERVER = 'UNKNOWN' / Observer name [AMBER] 12:21:51> PIPEFILE = 'p2vm.fits' / Filename of data product [AMBER] 12:21:51> INS GRAT1 NAME = 'GHR' / Grating common name. [AMBER] 12:21:51> INS GRAT1 RESOL = ; / Encoder resolution [Enc/deg]. [AMBER] 12:21:51> INS GRAT1 WLEN = ; / Grating central wavelength [nm]. [AMBER] 12:21:51> INS GRAT1 ZORDER = 40319; / Grating zero order position [Enc]. [AMBER] 12:21:51> INS GRIS1 NAME = 'NAR_SLT' / OPTIi name. [AMBER] 12:21:51> INS GRIS2 NAME = '3T_K' / OPTIi name. [AMBER] 12:21:51> INS MODE = '3Tstd_High_K_1_2.365' / Instrument mode used. [AMBER] 12:21:51> PRO DID = 'ESO-VLT-DIC.PRO-1.15' / Data dictionary for PRO [AMBER] 12:21:51> PRO CATG = 'P2VM_REDUCED' / pipeline product category [AMBER] 12:21:51> PRO TYPE = 'REDUCED' / Product type [AMBER] 12:21:51> PRO REC1 ID = 'amber_p2vm' / Pipeline recipe (unique) identifier [AMBER] 12:21:51> PRO REC1 DRS ID = 'cpl-3.0' / Data Reduction System identifier [AMBER] 12:21:51> PRO REC1 PIPE ID = 'AMBER/2.3.2' / Pipeline (unique) identifier [AMBER] 12:21:51> PRO REC1 RAW1 NAME = 'AMBER T12:13: fits' / File name of raw frame [AMBER] 12:21:51> PRO REC1 RAW1 CATG = 'AMBER_3WAVE' / Frame category of raw frame [AMBER] 12:21:51> PRO DATANCOM = 14; / Number of frames combined [AMBER] 12:21:51> PRO REC1 CAL1 NAME = 'FlatFieldMap.fits' / File name of calibration frame [AMBER] 12:21:51> PRO REC1 CAL1 CATG = 'AMBER_FLATFIELD' / Frame category of calibration frame [AMBER] 12:21:51> PRO REC1 PARAM1 NAME = 'dummy' / Name of recipe parameter [AMBER] 12:21:51> DET NTEL = 3; / Number of telescopes [AMBER] 12:21:51> QC P1 OFFSETY = 0.02; / Offset wavelength calibration [AMBER] 12:21:51> QC P2 OFFSETY = 0.05; / Offset wavelength calibration [AMBER] 12:21:51> QC P3 OFFSETY = 0.03; / Offset wavelength calibration [AMBER] 12:21:51>-STOP GROUP / Stop [AMBER]

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Garching: QC Main Tasks Create master calibrations Derive and trend QC parameters Create science products (Service Mode) Prepare data packages (Service Mode) Pipelines & QC parameters: requirements & testing Customers Paranal Science Operations ESO community (PIs, archive users) Home Page:

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Information by Instrument Similar pages for every VLT/VLTI instrument

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Public Releases Linked from all ESO instrument pages Release package, documentation, demonstration data 13

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Desktop Data Reduction: Gasgano VLT interactive data organisation tool – FITS file browsing – Grouping – Classification Interactive front-end – Interface to CPL plugins – Interface to vizualisation tools Features – Java language – FITS format

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Desktop Data Reduction: Reflex -CPL recipes -External tools -Python scripts -Visualisation -Beta users and internal evaluation

SDD/PSD P.Ballester INS Software Workshop - 10 Oct Pipeline Maintenance DFS Tickets Issued by PSO, DFO, INS, and the user community Pipeline priority meetings Bi-yearly meetings with representation of INS, DFO, PSO, SDD General issues and for each instrument closed, in process, open tickets Priority setting for further pipeline development Instrument Evolution Commissioning of new modes Detector upgrades

SDD/PSD P.Ballester INS Software Workshop - 10 Oct