raw data recipes SGDP Science Grade Data Products Wolfram Freudling DPD / SDP Group

Science grade data products (SGDPs) are data products which can be used as-is to extract scientific conclusions, or to carry out quantitative measurements. Instrument signature removed physical units signal-to-noise ratio close to the optimum error estimates all aspect of data taking and processing fully documented

Data Product Survey Nov 2008 About 50% overlap with DFS tickets

Proposed Projects

Science Ready Pipelines as reported by Survey CRIRES LS Spectroscopy GIRAFFE/MEDUSA Hawk-I UVES VISIR

address issues which apply to several instruments Multi-OB image combination Telluric lines tool illumination correction distortions background subtraction detector “monitoring”  Austrian contribution

Looking for science-ready instrument to develop Reflex workflow Hawk-I: – Major problem with background subtraction => fixed – Distortions: not enough calibration data needed recipe to analyze distortions distortions not stable => Austrian in-kind

Looking for science-ready instrument to develop Reflex workflow Hawk-I: – Major problem with background subtraction => fixed – Distortions: - not enough data - needed recipe to analyze distortions - distortions not stable => Austrian in-kind UVES: – error-bars – Combination of calibrations – CR-like artefacts => most issues fixed draft of report by Daniel Bramich

Extrapolation: Every instrument needs substantial work before data products can be considered science-grade Many easy to spot problems Problems often involve both calibration data and pipeline Bottle-neck is the resources available for validation and verification Only 2.5 astronomers work in SDP group => need IOT involvement Conclusions so far

New process to set priorities

Why? Tickets driven development not good way to set priorities Rush to submit something to pipeline priority meeting PSD resources are spread too thin: – do fewer things with more resources Projects are too software-centric – full support of projects by IOT/SDP Goals of projects are ill-defined – start with good specifications, test plan

“Proposals should be articulated, comprehensive, and set some specific measurable goals to improve data quality. “ DP B Proposals will be assessed by Data Products Board. Instrument upgrades should use the same route. SDP Proposals

The Board The Data Product Board is chaired by the head of SDP meets twice per year, monthly progress meetings Members: IOT Coordinator, head of LPO/PSO, the head of DMO/DPD, head of SDD/PSD, representative from INS/IPD First meeting: Feb 2011

but... IOT/IS does not have the time, resources, motivation... to write a SDP proposals It is de-motivating if IOT proposal needs DPB approval We will not be able to process some data if operational critical tickets are not addressed right away I prefer to talk directly to software developer We should work only on projects which we can complete. Approved projects get lots of resources. We need to use our resources efficiently. This is understood and accepted.

What should be in a proposal / data products requirements Specific requirements for data quality Proposal is not compilation of tickets Separate requirements from implementation suggestions Prototypes for algorithms are great, but... Detail resources

Effect on Pipeline Development All pipeline development frozen be default Tickets should be submitted but will not be attended immediately. Exceptions have to be approved on a case- by-case basis “new” pipelines under development not affected.

Pipelines not frozen VIMOS: support for upgrade XSHOOTER: still in development for several modes VISIR: basic support for the new detector only; additional development must go through an enhancement proposal OMEGACAM: support for commissioning VIRCAM: pipeline still in development for several modes PRIMA: still in development

Questions? Comments?