1. European Southern Observatory
Scoring: a novel approach towards automated certification of pipeline products
Reinhard Hanuschik, Wolfgang Hummel, Mark Neeser, Burkhard Wolff Data Processing and Quality Control Group
European Southern Observatory

2. Complexity and mass production
VLT: 11 instruments
VLTI: 2 instruments
2009/10: 2 survey telescopes
roughly 50 modes
about 150 data types
most of them pipeline-supported

3. Complexity and mass production
each data type or instrument component controlled by QC1 parameters
metrics for instrument performance extracted by data pipelines
stored in database
data frequency varies on timescales between a day and a semester

4. Complexity and mass production
two challenges …
… the complexity challenge!

5. Complexity and mass production
current VLT instruments:
avg: 30 GB/night
… the volume challenge!
VIRCAM:
OMEGACAM:
400 GB/night

6. The solution: The solution: scoring
scoring: well-established outside astronomy …
QC process:
measure quality
compare quality
assess quality: score and certify

7. Scoring: the solution measure quality: done by automatic procedures
pipelines
other QC procedures
implemented for CALIB data
progress on SCIENCE data

8. Scoring: the solution compare quality: trending
first step towards assessment:
same/different behaviour as others?

9. Scoring: the solution assess quality: scoring
compare new result to trending
pick out outliers and flag them

10. Scoring: the solution required: thresholds
statistical thresholds: e.g. +/- 3 sigmas + independent from specs - sensitivity variable control charts
specified thresholds: e.g. lower/upper level + stable thresholds - requires careful configuration and thinking
saturation
noise

11. Scoring of pipeline products
each new pipeline product is scored
most relevant QC1 parameters: compare to trending, score as
OK: 0
NOK: 1 (outlier)
count scores, assign total score, e.g.

12. Scoring of pipeline products
score report:

13. Scoring of pipeline products
score reports for multi-detector instruments (CRIRES: N=4)
… detailed information on demand!

14. Scoring of instrument health
all scores go into database
re-arrange them by instrument properties
detector health
calibration lamp performance
system efficiency
score-based instrument QC
cal file1 scores
FLAT
instrument scores
cal LAMPs
cal file2 scores
ARC

15. Scoring of instrument health
example: detector properties for GIRAFFE
instead of …

16. Scoring of instrument health
and finally:

17. Conclusions scoring: automatic flagging of outliers
useful to auto-certify pipeline products
quick-look monitoring of instrument health
powerful concept to handle large and complex data sets
hierarchical approach: from global overview (instrument score) to QC1 parameter per product and detector (6 levels)