Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Improved and automated QC process Reinhard Hanuschik and the DPQC group fast data transfer: July 2008 full data stream VLT and VLTI (ex burst-mode data) > 50% within 10 minutes new ways in data DPQC impact on shared QC 2 main goals: calibration completeness calibration quality

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Mission #1: calibration completeness (“save last night”) calibrations incomplete: science OBs to be repeated focus on calibration plan: –known, predictable, configured –can be checked automatically calChecker 1. Setting the stage

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Mission #2: calibration product quality (“save next night”) scores combining Health Check monitor and scores: quick-look HC monitor

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 data acq (Paranal) data in Garching opslog any problem not discovered on site: up to 10 days affected strong need for 2 QC processes: daytime astronomer Paranal (preliminary), QC scientist Garching (final) basic real-time QC with opslog data (after ~ 10 days) until 2008:

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 data acq (Paranal) data in Garching now (2009): fast data delivery fast QC feedback improved QC feedback –measure quality (QC1 parameters) –compare quality (trending) scoring –rate quality (scoring) new!

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, evaluation for 7 last days (columns) and all science setups (rows) driven by science data checks: –all science setups for a given instrument –checks calibration plan –knows about validity and required number of files  life exampleslife examples 2. calChecker

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 evaluates each box in colors: –green (OK): all calibrations complete and within validity (e.g.: 5 BIAS & 3 FLATs & 1 ARC & 1 STD within validity) –yellow (NOK): calibrations complete but some outdated (e.g.: as above but 3 FLATs are 2 days old) –red (MISS): calibrations incomplete, at least one missing (e.g.: 5 BIAS & 3 FLATs ok, 1 ARC & 1 STD missing) missing or outdated calibrations are indicated

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 links to details –for each box: complete list of science files –prog_id, obs_id etc. for easy reference –OB grade and comment: useful for analysis

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 ANALYSIS –QC scientist to provide analysis/comments, but this requires interaction (  not always available!) –may override tool NOK/MISS results BOCAL, EOCAL –signal about daytime calib queue –calChecker displays “pending”, “ongoing”, “finished” for current date

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 called: once per hour by cronjob –has update information, knows about its age don’t hesitate to contact QC!

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 documentation association rules contact planned: –include ticket mechanism (PSO scheme) –memory function for long-term calibrations (once a month etc.) and HC calibrations

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 general rule: calChecker follows calOBbuilder not planned: –short-term night calibrations (like standard stars within 2 hrs) –complex dependencies (multiple standard stars a night; alternative calibrations; calibrations for calibrations) to be done on Paranal: –optimize calOBbuilder queue (input provided by QC)

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 goes beyond completeness, includes quality AB product monitorlinks to the QC processing page for that date (AB product monitor) 3. Links to product quality calCheckerHC monitor products/ AB monitor

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 AB product monitor

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Scores automatic evaluation scheme: 2 thresholds defined QC1 parameter evaluated total AB score: 0 / 1 / more than 1 counts outliers

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 links to other reports –data report –nightlog (password protected, sciops )

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, final QC Garching data now display as quick as/quicker than opslog! (forget 10 day gap…) 4. Quick-look HC monitor calCheckerHC monitor products/ AB monitor

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 full version score version (Quick-look)

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Information on demand: click on “more” if you want graphs, on “less” if you want scores only

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 HC monitor: selected scores selection: “relevant for instrument health” AB product monitor: all scores of a date

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 score propagation: –data scores for last week only –plot: last data scores summed up plot score green if last data score is green (previous red scores ignored) report score: plot scores are aggregated in report score report scores in group scores, and finally in instrument score all higher-level scores turn red if at least one lower-level score is red

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 big advantage of score hierarchies: within a second –instrument score green  you know within a second: all is OK! –red: the score overview can tell you more scheme intended to guide you to issues

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Health Check monitor –update pattern: refresh most important ones upon new opslog data –less important ones at least once per 24 hour process health –monitored: opslog transfer, parser (1x per hour) –coming soon: data transfer, ngas access

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 planned: –defocus the OPSLOG data (to become a fallback solution except for VIRCAM/OCAM) –improve update pattern for HC reports (immediate after new data) –product quality links (same as for calChecker) –agree on mail alert mechanism

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 calCheckerHC monitor products/ AB monitor

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 the end …

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Other QC services within this context Agora: FAQ page for PSO-QC related issues refs: reference frames web page QC: tutorial pages, documentation

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 avoid being under-sensitive: –overlook that raw file is saturated or empty –miss important properties = QC1 parameters avoid being too sensitive: –have too narrow thresholds –have unstable thresholds (3SIG) –use not fully understood parameters –select data sets (setups) rarely acquired (  trending impossible or uncertain) –use same QC1 parameter in various HC plots (  multiple alarms) rule of thumb: –every red score should have an explanation –likely trigger an action [Excursion: The art of scoring … ] [E. Munch, Cry]

Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, Operational impact daytime astronomer: check completeness –take action for ylw/red boxes: e.g. trigger calibrations check Health Check monitor –check scores –pick red ones –analyze problem –trigger actions check product quality (AB monitor)