To hide, or not to hide, that is the question Towards common philosophy about rejecting, hiding badqual of the calibration data Reason for rejecting: to.

Slides:



Advertisements
Similar presentations
26 January 2007 CAL 07 Garching 1 The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit.
Advertisements

July 2006Instrument Scientist Retreat, Santiago Quality Control: closing the loop Reinhard Hanuschik, DFO/QC 1. Principles 2. Processes 3. Critical components.
SDD/DFS P.Ballester VLT 2 nd Generation Instrumentation Pipelines 18 Apr Pipeline Integration into the Data Flow System.
NOAO/Gemini Data workshop – Tucson,  Hosted by CADC in Victoria, Canada.  Released September 2004  Gemini North data from May 2000  Gemini.
QARTOD II Currents and Waves In-Situ Currents: Breakout Group Report Out QARTOD II February 28 – March 2, 2005.
IRAC-Data Processing Pipeline Review / January 26, 2001 Jason Surace (1) IRAC Pipeline Data Analysis and Pipeline Validation Plan Jason Surace January.
Backup and Recovery Part 1.
18 April 2007 Second Generation VLT Instruments 1 VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit.
PAT Validation Working Group Process and Analytical Validation Working Group Arthur H. Kibbe, Ph.D. Chair June 13, 2002.
Year 10 Work Experience. Information  Work Experience for ALL Year 10s starts on Monday 27 th June  Everyone does one week so will be back in school.
Testing Hypotheses I Lesson 9. Descriptive vs. Inferential Statistics n Descriptive l quantitative descriptions of characteristics n Inferential Statistics.
Means Tests Hypothesis Testing Assumptions Testing (Normality)
Hunt for Molecules, Paris, 2005-Sep-20 Software Development for ALMA Robert LUCAS IRAM Grenoble France.
All rights reserved. FUTEK 2006 SENSOR SOLUTION TIPS Presented by: Javad Mokhbery (President)
Operations Specialists Paranal [calChecker] [HC monitor]calCheckerHC monitor September 2013 The big picture I Why do we take calibrations? SCIENCE observations.
NGC 6217 in DSI mode with F658N Mosaic before new superbias why we don’t release images until after SMOV! ACS CCD monitoring and pipeline calibration review.
Detector Monitoring as part of VLT Science and Data Flow Operations Wolfgang Hummel DMO/DFO/QC group Lander de Bilbao, SDD/pipeline group Andrea Modigliani,
Weak positive correlation Weak negative correlation Strong positive correlation Strong negative correlation Q. Describe the correlation.
1 Information Filtering & Recommender Systems (Lecture for CS410 Text Info Systems) ChengXiang Zhai Department of Computer Science University of Illinois,
MASSACHUSETTS INSTITUTE OF TECHNOLOGY NASA GODDARD SPACE FLIGHT CENTER ORBITAL SCIENCES CORPORATION NASA AMES RESEARCH CENTER SPACE TELESCOPE SCIENCE INSTITUTE.
1 Institute of Engineering Mechanics Leopold-Franzens University Innsbruck, Austria, EU H.J. Pradlwarter and G.I. Schuëller Confidence.
[ §3 : 1 ] 2. Life-Cycle Perspective Overview 2.1 Motivation 2.2 Waterfall Model 2.3 Requirements in Context.
EMIS-R Data Collector Uncovered Teresa Williams NWOCA/SSDT OAEP
Doc.: IEEE /1523r0 Submission Offline Discussion Minutes of SLS Calibration Date: Authors: Slide 1 Nov 2014 Jiyong Pang (Huawei Technologies)
TOSCA Monitoring Working Group Status Roger Dev June 17, 2015.
Raw data recipes SGDP Science Grade Data Products Wolfram Freudling DPD / SDP Group.
TIE STUDENT APPLICATION [YOUR APPLICATION NAME HERE] Group number Student name 1.
European Southern Observatory ESO Proposals, Prague  27 February 2009 Successful ESO proposals an overview Bruno Leibundgut (ESO)
Solutions for Quality Control of multi-detector instruments SPIE Marseille, 25 June 2008 Solutions for Quality Control of multi-detector instruments and.
VLA Archive Image Pilot Pilot to create images from VLA archive data Loránt Sjouwerman National Radio Astronomy Observatory.
SDD/DFS Y. Jung VLT 2 nd Generation Instrumentation Pipelines, 19 Apr General Introduction Yves Jung.
Copyright © 1998, Triola, Elementary Statistics Addison Wesley Longman 1 Testing a Claim about a Proportion Section 7-5 M A R I O F. T R I O L A Copyright.
too.
SCIOPS 2013 Reinhard Hanuschik, ESO Garching The VLT Quality Control Loop.
Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Improved and automated QC process Reinhard.
Bargaining as Constraint Satisfaction Simple Bargaining Game Edward Tsang
Schedule Agreements Training Presentation for Supply Chain Platform: BAE Systems July 2012.
Herschel Open Time Cycle 1 DP workshop ESAC, March page 1 Spurs in HIFI data Colin Borys.
Slide 1 Archive Computing: Scalable Computing Environments on Very Large Archives Andreas J. Wicenec 13-June-2002.
Lesson 2: association Lesson 2: Association Lesson 1: dfos - vision and architecture Directory structure Installation Tips and tricks Lesson 3: Operations.
14 January Observational Astronomy SPECTROSCOPIC data reduction Piskunov & Valenti 2002, A&A 385, 1095.
Operations Specialists Paranal [calChecker] [HC monitor]calCheckerHC monitor September 2013 Quality Control Loop and Daytime Checks Reinhard Hanuschik,
Data Processing & Quality Control Group [ Faculty Meeting Sept Data Processing & Quality Control Quality Control Group head: Reinhard.
1 Automatic Music Style Recognition Arturo Camacho.
T Iteration I1 Demo Software Trickery PP Iteration
RAW DATA BIAS & DARK SUBTRACTION PIXEL-TO-PIXEL DQE CORR. LOCATE EXTR. WINDOW THROUGHPUT CORRECTION (incl. L-flat, blaze function, transmission of optics,
Doc.: IEEE /1523r4 Submission Offline Discussion Minutes of SLS Calibration Date: Authors: Slide 1 Jan 2015 Jiyong Pang (Huawei Technologies)
ESO – A.Longinotti ESO Instrumentation Software Workshop Antonio Longinotti Testing Instrumentation Software Applications.
NHSC HIFI DP workshop Caltech, 7-9 February page 1 Spurs in HIFI data.
Rekayasa Perangkat Lunak Kuliah 2. Outline of this presentation Attributes of Good Software Why Software Engineering ? What is Software Product ? Software.
Double Star Active Archive - STAFF-DWP Data errors and reprocessing Keith Yearby and Hugo Alleyne University of Sheffield Nicole Cornilleau-Wehrlin LPP.
Tips and Tricks: Stress Free Security in Dynamics AX Chris Haley, Microsoft.
1 Press Ctrl-A ©G Dear 2008 – Not to be sold/Free to use Organising Data Stage 6 - Year 11 General Mathematics Preliminary.
DAA Status/Progress B. Mihaljčić, A. Fazakerley, N. Doss, G. Watson UCL Department of Space and Climate Physics Mullard Space Science Laboratory 18 th.
A.Zanichelli, B.Garilli, M.Scodeggio, D.Rizzo
ABG Instrumentation Analytical Errors
European Southern Observatory
ROCC Operations and The PSA RSP Archive Concept Review 28 September 2017 Tanya Lim Archive Scientist.
PDS4 Data From The Rover RSP Archive Concept Review 28 September 2017
Florian Lütticke, Carlos Marinas, Norbert Wermes
Data Processing Status
New static DQ masks for NICMOS
The VIMOS IFU Pipeline Carlo Izzo.
Risk Management with Minimum Weight
Saravana Kumar CEO/Founder - Kovai Atomic Scope – Product Update.
Section 12.5 Tree Diagrams.
Rekayasa Perangkat Lunak
Progressive Response to Challenging Behavior
Operations Hub data flow tools and concepts
Progressive Response to Challenging Behavior
Presentation transcript:

To hide, or not to hide, that is the question Towards common philosophy about rejecting, hiding badqual of the calibration data Reason for rejecting: to assure that only certified/good calibrations are used to be associated with science data (science ABs, the static solution of the calSelector,) and only certified/good master calibrations are ingested in the Archive; Reason for hiding : to clean the ESO Archive from bad/unusable calibration raw frames that could otherwise be picked up by future users (the dynamic solution of the calSelector); Reason for badqual : to assure that, on the calChecker level, only certified/good calibrations are considered for science association; main motivation is to alert Paranal if good calibrations are missing; Good data – definition: stable in time, closely correspond to the status of the instrument; should be “scored green”, but can also be “scored red” if the reasons are understood and accepted;

Data REJECT VCAL rm HIDE BADQUAL saturated, severely underexposed, contaminated by artifacts, unusable/bad for science YES pipeline failed because data are faulty/bad YES “scored red” but reflect out of ordinary, acceptable status of the instrument; correspond to the science data NO “scored red” because reflect inacceptable, out of ordinary status of the instrument; not needed for science YES NO “scored red” but show only slight/acceptable departure from thresholds; useful for science NO “scored red” but can still be useful for science; calibrations cannot be replaced NO but open ticket to USD NO “scored red” but still useful; new, good set was re-taken YES NO “scored red”, but may still be useful (e.g. IR twilight flats sampling too narrow dynamic range) YES NO YES pipeline failed but data fine, usable for science (e.g. recipe cannot handle instr. setup, incomplete template) YES NO YES pipeline failed, data fine but not usable for science (e.g. incomplete template) YES

FOOTNOTES: 1)Hiding of the raw frames in the Archive should be exception and not a habit. The assumption is that Paranal delivers mainly good data.