Data Processing & Quality Control Group [www.eso.org/qc] Faculty Meeting Sept. 2010 Data Processing & Quality Control Quality Control Group head: Reinhard.

Slides:



Advertisements
Similar presentations
Defining Decision Support System
Advertisements

NGAS – The Next Generation Archive System Jens Knudstrup NGAS The Next Generation Archive System.
July 2006Instrument Scientist Retreat, Santiago Quality Control: closing the loop Reinhard Hanuschik, DFO/QC 1. Principles 2. Processes 3. Critical components.
Open Range Software Welcome to a presentation of the Open Range Industrial Hygiene Program.
EHarmony in Cloud Subtitle Brian Ko. eHarmony Online subscription-based matchmaking service Available in United States, Canada, Australia and United Kingdom.
Preparing MIDI for science operations on Paranal Markus Schöller Jean-Gabriel Cuby, Sebastien Morel, Andrea Richichi Schloss Ringberg, September 03, 2003.
A viable Solution to track and report Supplier Performance for Visual Enterprise: Manufacturing and Quality Copyright © 2007 – 2013 VisualHelp.net. All.
VISTA/WFCAM pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,
Lime is… …an interactive visual analysis tool connected to Dynamics NAV Graphical representation of NAV data in your browser Real-time visualization of.
DCS Architecture Bob Krzaczek. Key Design Requirement Distilled from the DCS Mission statement and the results of the Conceptual Design Review (June 1999):
Tools and Services for the Long Term Preservation and Access of Digital Archives Joseph JaJa, Mike Smorul, and Sangchul Song Institute for Advanced Computer.
Leicester Database & Archive Service J. D. Law-Green, J. P. Osborne, R. S. Warwick X-Ray & Observational Astronomy Group, University of Leicester What.
Leicester Database & Archive Service J. D. Law-Green, S. W. Poulton, J. Osborne, R. S. Warwick Dept. of Physics & Astronomy, University of Leicester LEDAS.
18 April 2007 Second Generation VLT Instruments 1 VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit.
The iPlant Collaborative Community Cyberinfrastructure for Life Science Tools and Services Workshop Discovery Environment Overview.
STEREO Science Center SWG – Dec 2004 William Thompson NASA Goddard SFC Code Greenbelt, MD
To hide, or not to hide, that is the question Towards common philosophy about rejecting, hiding badqual of the calibration data Reason for rejecting: to.
CXC Implementing 2007 NRC Portals of the Universe Report Chandra X-ray Center Recommended Best Practices Roger Brissenden and Belinda Wilkes 25 April 2012.
GAUDI Ground-based Asteroseismology Uniform Database Interface E. Solano Bases de données en spectroscopie stellaire. Paris.
Hunt for Molecules, Paris, 2005-Sep-20 Software Development for ALMA Robert LUCAS IRAM Grenoble France.
Operations Specialists Paranal [calChecker] [HC monitor]calCheckerHC monitor September 2013 The big picture I Why do we take calibrations? SCIENCE observations.
DCS Overview MCS/DCS Technical Interchange Meeting August, 2000.
MASSACHUSETTS INSTITUTE OF TECHNOLOGY NASA GODDARD SPACE FLIGHT CENTER ORBITAL SCIENCES CORPORATION NASA AMES RESEARCH CENTER SPACE TELESCOPE SCIENCE INSTITUTE.
1 The EIS Experience: Lessons Learned May 12, 2005.
Data Management Subsystem Jeff Valenti (STScI). DMS Context PRDS - Project Reference Database PPS - Proposal and Planning OSS - Operations Scripts FOS.
2-1 DMS Requirements Status Daryl Swade DMS Systems Engineer Nov. 6, 2013 S&OC Data Management System Design Review 3.
Access Across Time: How the NAA Preserves Digital Records Andrew Wilson Assistant Director, Preservation.
Raw data recipes SGDP Science Grade Data Products Wolfram Freudling DPD / SDP Group.
NHSC Workshop August 2013 NASA Herschel Science Center - page 1 PACS David R. Ardila User Support coordinator / NHSC Archive Scientist The Herschel Science.
Doug Tody E2E Perspective EVLA Advisory Committee Meeting December 14-15, 2004 EVLA Software E2E Perspective.
Planetary Science Archive PSA User Group Meeting #1 PSA UG #1  July 2 - 3, 2013  ESAC PSA Archiving Standards.
L. NExScI Operational Readiness Rafael Millan-Gabet Project Scientist, NExScI & Mihseh Kong, NExScI Software Engineer David Imel, IPAC & NExScI Manager.
Solutions for Quality Control of multi-detector instruments SPIE Marseille, 25 June 2008 Solutions for Quality Control of multi-detector instruments and.
SDD/DFS R. Hook. VLT 2 nd Generation Instrumentation Pipelines, 18 Apr Richard Hook Sampo & ESO Reflex background, status and plans.
SDD/DFS Y. Jung VLT 2 nd Generation Instrumentation Pipelines, 19 Apr General Introduction Yves Jung.
The iPlant Collaborative Community Cyberinfrastructure for Life Science Tools and Services Workshop Discovery Environment Overview.
Data Demo: Atmospheric Visualization Collection Christopher Klaus 05/18/05.
SCIOPS 2013 Reinhard Hanuschik, ESO Garching The VLT Quality Control Loop.
Improved and automated QC process [calChecker] [HC monitor] [histo]calCheckerHC monitorhisto April 27, 2009 Improved and automated QC process Reinhard.
N. Summary Phil Hinz Principal Investigator. Performance N-2.
The iPlant Collaborative Community Cyberinfrastructure for Life Science Tools and Services Workshop Discovery Environment Overview.
2007Sep06 EAC Butler - Software Overview 1 Software Overview Bryan Butler.
What the Data Products Program Offers Users Todd Boroson Dick Shaw Presentation to NOAO Users Committee October 23, 2003.
14 June, 2004 EVLA Overall Design Subsystems II Tom Morgan 1 EVLA Overall Software Design Final Internal Review Subsystems II by Tom Morgan.
06-1L ASTRO-E2 ASTRO-E2 User Group - 14 February, 2005 Astro-E2 Archive Lorella Angelini/HEASARC.
NEON School NEON Archive Observing School Alberto Micol ESA Space Telescope Operations Division 15 July 04 ESO & HST Archives.
Lesson 3.  Use of computers in recording, classifying, manipulating, and summarizing data.  Also called Automatic data processing, data processing,
C2d Data flow diagram BCD from SSC Texas SAO Quality Analysis and Improved Calibrated Data Mapping team.
COS PIPELINE CDR Jim Rose July 23, 2001OPUS Science Data Processing Space Telescope Science Institute 1 of 12 Science Data Processing
ESO Reflex A Graphical Workflow Engine for Data Reduction Richard Hook Euro VO Data Centres Alliance Theory & Grid Workshop, Garching, April 2008.
U.S. Department of the Interior U.S. Geological Survey Decision Support Tools and USGS Data Management Best Practices Cassandra Ladino USGS Chesapeake.
N. RadziwillEVLA Advisory Committee Meeting May 8-9, 2006 NRAO End to End (e2e) Operations Division Nicole M. Radziwill.
Operations Specialists Paranal [calChecker] [HC monitor]calCheckerHC monitor September 2013 Quality Control Loop and Daytime Checks Reinhard Hanuschik,
Mountaintop Software for the Dark Energy Camera Jon Thaler 1, T. Abbott 2, I. Karliner 1, T. Qian 1, K. Honscheid 3, W. Merritt 4, L. Buckley-Geer 4 1.
November 1, 2004 ElizabethGallas -- D0 Luminosity Db 1 D0 Luminosity Database: Checklist for Production Elizabeth Gallas Fermilab Computing Division /
Faculty meeting - 13 Dec 2006 The Hubble Legacy Archive Harald Kuntschner & ST-ECF staff 13 December 2006.
June 27-29, DC2 Software Workshop - 1 Tom Stephens GSSC Database Programmer GSSC Data Servers for DC2.
CyVerse Workshop Discovery Environment Overview. Welcome to the Discovery Environment A Simple Interface to Hundreds of Bioinformatics Apps, Powerful.
Fundamentals of Analytical Chemistry Eighth Edition.
BSB Biomanufacturing CHAPTER 4 GMP – Documentation Part I (SOP)
European Southern Observatory
NRAO VLA Archive Survey
CyVerse Discovery Environment
Joseph JaJa, Mike Smorul, and Sangchul Song
Real IBM C exam questions and answers
CMS Pixel Data Quality Monitoring
CMS Pixel Data Quality Monitoring
Operations Hub data flow tools and concepts
The User Support Department Fernando Comerón DMD/USD
Presentation transcript:

Data Processing & Quality Control Group [ Faculty Meeting Sept Data Processing & Quality Control Quality Control Group head: Reinhard Hanuschik 7+1 QC scientists (astronomers) sitting in TOB (with DMO) Reinhard Hanuschik

Data Processing & Quality Control Group [ Faculty Meeting Sept pipeline-process all VLT+VLTI data certify calibration products, store in archive process science data, store in archive create data packages for PIs QC mission: ensure that science data can be calibrated, to a known and documented level

Data Processing & Quality Control Group [ Faculty Meeting Sept supported: –all Paranal instruments: 11 VLT, 2 VLTI instruments, plus survey telescopes (VISTA, OCAM) close interaction with: –Paranal Science Operations –User Support –other groups at DPD (External Data Products, Science Data Products) –Archive Department

Data Processing & Quality Control Group [ Faculty Meeting Sept statistics per month: – processing jobs ( VLT, VIRCAM) (+150% in 2 years) –6 TB raw data processed –0.8 TB product data created and archived –120 SM data packages –300 DVDs processing: –on 13 dual-core blades, plus –on a cluster of 20 dual-core blades (VIRCAM, HAWKI, OCAM)

Data Processing & Quality Control Group [ Faculty Meeting Sept how do data get here? –data transfer Paranal – Antofagasta – Santiago – Garching –microwave link: now at 10 Mbits/sec –sufficient for all VLT(I) data plus VIRCAM calibrations –most raw files available in archive within minutes –next step: fibre link Paranal – Antofagasta (EVALSO, 2010/2011)

Data Processing & Quality Control Group [ Faculty Meeting Sept how is QC making use of it? process calibration data: incremental and automatic processing –once per hour: look for new CALIB data –pipeline-process them in the QC way (dynamic associations within pool of data, fine-tuned processing parameters etc.) –create QC reports –calculate scores –put this information on the web –do this 24 hrs a day, 7 days a week  quasi-real time quality control and feedback –evaluated by: –QC scientist –Paranal daytime or nighttime astronomer –QC does review/certification (off-line)

Data Processing & Quality Control Group [ Faculty Meeting Sept an example … life product monitor examples: today (all fine)(quality problem)?

Data Processing & Quality Control Group [ Faculty Meeting Sept life product monitor examples: today (all fine)(quality problem)?

Data Processing & Quality Control Group [ Faculty Meeting Sept rms_fit – rms of fitted dispersion solution

Data Processing & Quality Control Group [ Faculty Meeting Sept for comparison: healthy situation HC monitor link:

Data Processing & Quality Control Group [ Faculty Meeting Sept nlin_acc - number of lines for the dispersion solution

Data Processing & Quality Control Group [ Faculty Meeting Sept mean_R – mean resolving power

Data Processing & Quality Control Group [ Faculty Meeting Sept certification / rejection: interaction & analysis by QC scientist product monitor examples: today (all fine)(quality problem)?

Data Processing & Quality Control Group [ Faculty Meeting Sept QC review and certification –this is done off-line (Mon-Fri, 5/2) –score-based –review all red scores, plus selected green scores –analyze issues –reject or certify –communicate with PSO certified master calibrations go into archive

Data Processing & Quality Control Group [ Faculty Meeting Sept one QC data point = one calibration product trending of all core instrument components for complete history of instrument

Data Processing & Quality Control Group [ Faculty Meeting Sept quick-look: score version –concentrates on compliance with predefined thresholds –numerical values replaced by scores: simple and graphical –hierarchical score scheme  instrument score

Data Processing & Quality Control Group [ Faculty Meeting Sept Health Check monitor: –central tool for quality feedback to Paranal SciOps –closes the feedback loop: raw – pipeline – scores - HC –up-to-date at timescale of 1 hour –based mostly on CALIB information (daytime, nighttime) –includes also SCIENCE information (image quality; VIRCAM: zero points)

Data Processing & Quality Control Group [ Faculty Meeting Sept The process for SCIENCE data –pipeline-processing: off-line –all pipeline supported instrument modes (40-50) –SM or VM: does not matter –use certified master calibrations –no strict certification process (on-line quality is graded ABCD by SciOps); feedback about quality problems or uncalibrated OBs given to USD and SciOps –products are ingested into archive –data packages are stored on the User Portal and available there for PIs –timescale for User Portal availability to PIs: –raw data within minutes after acquisition –product data within days (or faster)

Data Processing & Quality Control Group [ Faculty Meeting Sept the end

Data Processing & Quality Control Group [ Faculty Meeting Sept processing level of SCIENCE data –pipelines: originally mainly for quality control and quick look –goal: remove instrumental and atmospheric signatures strong push from community towards science-grade quality –sheer volume of raw data –complexity of data reduction push pipelines and calibration plans towards science-grade products (SDP) to be implemented within the QC scheme: mass production with pipelines, not fine-tuned towards specific scientific goal

Data Processing & Quality Control Group [ Faculty Meeting Sept the end

Data Processing & Quality Control Group [ Faculty Meeting Sept calChecker: automatic interface to check completeness of calibrations –again: feedback scale less than one hour –24/7 –checks completeness of all calibrations formally required for science, within validity range calCheckercalChecker live