VISTA/WFCAM pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

Slides:

Advertisements

Similar presentations

26 January 2007 CAL 07 Garching 1 The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit.

Advertisements

ADASS XVII Sep 2007The NOAO Pipeline Applications Francisco Valdes (NOAO) Robert Swaters (UMd) Derec Scott (NOAO) Mark Dickinson (NOAO)

MIPS Data Processing for SINGS George J. Bendo & Chad Engelbracht (A ''George's Adventures in Learning OpenOffice'' Presentation)

1 Databases in ALICE L.Betev LCG Database Deployment and Persistency Workshop Geneva, October 17, 2005.

Nicholas Cross, Rob Blake, Ross Collins, Mark Holliman, Mike Read, Eckhard Sutorius, Nigel Hambly, Andy Lawrence, Bob Mann, Keith Noddle Wide Field Astronomy.

10Tb 100Tb 1000Tb. WFCAM processing summary semesters: 05A, 05B, 06A, 06B, 07A nights: 81, 166, 95, 157, 50 (68%) 50 Tbytes raw data 17 Tbytes compressed.

15 December 2008Science from UKIDSS II WFCAM Science Pipeline Update WFCAM Science Pipeline Update Jim Lewis, Mike Irwin & Marco Riello Cambridge Astronomy.

Brian Schmidt, Paul Francis, Mike Bessell, Stefan Keller.

Cosmos Data Products Version Peter Capak (Caltech) May, 23, 2005 Kyoto Cosmos Meeting.

VISTA pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

WFAU Tapes === Store 6 x dual PCs 3GHz Xeons 6 x 1 Tbyte local disks 2 Tbyte ingest buffer 10 Tbytes of raw 20 Tbytes of processed WFCAM archive storage.

Introduction to Spitzer and some applications Data products Pipelines Preliminary work K. Nilsson, J.M. Castro Cerón, J.P.U. Fynbo, D.J. Watson, J. Hjorth.

Brief Overview of Data Processing of Afghanistan Household Listing, Pilot Census Results, Population and Housing Census and NRVA Survey Brief Overview.

18 April 2007 Second Generation VLT Instruments 1 VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit.

Data provenance in astronomy Bob Mann Wide-Field Astronomy Unit University of Edinburgh

A Preliminary Cycle 1 Calibration Plan for the JWST Science Instruments James Muzerolle.

Commissioning the NOAO Data Management System Howard H. Lanning, Rob Seaman, Chris Smith (National Optical Astronomy Observatory, Data Products Program)

Memorandam of the discussion on FMOS observations and data kicked off by Ian Lewis Masayuki Akiyama 14 January 2004 FMOS Science Workshop.

The Astro-Wise Pipeline Conceptual Design Review Roeland Rengelink Requirements Development Framework Prototype.

DCS Overview MCS/DCS Technical Interchange Meeting August, 2000.

MASSACHUSETTS INSTITUTE OF TECHNOLOGY NASA GODDARD SPACE FLIGHT CENTER ORBITAL SCIENCES CORPORATION NASA AMES RESEARCH CENTER SPACE TELESCOPE SCIENCE INSTITUTE.

1 The EIS Experience: Lessons Learned May 12, 2005.

Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 1 The Data-reduction Pipeline for the INT Wide Field Camera.

Data Management Subsystem Jeff Valenti (STScI). DMS Context PRDS - Project Reference Database PPS - Proposal and Planning OSS - Operations Scripts FOS.

Science Archive for Sky Surveys Data Providers and the VO - NeSC 2003 March Wide Field Astronomy Unit Institute for Astronomy.

Hristo Pavlov 4-th Trans Tasman Symposium on Occultations 5 April 2010, Canberra.

US Planck Data Analysis Review 1 Peter MeinholdUS Planck Data Analysis Review 9–10 May 2006 Where we need to be 2 months before launch- Instrument view.

WFCAM Science Archive Critical Design Review, April 2003 The SuperCOSMOS Science Archive (SSA) WFCAM Science Archive prototype Existing ad hoc flat file.

10 June 2002Towards an International VO - Garching bei Munchen 1 ASTRO-WISE An Astronomical Wide-field Imaging System for Europe Konrad Kuijken, Edwin.

Advanced Data Products at ESO P.Rosati (DMO/VOS) with special thanks to: B.Vandame, C.Rite, J.Retzlaff, R.Slijkhuis, M.Nonino Faculty Meeting - Dec 13,

Solutions for Quality Control of multi-detector instruments SPIE Marseille, 25 June 2008 Solutions for Quality Control of multi-detector instruments and.

AstroWise((G)A)VO Meeting 6 May 2004 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise.

Students: Anurag Anjaria, Charles Hansen, Jin Bai, Mai Kanchanabal Professors: Dr. Edward J. Delp, Dr. Yung-Hsiang Lu CAM 2 Continuous Analysis of Many.

DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008 DC3 Goals and Objectives Jeff Kantor DM System Manager Tim Axelrod DM System Scientist.

RVS Calibration Workshop, Paris RVS RVS Calibration RVS Calibration & First Look Workshop, Paris Mark Cropper.

Virtual Survey System sept 04 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise a Virtual.

Data Analysis Software Development Hisanori Furusawa ADC, NAOJ For HSC analysis software team 1.

VST ATLAS: Requirements, Operations and Products Tom Shanks, Nigel Metcalfe, Jamie McMillan (Durham Univ.) +CASU.

NEON School NEON Archive Observing School Alberto Micol ESA Space Telescope Operations Division 15 July 04 ESO & HST Archives.

MOS Data Reduction Michael Balogh University of Durham.

C2d Data flow diagram BCD from SSC Texas SAO Quality Analysis and Improved Calibrated Data Mapping team.

Pan-STARRS Seminar: IPPEugene Magnier Pan-STARRS Image Processing Pipeline An Overview IFA Pan-STARRS Seminar 735 October 6, 2004.

Distributed Pipeline Programming for Mosaics Or Mario Tips’N’Tricks.

Difference Image Analysis at OAC Groningen, 1st Dec 2004 AW-OAC team.

IPHAS Early Data Release E. A. Gonzalez-Solares IPHAS Consortium AstroGrid National Astronomy Meeting, 2007.

Data Processing & Quality Control Group [ Faculty Meeting Sept Data Processing & Quality Control Quality Control Group head: Reinhard.

STScI 2010 Calibration Workshop The NICMOS Legacy Archival Recalibration Project Anton M. Koekemoer and the STScI NICMOS Team (E. Barker, E. Bergeron,

July 21, 2004 Natalia Kuznetsova NICMOS Analysis of ACS ● Data Inventory ● Calibration ● Bad pixel mask ● Extracting photometry ● Next steps 1.

Automated reduction and calibration of SCUBA archive data using ORAC-DR Abstract The Sub-millimetre Common User Bolometer Array (SCUBA) instrument has.

EVLA Data Processing PDR Pipeline design Tim Cornwell, NRAO.

GSPC -II Program GOAL: extend GSPC-I photometry to B = V ˜ 20 add R band to calibrate red second-epoch surveys HOW: take B,V,R CCD exposures centered at.

1 Core Data Processing Software Plan Review – University of Washington, Seattle, WA – Sept th Data Management XXVIII IAU General Assembly.

The GLORIA demonstrator experiment Ariel Majcher National Centre for Nuclear Research Warsaw, Poland XXXII IEEE-SPIE Joint Symposium Wilga 2013, May 29.

Pi of the Sky off-line experiment with GLORIA Ariel Majcher National Centre for Nuclear Research Warsaw, Poland 10th INTEGRAL/BART Workshop, April.

Critical Design Review, April 2003

Calibration meeting summary

A Difference Imaging Tool

QC workshop Goals: implement QC tools and have a demo of what we have reached by end of the week (i.e. Thursday) Learn each other new developpements.

JWST Pipeline Overview

Data Processing Status

An Overview of the TERAPIX pipeline

SGS for OUSIM Meeting goals:

ESO: Applications to the ESO Imaging Survey and to the VLT

Imaging data obtained with the

ASTRO-WISE workshop Nov 03

WFCAM Photometric Calibration

WFCAM Photometric Calibration

Observing with Modern Observatories (the data flow)

First Public Data Releases from the VISTA Science Archive

Web Service Security support in the SSE Toolbox

Presentation transcript:

VISTA/WFCAM pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production, astrometric and photometric calibration further processing: generate well sampled PSF, PSF and Sersic profile fitting advanced pipeline: database driven, deep stacking and mosaicing, merging catalogues, list driven photometry, overlap calibration

WFAU tapes disks internet

Technical challenges non-linearity calibration and measurement cross-talk between channels and detectors image persistence from frame-to-frame sky level variations on short timescales rapid variability of OH fringing seeing variations on short timescales possible time varying flatfield characteristics calibration and extinction monitoring data volume and throughput robust automatic DQC measures

Progress work package spreadsheets and timelines instigated regular minuted meetings with actions provisional ICDs with JAC and WFAU agreed FITS header definition and test MEF data processed automated DQC monitoring developed pipeline monitoring tools with web interface programming completed for all basic modules ORAC-DR test routines for UFTI delivered and installed software module development for standard pipeline on target good progress with VDFS documentation for PDR

user manuals written for development pipeline/toolkits Bad pixel monitoring, mask generating, confidence maps difference imaging software developed simulations and completeness tests done prototype list-driven photometry package written and tested devised general purpose stacking/mosaicing framework NN stacking algorithm with simple rejection completed large area mosaicing algorithm developed automated astrometric and photometric calibration astrometric and photometric distortion for WFCAM and VISTA quantified

Processing tools / pipelines INT Wide Field Camera (4 x 2k x 4k) ESO WFI on 2.2m at La Silla (8 x 2k x4k) KPNO and CTIO Mosaic1,2 4m (8 x 2k x 4k) WFI on AAT (8 x 2k x 4k) CFHT 12k imager (12 x 2k x 4k) CIRSI mosaic on INT/DuPont (4 x 1k x 1k) INGRID on WHT (1k x 1k) UFTI on UKIRT (1k x 1k) ISAAC on VLT (1k x 1k) ****

CASU technical expertise real time analysis and discovery photometry and photometric system calibration astrometry and astrometric system calibration data modelling and simulations archiving and DB management Sybase, PostgresQL, SQL-Server data management and processing