10Tb 100Tb 1000Tb. WFCAM processing summary semesters: 05A, 05B, 06A, 06B, 07A nights: 81, 166, 95, 157, 50 (68%) 50 Tbytes raw data 17 Tbytes compressed.

Slides:

Advertisements

Similar presentations

Astronomical works with students Irina GUSEVA St Petersburg - Central (Pulkovo) Astronomical Observatory of the Russian Academy of Sciences - St Petersburg.

Advertisements

26 January 2007 CAL 07 Garching 1 The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit.

July 2006Instrument Scientist Retreat, Santiago Quality Control: closing the loop Reinhard Hanuschik, DFO/QC 1. Principles 2. Processes 3. Critical components.

GMOS Data Reduction Richard McDermid Gemini Data Reduction Workshop Tucson, July 2010.

ADASS XVII Sep 2007The NOAO Pipeline Applications Francisco Valdes (NOAO) Robert Swaters (UMd) Derec Scott (NOAO) Mark Dickinson (NOAO)

Towards Creation of a JWST Astrometric Reference Field: Calibration of HST/ACS Absolute Scale and Rotation Roeland van der Marel Jay Anderson, Colin Cox,

Echelle spectra reduction with IRAF*

Palomar Transient Factory Data Flow Jason Surace IPAC/Caltech.

VISTA/WFCAM pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

15 December 2008Science from UKIDSS II WFCAM Science Pipeline Update WFCAM Science Pipeline Update Jim Lewis, Mike Irwin & Marco Riello Cambridge Astronomy.

VISTA pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

Pan-STARRS Seminar: IPPEugene Magnier Pan-STARRS Image Processing Pipeline Astrometry and Photometry IFA Pan-STARRS Seminar 735 October 14, 2004.

WFAU Tapes === Store 6 x dual PCs 3GHz Xeons 6 x 1 Tbyte local disks 2 Tbyte ingest buffer 10 Tbytes of raw 20 Tbytes of processed WFCAM archive storage.

Stellar Linearity Test Jason Surace (Spitzer Science Center)

Photometric Catalog I-band selected photometric catalog containing ~800,000 galaxies (I (AB) < 26 mag) in 8 bands (UBVgRizK) Star-Galaxy separation using.

18 April 2007 Second Generation VLT Instruments 1 VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit.

Bruno Altieri September # 1 PACS photometer pipeline overview Bruno Altieri (ESA/HSC)

0. To first order, the instrument is working very well ! 1.Evolution of the IR detector with time 2.Stability of the L channel 3.Saturation 4.Linearity.

Memorandam of the discussion on FMOS observations and data kicked off by Ian Lewis Masayuki Akiyama 14 January 2004 FMOS Science Workshop.

E. BertinAstro-WISE workshop 11/ TERAPIX: SCAMP E.Bertin (IAP & Obs. de Paris/LERMA)

NAOKI YASUDA, MAMORU DOI (UTOKYO), AND TOMOKI MOROKUMA (NAOJ) SN Survey with HSC.

Distortion in the WFC Jay Anderson Rice University

KIDS requirements Dave Wilman for the KIDS team The KIDS operational plan and the role of AstroWise. List focussed on open questions: discussion required!

Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 1 The Data-reduction Pipeline for the INT Wide Field Camera.

Page 1. Page 2 SMOV Status SMOV is proceeding on-track Key programs executed –ERO program –Geometric distortion program completed –ACS to FGS alignment.

December 11, 2006 DES DM - Mohr The DES DM Team Tanweer Alam 1 Dora Cai 1 Joe Mohr 1,2 Jim Annis 3 Greg Daues 1 Choong Ngeow 2 Wayne Barkhouse 2 Patrick.

WFCAM Science Archive Critical Design Review, April 2003 The SuperCOSMOS Science Archive (SSA) WFCAM Science Archive prototype Existing ad hoc flat file.

GSCII Project Gauging the quality of the GSCII DB with QA_2plates: First results R. Pannunzio, R. Morbidelli, A Spagna, M. G.Lattanzi.

Palomar Transient Factory Data Flow Jason Surace IPAC/Caltech.

AST 443/PHY 517 : Observational Techniques November 6, 2007 ASTROMETRY By: Jackie Faherty.

DC2 Post-Mortem/DC3 Scoping February 5 - 6, 2008 DC3 Goals and Objectives Jeff Kantor DM System Manager Tim Axelrod DM System Scientist.

ACS Drizzling Overview J. Mack; DA Training 10/5/07 Distortion Dither Strategies MultiDrizzle ‘Fine-tuning’ Data Quality Photometry.

Pachon Sky Camera Christopher Stubbs Chuck Claver Jan 25, 2014.

SCIOPS 2013 Reinhard Hanuschik, ESO Garching The VLT Quality Control Loop.

Data products of GuoShouJing telescope(LAMOST) pipeline and current problems LUO LAMOST Workshop.

AST3-1 photometry from Dome A Bin Ma, Peng Wei, Yi Hu, Zhaohui Shang NAOC AST3

1 Groningen, November 2003 ASTROWISE OAC TEAM The ASTRO-WISE project: status at OAC The OAC AW team: J.M. Alcalà, F. Getman, A. Grado, M. Pavlov,

Measuring Sky Brightness with a Digital Camera Paris 2004.

Data Analysis Software Development Hisanori Furusawa ADC, NAOJ For HSC analysis software team 1.

1 CORE DATA PROCESSING SOFTWARE PLAN REVIEW | SEATTLE, WA | SEPTEMBER 19-20, 2013 Name of Meeting Location Date - Change in Slide Master Data Release Processing.

SNAP Calibration Program Steps to Spectrophotometric Calibration The SNAP (Supernova / Acceleration Probe) mission’s primary science.

VST ATLAS: Requirements, Operations and Products Tom Shanks, Nigel Metcalfe, Jamie McMillan (Durham Univ.) +CASU.

1 ACS Press Releases Science  Grism spectroscopy of UDF galaxies  Mass of the MACHO-LMC-5 lens  Earliest star-forming galaxies in UDF  Kepler’s SNR.

LSST and VOEvent VOEvent Workshop Pasadena, CA April 13-14, 2005 Tim Axelrod University of Arizona.

Pan-STARRS Seminar: IPPEugene Magnier Pan-STARRS Image Processing Pipeline An Overview IFA Pan-STARRS Seminar 735 October 6, 2004.

Difference Image Analysis at OAC Groningen, 1st Dec 2004 AW-OAC team.

14 January Observational Astronomy SPECTROSCOPIC data reduction Piskunov & Valenti 2002, A&A 385, 1095.

IPHAS Early Data Release E. A. Gonzalez-Solares IPHAS Consortium AstroGrid National Astronomy Meeting, 2007.

July 21, 2004 Natalia Kuznetsova NICMOS Analysis of ACS ● Data Inventory ● Calibration ● Bad pixel mask ● Extracting photometry ● Next steps 1.

The OmegaCAM photometric systems. OmegaCAM calibration plan (1) Basic OmegaCAM terminology : ● Key bands vs User bands ● Key band -> User band transformation.

Spotting the life of stars „Pi of the Sky” Project Katarzyna Kwiecińska UKSW-SNŚ on behalf of the Pi of the Sky collaboration.

Bad pixels treatment Previous and existing tools Examples and problems.

26th October 2005 HST Calibration Workshop 1 The New GSC-II and it’s Use for HST Brian McLean Archive Sciences Branch.

Photometry and Astrometry: Bright Point Sources May 16, 2006 Cullen Blake.

GSPC -II Program GOAL: extend GSPC-I photometry to B = V ˜ 20 add R band to calibrate red second-epoch surveys HOW: take B,V,R CCD exposures centered at.

Data Pipeline Wei Zheng Johns Hopkins University.

WFC3 SMOV UVIS and IR Geometric Distortion Calibration and Multidrizzle Vera Kozhurina-Platais and WFC3 team.

FILE COMPRESSION Lossy vs Lossless. Why compress a file? To save storage space. To speed up data transmission.

A. Ealet Berkeley, december Spectrograph calibration Determination of specifications Calibration strategy Note in

The First Release of AST3 point source catalog from Dome A, Antarctica in 2012 MA Bin, WEI Peng, SHANG Zhaohui, LIU Qiang, HU Yi (NAOC) 973 课题二

SINFONI data reduction using the ESO pipeline. Instrument design and its impact on the data (I) integral field spectrometer using mirrors brickwall pattern.

Meteor detection software for low-cost single-board computers Denis Vida Dario Zubović, Peter Gural, Damir Šegon, Robert Cupec.

CCD Calibrations Eliminating noise and other sources of error.

Searching for transiting Extra-Solar planets “Pre-OmegaTranS”

PESSTO Data Reduction PESSTO meeting 6/1/2018.

QC workshop Goals: implement QC tools and have a demo of what we have reached by end of the week (i.e. Thursday) Learn each other new developpements.

MODIS Characterization and Support Team Presented By Truman Wilson

Imaging data obtained with the

Photometric Analysis of Asteroids

Karen Meech Institute for Astronomy TOPS 2003

Presentation transcript:

10Tb 100Tb 1000Tb

WFCAM processing summary semesters: 05A, 05B, 06A, 06B, 07A nights: 81, 166, 95, 157, 50 (68%) 50 Tbytes raw data 17 Tbytes compressed 75 Tbytes processed data 28 Tbytes compressed 3 million 2kx2k images storage: 6 x 4Tbyte RAID5 units 2 x 7Tbyte RAID5 units 4 x 10Tbyte RAID6 units = capacity until end of 07B (grant rollover)

06B features and processing status missing files null files NDR ADU/s channel edge pedestals bar-code corrupted science frames + calibration darks, flats skys impact:

Quality Control parameters sky, images, astrometric, photometric

Nightly photometric zeropoints 06B

WFCAM photometric calibration H J Y Z K WFCAM K-band Zpt semester 05B

Recent processing updates/news more robust correction for cross-talk 06B -> photometric solutions inc. extinction corr (all data) [2MASS s:n>10 in J,H,Ks and 0 ce + ext corr] improved x-y centroiding for all detected objects 06B -> band-specific astrometric WCS solution (all data) MAGZPT, MAGZRR, NUMZPT per pointing (all data) NIGHTZPT, NIGHTZRR, NIGHTNUM /band/night (all data) direct products to PIs, reprocessing PI data, raw data to PIs internet transfers

Astrometric Calibration 2MASS Linear solution per detector Tabulated systematics from stacked residuals WCS - ZPN projection rms < 100 mas

ZYJHKZYJHK ZP galactic extinction - corrected previously E(B-V) ZYJHKZYJHK Photometric calibration 2MASS

Y-band zeropoint calibration (Z)

Photometric residuals J-band 05B 06A (illumination correction ++)

J-band 05B Aug -v- Dec flat

07A 1st April bright moon

fin

WFCAM processing steps prepare - ingest MEFs, check, index, select, preview, process instrumental signature removal - darks, flats, curtains, cross-talk sky artefact correction - group master skys by time/ OB combine - compute shifts, interleave, stack dithers catalog - detect and parameterise objects [aperture photometry] classify - morphological classification [curve-of-growth] astrometry - astrometric calibration per detector photometry - photometric calibration per pointing [detector gain] check - examine QC, reject bad products, visual inspection

Example WFCAM illumination correction 2” diam aperture 4” diam aperture