ADASS XVII Sep 2007The NOAO Pipeline Applications Francisco Valdes (NOAO) Robert Swaters (UMd) Derec Scott (NOAO) Mark Dickinson (NOAO)

Slides:

Advertisements

Similar presentations

Institute for Gravitational Research

Advertisements

Institute for Gravitational Research

26 January 2007 CAL 07 Garching 1 The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit.

GMOS Data Reduction Richard McDermid Gemini Data Reduction Workshop Tucson, July 2010.

Small Refractor CCD Imaging David Haworth

1 FNAL-ANL PreCam Reductions Douglas L. Tucker (FNAL) DES Collaboration Meeting ICG, Portsmouth PreCam Parallel Session 29 June 2011.

MIPS Data Processing for SINGS George J. Bendo & Chad Engelbracht (A ''George's Adventures in Learning OpenOffice'' Presentation)

Palomar Transient Factory Data Flow Jason Surace IPAC/Caltech.

VISTA/WFCAM pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

1 Preliminary Design Review Tucson, Arizona August 29 th – September 2 nd, 2011 Data Management: Input Data Prepared by: Simon Krughoff ImSim Project:

NWO Meeting 24 May 04 a Virtual Survey System AstroWise- OmegaCAM – Lofar – Euro-VO a Virtual Survey System AstroWise- OmegaCAM – Lofar – Euro-VO Edwin.

Growth of Structure Measurement from a Large Cluster Survey using Chandra and XMM-Newton John R. Peterson (Purdue), J. Garrett Jernigan (SSL, Berkeley),

15 December 2008Science from UKIDSS II WFCAM Science Pipeline Update WFCAM Science Pipeline Update Jim Lewis, Mike Irwin & Marco Riello Cambridge Astronomy.

Brian Schmidt, Paul Francis, Mike Bessell, Stefan Keller.

Cosmos Data Products Version Peter Capak (Caltech) May, 23, 2005 Kyoto Cosmos Meeting.

VISTA pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

WFAU Tapes === Store 6 x dual PCs 3GHz Xeons 6 x 1 Tbyte local disks 2 Tbyte ingest buffer 10 Tbytes of raw 20 Tbytes of processed WFCAM archive storage.

Introduction to Spitzer and some applications Data products Pipelines Preliminary work K. Nilsson, J.M. Castro Cerón, J.P.U. Fynbo, D.J. Watson, J. Hjorth.

18 April 2007 Second Generation VLT Instruments 1 VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit.

The KPNO 4m “Mayall” Telescope Arjun Dey (NOAO). National Optical Astronomy Observatory Mission: provide the best ground-based astronomical capabilities.

Commissioning the NOAO Data Management System Howard H. Lanning, Rob Seaman, Chris Smith (National Optical Astronomy Observatory, Data Products Program)

S. Baggett, J. Anderson, K. S. Long, J. W. MacKenty, K. Noeske, J. Biretta, and the WFC3 team (STScI) WFC3 : Understanding and mitigating UVIS charge.

Why Build Image Mosaics for Wide Area Surveys? An All-Sky 2MASS Mosaic Constructed on the TeraGrid A. C. Laity, G. B. Berriman, J. C. Good (IPAC, Caltech);

HST Archival Studies of HH 30 WFPC2 Data Retrieval and Reduction Liu, Jimmy Chun-Fan Supervisor: Dr. Shang, Hsien August 23, 2002 Summer Student Program.

ZTF Server Architecture Roger Smith Caltech

PPA Stack User Driven Image Stacking for ODI data via a Highly Customizable Web Interface Soichi Hayashi Indiana University - Pervasive Technology Institute.

Detector Monitoring as part of VLT Science and Data Flow Operations Wolfgang Hummel DMO/DFO/QC group Lander de Bilbao, SDD/pipeline group Andrea Modigliani,

National Center for Supercomputing Applications Observational Astronomy NCSA projects radio astronomy: CARMA & SKA optical astronomy: DES & LSST access:

Calibration Plan for Early DES Data

AMICA Antarctic Multiband Infrared CAmera Favio Bortoletto INAF – Osservatorio Astronomico di Padova on behalf of the AMICA collaboration Telescope and.

Jim Lewis and Guy Rixon, CASU. 24 April, 2001 Data-reduction Pipeline for the INT WFC: slide 1 The Data-reduction Pipeline for the INT Wide Field Camera.

Photometry and Astrometry of SIM Planetquest Globular Cluster Targets T. M. Girard (Yale), A. Sarajedini (U. Florida), B. Chaboyer (Dartmouth) Table 1.

Spectroscopic Observations (Massey & Hanson 2011, arXiv v2.pdf) Examples of Spectrographs Spectroscopy with CCDs Data Reduction and Calibration.

Palomar Transient Factory Data Flow Jason Surace IPAC/Caltech.

Tutorial T-10 Amber Data Reduction (practical point of view) EuroSummer School Observation and data reduction with the Very Large Telescope Interferometer.

December 16, NOAO Mosaic Pipeline CoDR NOAO Mosaic Pipeline Technical Presentation.

Pachon Sky Camera Christopher Stubbs Chuck Claver Jan 25, 2014.

EScience May 2007 From Photons to Petabytes: Astronomy in the Era of Large Scale Surveys and Virtual Observatories R. Chris Smith NOAO/CTIO, LSST.

PTF Data Caveats and Cautionary Notes Mansi M. Kasliwal Carnegie Institution for Science California Institute of Technology Thanks to Richard Walters,

Data products of GuoShouJing telescope(LAMOST) pipeline and current problems LUO LAMOST Workshop.

1 Groningen, November 2003 ASTROWISE OAC TEAM The ASTRO-WISE project: status at OAC The OAC AW team: J.M. Alcalà, F. Getman, A. Grado, M. Pavlov,

Data Analysis Software Development Hisanori Furusawa ADC, NAOJ For HSC analysis software team 1.

1 The DES Calibrations Effort Douglas L. Tucker (DES Calibrations Scientist) DES NSF/DOE Review, 8-9 June 2009 The DES Calibrations Effort has connections.

MIRI Dither Patterns Christine H Chen. Dithering Goals 1.Mitigate the effect of bad pixels 2.Obtain sub-pixel sampling 3.Self-calibrate data if changing.

April 2001 OPTICON workshop in Nice 1 The PSF homogenization problem in large imaging surveys Emmanuel BERTIN (TERAPIX)

1 Imaging Surveys: Goals/Challenges May 12, 2005 Luiz da Costa European Southern Observatory.

From photons to catalogs. Cosmological survey in visible/near IR light using 4 complementary techniques to characterize dark energy: I. Cluster Counts.

ASKAP: Setting the scene Max Voronkov ASKAP Computing 23 rd August 2010.

August 10, 2004 Apache Point Observatory, NM FINDING SUPERNOVAE IN A SLICE OF PI Dennis J. Lamenti San Francisco State University.

NIRISS Calibration Working Group Activities NIRISS Science Team Meeting Montréal, 2015 October 20/21 Kevin Volk STScI / HIA.

LSST and VOEvent VOEvent Workshop Pasadena, CA April 13-14, 2005 Tim Axelrod University of Arizona.

Multiobject Spectroscopy: Preparing and performing Michael Balogh University of Durham.

Pan-STARRS Seminar: IPPEugene Magnier Pan-STARRS Image Processing Pipeline An Overview IFA Pan-STARRS Seminar 735 October 6, 2004.

Distributed Pipeline Programming for Mosaics Or Mario Tips’N’Tricks.

IPHAS Early Data Release E. A. Gonzalez-Solares IPHAS Consortium AstroGrid National Astronomy Meeting, 2007.

1 AWAIC: A WISE Astronomical Image Co-adder Frank Masci Infrared Processing and Analysis Center / Caltech.

STScI 2010 Calibration Workshop The NICMOS Legacy Archival Recalibration Project Anton M. Koekemoer and the STScI NICMOS Team (E. Barker, E. Bergeron,

GSPC -II Program GOAL: extend GSPC-I photometry to B = V ˜ 20 add R band to calibrate red second-epoch surveys HOW: take B,V,R CCD exposures centered at.

1 Core Data Processing Software Plan Review – University of Washington, Seattle, WA – Sept th Data Management XXVIII IAU General Assembly.

Data Pipeline Wei Zheng Johns Hopkins University.

SINFONI data reduction using the ESO pipeline. Instrument design and its impact on the data (I) integral field spectrometer using mirrors brickwall pattern.

The GLORIA demonstrator experiment Ariel Majcher National Centre for Nuclear Research Warsaw, Poland XXXII IEEE-SPIE Joint Symposium Wilga 2013, May 29.

Pan-STARRS observational requirements specification.

Pi of the Sky off-line experiment with GLORIA Ariel Majcher National Centre for Nuclear Research Warsaw, Poland 10th INTEGRAL/BART Workshop, April.

Mark Hunten NOAO MI Program

A galaxy at redshift 10? Brigitta Eder Vera Könyves

Imaging data obtained with the

Integral Field Spectroscopy

The Image The pixels in the image The mask The resulting image 255 X

Spectroscopic Observations (Massey & Hanson 2011, arXiv v2

Presentation transcript:

ADASS XVII Sep 2007The NOAO Pipeline Applications Francisco Valdes (NOAO) Robert Swaters (UMd) Derec Scott (NOAO) Mark Dickinson (NOAO)

ADASS XVII Sep 2007 The NOAO Pipeline Applications Outline Pipeline System Pipeline Applications Mosaic Pipeline Functionality Structure and Parallelization Performance

ADASS XVII Sep 2007 The NOAO Pipeline Applications Pipeline System (E2E) Data Source Pipeline Framework Pipeline Applications Data Sink

ADASS XVII Sep 2007 The NOAO Pipeline Applications Pipeline Framework NOAO High Performance Pipeline System Parallel and Distributed Event Driven Multiple Pipeline Application Support (new) Node Manager, Pipeline Manager Data Manager, Calibration Manager Pipeline Scheduling Agent (new) See: Valdes, et al, 2006, The NOAO High-Performance Pipeline System (

ADASS XVII Sep 2007 The NOAO Pipeline Applications Data Source and Sink Telescope Data Handling System NEWFIRM Quick Reduce Pipeline NOAO Science Archive Source/Sink for Mosaic and NEWFIRM Science PLs Access for selected PI Mosaic PL Data Products Access for NEWFIRM Science PL Data Products

ADASS XVII Sep 2007 The NOAO Pipeline Applications Pipeline Applications Current NOAO CCD Mosaic Imagers 2 cameras, 8 x 2K x 4K SITe CCDs (64 Mpix) 36 x 36 arcmin 4m PF NOAO Extremely Wide-Field IR Mosaic 4 x 2K x 2K Orion InSb Arrays (32 Mpix) 27.6 x 27.6 arcmin 4m CF Future Enhancements: Stacking, Catalogs, Transients WIYN One Degree Imager (1 Gpix)

ADASS XVII Sep 2007 The NOAO Pipeline Applications Mosaic and NEWFIRM

ADASS XVII Sep 2007 The NOAO Pipeline Applications NEWFIRM Pipeline Application Quick reduction pipeline at telescope Science pipeline in Tucson Photometric and astrometric calibrations: including: non-linearity and sky subtraction Registration and stacking See: Swaters, et al., P1.141

ADASS XVII Sep 2007 The NOAO Pipeline Applications Mosaic Pipeline Application Functionality and Data Products Bad pixel mapping and cleaning Flux calibration Astrometric calibration Data quality characterization Dome and sky calibration files Single exposures in multi-extension format Resampled standard image format Bad pixel masks PNG images (preview and near full scale)

ADASS XVII Sep 2007 The NOAO Pipeline Applications

ADASS XVII Sep 2007 The NOAO Pipeline Applications Mosaic Pipeline Application Major Pipeline Groups Staging: PSQ, fetching, archiving, ftp, clean-up Dome Calibration Sequences: bias, dome flat, … Basic Individual Exposure Calibration: CCD proc Sky Calibration: pupil, fringe, dark sky flat Resampling: sinc interpolation to standard image Data Products: MEF, resampled, masks, … 22 pipelines, 165 stages

ADASS XVII Sep 2007 The NOAO Pipeline Applications Block of Nights Mosaic Parallelization Basic Cal … Data Products Sky Cals Resample 1..* 1 Night /1 Filter

ADASS XVII Sep 2007 The NOAO Pipeline Applications 1 Mosaic 1 CCD Mosaic Parallelization Basic Calibration 1 Night /1 Filter 1..* crosstalk WCS mask overscan bias dome flat merge

ADASS XVII Sep 2007 The NOAO Pipeline Applications Sky Template Apply 1 CCD 1..* Mosaic Parallelization Sky Calibrations 1 Night /1 Filter 1 Night /1 CCD 1..* Sky Template Apply 1 CCD 1..* Sky Template Apply 1 CCD 1..* create or use library pupil reflection fringing dark sky flat coverage analysis

ADASS XVII Sep 2007 The NOAO Pipeline Applications Mosaic Parallelization Resampling 1 Night /1 Filter 1 Night /1 CCD Tangent Points Resample 1 CCD 1..* grouping sky grid

ADASS XVII Sep 2007 The NOAO Pipeline Applications Data Products Mosaic Parallelization Data Products 1 Night /1 Filter 1 Mosaic 1..* MEF file mosaic image masks pngs archive preview html tar file (bg) ftp stage (bg)

ADASS XVII Sep 2007 The NOAO Pipeline Applications NOAO Pipeline Cluster 1 data manager 8 processing nodes Dual-CPU Athelon rack mounted nodes Linux (Redhat 9) Raid Level 0 disks Performance

ADASS XVII Sep 2007 The NOAO Pipeline Applications

ADASS XVII Sep 2007 The NOAO Pipeline Applications

ADASS XVII Sep 2007 The NOAO Pipeline Applications

ADASS XVII Sep 2007 The NOAO Pipeline Applications

ADASS XVII Sep 2007 The NOAO Pipeline Applications See Also P1.141, Swaters, NEWFIRM Pipeline NOAO Demo Pipeline demonstration