Presentation is loading. Please wait.

Presentation is loading. Please wait.

26 January 2007 CAL 07 Garching 1 The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit.

Published byPatrick Beswick Modified over 10 years ago

Similar presentations

Presentation on theme: "26 January 2007 CAL 07 Garching 1 The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit."— Presentation transcript:

1 26 January 2007 CAL 07 Garching 1 The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit

2 26 January 2007CAL 07 Garching2 Introduction to CASU Small group within the Institute of Astronomy, specialising in survey astronomy. Small group within the Institute of Astronomy, specialising in survey astronomy. Pipeline reduction of imaging data Pipeline reduction of imaging data APM (Schmidt Plates 1 x 40k x 40k) APM (Schmidt Plates 1 x 40k x 40k) INT Wide Field Camera (4 x 2k x 4k) INT Wide Field Camera (4 x 2k x 4k) ESO WFI on 2.2m at La Silla (8 x 2k x 4k) ESO WFI on 2.2m at La Silla (8 x 2k x 4k) MOSAIC-1 on KPNO 4m (8 x 2k x 4k) MOSAIC-1 on KPNO 4m (8 x 2k x 4k) MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k) MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k) AAO WFI on AAT (8 x 2k x 4k) AAO WFI on AAT (8 x 2k x 4k) CIRSI on INT (4 x 1k x 1k) CIRSI on INT (4 x 1k x 1k) INGRID on WHT (1k x 1k) INGRID on WHT (1k x 1k) UFTI on UKIRT (1k x 1k) UFTI on UKIRT (1k x 1k) WFCAM on UKIRT (4 x 2k x 2k) WFCAM on UKIRT (4 x 2k x 2k)

3 26 January 2007CAL 07 Garching3 Introduction to VDFS PPARC funded facility to provide an end-to-end data-flow system for VISTA and WFCAM. PPARC funded facility to provide an end-to-end data-flow system for VISTA and WFCAM. Quality control and calibration pipelines Quality control and calibration pipelines Paranal and Garching Paranal and Garching Science pipeline for full calibration of science data. Science pipeline for full calibration of science data. Cambridge Cambridge Science archive acts as the point of access of the reduced data. Plus some further processing. Science archive acts as the point of access of the reduced data. Plus some further processing. WFAU, Edinburgh WFAU, Edinburgh

4 26 January 2007CAL 07 Garching4 Data Flow Raw telescope data is assessed by the summit pipeline (QC1) Raw telescope data is assessed by the summit pipeline (QC1) Shipped to Garching (discs) Shipped to Garching (discs) Shipped to CASU for science reduction and calibration. (discs) Shipped to CASU for science reduction and calibration. (discs) Calibrated data shipped to Edinburgh for archiving (ftp) Calibrated data shipped to Edinburgh for archiving (ftp)

5 26 January 2007CAL 07 Garching5 IR Data Reduction Worries IR detectors are currently inherently more unstable than optical CCDs. IR detectors are currently inherently more unstable than optical CCDs. Some odd electronic effects Some odd electronic effects Sky emission > 100x brighter than most objects Sky emission > 100x brighter than most objects And its variable both spatially and temporally! And its variable both spatially and temporally! Exposure times are short, so data rates are very high. Exposure times are short, so data rates are very high. 200-500 Gb/night expected for VISTA public surveys 200-500 Gb/night expected for VISTA public surveys Rice tile compression can save factors of 3-4 in 32 bit integer data Rice tile compression can save factors of 3-4 in 32 bit integer data

6 26 January 2007CAL 07 Garching6 VDFS Pipeline Recipes Create master calibration frames (dark, twilight flats, confidence maps, etc) Create master calibration frames (dark, twilight flats, confidence maps, etc) Linearity analysis Linearity analysis Detector noise & dark current properties Detector noise & dark current properties Persistence and crosstalk analysis Persistence and crosstalk analysis Illumination correction analysis Illumination correction analysis Full reduction recipes for standard star and programme fields Full reduction recipes for standard star and programme fields

7 26 January 2007CAL 07 Garching7 Processing Steps Reset correction (debias) Reset correction (debias) Linearity correction Linearity correction Dark and reset anomaly correction Dark and reset anomaly correction

8 26 January 2007CAL 07 Garching8 Reset Anomaly (WFCAM)

9 26 January 2007CAL 07 Garching9 Processing Steps Reset correction Reset correction Linearity correction Linearity correction Dark and reset anomaly correction Dark and reset anomaly correction Flat field correction Flat field correction Background correction (defringing) Background correction (defringing) Destriping Destriping

10 26 January 2007CAL 07 Garching10 Stripes Close Up (VISTA)

11 26 January 2007CAL 07 Garching11 Stripes (VISTA)

12 26 January 2007CAL 07 Garching12 Processing Steps Reset correction Reset correction Linearity correction Linearity correction Dark and reset anomaly correction Dark and reset anomaly correction Flat field correction Flat field correction Background estimation and subtraction (defringing) Background estimation and subtraction (defringing) Destriping Destriping Image persistence and detector crosstalk removal Image persistence and detector crosstalk removal

13 26 January 2007CAL 07 Garching13 Crosstalk (WFCAM)

14 26 January 2007CAL 07 Garching14 Processing Steps Reset correction Reset correction Linearity correction Linearity correction Dark and reset anomaly correction Dark and reset anomaly correction Flat field correction Flat field correction Background estimation and subtraction (defringing) Background estimation and subtraction (defringing) Destriping Destriping Image persistence and detector crosstalk removal Image persistence and detector crosstalk removal Interleaving Interleaving Dithering/Jittering Dithering/Jittering Catalogue generation Catalogue generation Astrometric calibration Astrometric calibration Photometric zeropoint calibration Photometric zeropoint calibration (Tiling) (Tiling)

15 26 January 2007CAL 07 Garching15 Summit & Garching Pipelines QC1 parameters QC1 parameters e.g. photometric zeropoints, astrometric fit quality e.g. photometric zeropoints, astrometric fit quality Written using ESO qfits/CPL infrastructure Written using ESO qfits/CPL infrastructure Both use the same software modules Both use the same software modules Reduce a pawprint Reduce a pawprint Amount of processing can be scaled down Amount of processing can be scaled down Calibration images (flats etc) from a master library. Calibration images (flats etc) from a master library.

16 26 January 2007CAL 07 Garching16 Cambridge Pipeline Full reduction Full reduction Tiling and sky correction Tiling and sky correction Catalogue generation is done for both pawprints and tiles Catalogue generation is done for both pawprints and tiles Results shipped to WFAU Results shipped to WFAU Some of the same CPL based modules reused. Some additional modules required. Some of the same CPL based modules reused. Some additional modules required.

17 26 January 2007CAL 07 Garching17 Current Status of VISTA Pipelines Version 0.4 ESO pipelines currently being tested Version 0.4 ESO pipelines currently being tested Version 0.5 to be released end of February Version 0.5 to be released end of February Full compliment of recipes (almost!) Full compliment of recipes (almost!) PAE PAE WFCAM pipeline is a prototype for the extra functionality required by Cambridge VISTA pipeline WFCAM pipeline is a prototype for the extra functionality required by Cambridge VISTA pipeline

18 26 January 2007CAL 07 Garching18 A Typical Recipe Run…

Download ppt "26 January 2007 CAL 07 Garching 1 The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit."

Similar presentations

DSLR AstrophotograpHy…. My way

DSLR AstrophotograpHy…. My way
Nicholas Cross, Rob Blake, Ross Collins, Mark Holliman, Mike Read, Eckhard Sutorius, Nigel Hambly, Andy Lawrence, Bob Mann, Keith Noddle Wide Field Astronomy.

Nicholas Cross, Rob Blake, Ross Collins, Mark Holliman, Mike Read, Eckhard Sutorius, Nigel Hambly, Andy Lawrence, Bob Mann, Keith Noddle Wide Field Astronomy.
July 2006Instrument Scientist Retreat, Santiago Quality Control: closing the loop Reinhard Hanuschik, DFO/QC 1. Principles 2. Processes 3. Critical components.

July 2006Instrument Scientist Retreat, Santiago Quality Control: closing the loop Reinhard Hanuschik, DFO/QC 1. Principles 2. Processes 3. Critical components.
VISTA as seen by the Prime Contractor Adrian Russell.

VISTA as seen by the Prime Contractor Adrian Russell.
ADASS XVII Sep 2007The NOAO Pipeline Applications Francisco Valdes (NOAO) Robert Swaters (UMd) Derec Scott (NOAO) Mark Dickinson (NOAO)

ADASS XVII Sep 2007The NOAO Pipeline Applications Francisco Valdes (NOAO) Robert Swaters (UMd) Derec Scott (NOAO) Mark Dickinson (NOAO)
Panoramic near-Infrared Astronomy A first look at the UKIDSS data Steve Warren UKIDSS Survey Scientist with Steve Weatherley (Deteca Ltd), Paul Hewett.

Panoramic near-Infrared Astronomy A first look at the UKIDSS data Steve Warren UKIDSS Survey Scientist with Steve Weatherley (Deteca Ltd), Paul Hewett.
Data processing of filament observations using a tunable H-a filter Hyungmin Park 1,3, Jongchul Chae 2, Yong-Jae Moon 3, Young-Deuk Park 3 1 Dept. of Astronomy.

Data processing of filament observations using a tunable H-a filter Hyungmin Park 1,3, Jongchul Chae 2, Yong-Jae Moon 3, Young-Deuk Park 3 1 Dept. of Astronomy.
Echelle spectra reduction with IRAF*

Echelle spectra reduction with IRAF*
VISTA/WFCAM pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

VISTA/WFCAM pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,
FMOS Observations and Data 14 January 2004 FMOS Science Workshop.

FMOS Observations and Data 14 January 2004 FMOS Science Workshop.
Nicholas Cross, Rob Blake, Ross Collins, Mark Holliman, Mike Read, Eckhard Sutorius, Nigel Hambly, Andy Lawrence, Bob Mann, Keith Noddle Wide Field Astronomy.

Nicholas Cross, Rob Blake, Ross Collins, Mark Holliman, Mike Read, Eckhard Sutorius, Nigel Hambly, Andy Lawrence, Bob Mann, Keith Noddle Wide Field Astronomy.
HAWCPol / SuperHAWC Software & Operations J. Dotson July 28, 2007.

HAWCPol / SuperHAWC Software & Operations J. Dotson July 28, 2007.
AEOS Telescope + Image-Slicer/ Spectrograph Data Reduction and Results for June 8, 2006.

AEOS Telescope + Image-Slicer/ Spectrograph Data Reduction and Results for June 8, 2006.
10Tb 100Tb 1000Tb. WFCAM processing summary semesters: 05A, 05B, 06A, 06B, 07A nights: 81, 166, 95, 157, 50 (68%) 50 Tbytes raw data 17 Tbytes compressed.

10Tb 100Tb 1000Tb. WFCAM processing summary semesters: 05A, 05B, 06A, 06B, 07A nights: 81, 166, 95, 157, 50 (68%) 50 Tbytes raw data 17 Tbytes compressed.
15 December 2008Science from UKIDSS II WFCAM Science Pipeline Update WFCAM Science Pipeline Update Jim Lewis, Mike Irwin & Marco Riello Cambridge Astronomy.

15 December 2008Science from UKIDSS II WFCAM Science Pipeline Update WFCAM Science Pipeline Update Jim Lewis, Mike Irwin & Marco Riello Cambridge Astronomy.
VISTA pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,

VISTA pipelines summit pipeline: real time DQC verified raw product to Garching standard pipeline: instrumental signature removal, catalogue production,
Extracting the Mystery from the Red Rectangle Meghan Canning, Zoran Ninkov, and Robert Slawson Chester Carlson Center for Imaging Science Rochester Institute.

Extracting the Mystery from the Red Rectangle Meghan Canning, Zoran Ninkov, and Robert Slawson Chester Carlson Center for Imaging Science Rochester Institute.
WFAU Tapes === Store 6 x dual PCs 3GHz Xeons 6 x 1 Tbyte local disks 2 Tbyte ingest buffer 10 Tbytes of raw 20 Tbytes of processed WFCAM archive storage.

WFAU Tapes === Store 6 x dual PCs 3GHz Xeons 6 x 1 Tbyte local disks 2 Tbyte ingest buffer 10 Tbytes of raw 20 Tbytes of processed WFCAM archive storage.
Introduction to Spitzer and some applications Data products Pipelines Preliminary work K. Nilsson, J.M. Castro Cerón, J.P.U. Fynbo, D.J. Watson, J. Hjorth.

Introduction to Spitzer and some applications Data products Pipelines Preliminary work K. Nilsson, J.M. Castro Cerón, J.P.U. Fynbo, D.J. Watson, J. Hjorth.
Signal vs Noise: Image Calibration First… some terminology:  Light Frame: The individual pictures you take of your target.  Dark Frame: An image taken.

Signal vs Noise: Image Calibration First… some terminology:  Light Frame: The individual pictures you take of your target.  Dark Frame: An image taken.

Similar presentations

Ads by Google