page 1 PACS NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Overview of SPIRE Photometer Pipeline Kevin Xu NHSC/IPAC on behalf of the SPIRE ICC

PACS page 2 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 The Goal Show how SPIRE Photometer pipeline works (functionalities of major modules). Reference: “SPIRE Data Reduction Guide” in HIPE (under “Help”) or in:

PACS page 3 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug ’5’ Pipeline & Data Products Level 2.5 Data Product Level 3 Data Product Maps Maps of combined parallel observations with cross-scans Maps of overlap observations belonging to the same proposal New in HIPE 11 (only in archive data from HSA) Scan Map Pipe- line (for small- map, large-map, & SPIRE/PACS parallel modes). (in HIPE)

PACS page 4 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart A “reversed sequence” relative to the chain of data acquisition Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching

PACS page 5 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching A “reversed sequence” relative to the chain of data acquisition significant for very bright sources

PACS page 6 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching flag thermistors affected by “signal jumps” A “reversed sequence” relative to the chain of data acquisition

PACS page 7 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching A “reversed sequence” relative to the chain of data acquisition detect & remove glitches shared by multiple detectors

PACS page 8 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching A “reversed sequence” relative to the chain of data acquisition Single detector deglitcher Alternative: σ-κ deglitcher

PACS page 9 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching A “reversed sequence” relative to the chain of data acquisition FFT based (run after deglitch to avoid ringing)

PACS page 10 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching A “reversed sequence” relative to the chain of data acquisition Include: (1) Non-linearity correction (2) Volt to Jy/beam conversion

PACS page 11 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching Remove 1/f noise due to T-drift A “reversed sequence” relative to the chain of data acquisition

PACS page 12 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching Remove 1/f noise due to T-drift A “reversed sequence” relative to the chain of data acquisition flag thermistors affected by “signal jumps” Exclude flagged thermistors

PACS page 13 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching FFT based A “reversed sequence” relative to the chain of data acquisition

PACS page 14 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Map Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching A “reversed sequence” relative to the chain of data acquisition telescope pointing & orientation SPIRE detectors positions

PACS page 15 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Scan Pipeline Flow Chart Level 0.5 Products Photometer Detector Timeline Remove electrical crosstalk Electric crosstalk matrix Concurrent deglitcher Wavelet deglitcher Thermistor Jump detector Correct for electrical filter Convert to flux density Temperature drift correction Temperature derivatives LPF parameters Conversion table Correlation parameters First level deglitching A “reversed sequence” relative to the chain of data acquisition Destriper + NaiveMapper

PACS page 16 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Destriper: Offsets Removal Destriper: remove the relative offsets of timelines of individual bolometers by minimizing the dispersions in overlap sky pixels (using the Naïve-Mapper iteratively).

PACS page 17 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 SPIRE Naïve Mapper signal samplings Sky pixel Two options: (1) No weighting (pipeline default): Flux of a sky pixel is the simple average of all signal samplings (by all bolometers) in the pixel: f3f3 f1f1 f2f2 f4f4 f5f5 (2) Inverse variance (of instrument noise) weighted: Flux of a sky pixel is the inverse variance weighted mean of all signal samplings in the pixel, the variance is calculated using the white noise of the bolometer with which a given sampling is taken:,

PACS page 18 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Details on Level 2 Products Generation point source maps (PLW, PMW, PSW) extended emission maps sso maps (only for solar system objects) only for sso: change to extended emission calibraiton

PACS page 19 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Status of SPIRE Photometer Pipeline in HIPE 11.0 General assessment Overall, It works very well. In most cases, data from HSA are already of science quality! The official calibration accuracy is ±6% (4% from model, 2% RMS). An example (on the right): The image from HSA looks good. SPIRE 3-color map of NGC 5315 (a planetary nebula) (Public data taken from HSA)

PACS page 20 NHSC Data Processing Workshop – Pasadena 26 th - 30 th Aug 2013 Summary SPIRE Photometer Scan Map Pipeline handles data in the following observational modes: Small Map, Large Lap, SPIRE/PACS Parallel Mode. Corrections for instrumental effects (between Level 0.5 and Level 1 products) follow a “reversed sequence” relative to the chain of data acquisition. The map-making (between Level 1 and Level 2 products) is carried out using a Destriper-NaiveMapper combination. The current pipeline (HIPE ) does a good job (“science ready”) in general. In subsequent talks, we will discuss how to use the “user pipelines” (HIPE scripts) and other tools in HIPE to resolve data problems caused by known issues.