1. SPIRE Consortium Meeting La Palma, Oct. 1 – 2 2008 SPIRE FTS Pipeline Trevor Fulton Blue Sky Spectroscopy, Lethbridge, Canada

2. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 2 Spectrometer Pipeline Point source/sparse map (SOF1/3) –A spectrum of a point source that is well centred on the central detectors of the FTS arrays and/or simultaneously obtain a sparse map of an area roughly 2'' in diameter. For sparse mapping of larger areas, a raster of point source observations will be made. Field mapping (SOF2/4) –To take a spectrum of a region of sky or an extended source that is within the FOV of the spectrometer – i.e. less than 2.6'' circular. This is achieved by using the beam steering mirror to perform a low-frequency jiggle and observing multiple interferograms at each point of the jiggle pattern. For fully-sampled mapping of larger areas, a raster of multiple jiggle maps will be observed. AOTs

3. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 3 Generic FTS Observation Time... – The Spectrometer detector arrays are pointed at a target via a movement of the Herschel telescope (Raster) and/or the Beam Steering Mirror (Jiggle). Pointing Building Block Observation Building Block – At each pointing, an FTS observation is performed by scanning the spectrometer mechanism

4. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 4 Spatial Sampling Options SparseIntermediateFull N BSM =1N BSM =4N BSM =16

5. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 5 Spectral Sampling Options

6. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 6 Spectrometer Pipeline The flow of the SPIRE Spectrometer data processing pipelines has been designed to follow the spectrometer AOTs. The overall structure of each pipeline maximizes the benefit of the redundant information that exists within each observation building block. The data products that will be made available to observers are derived from the output of specific pipeline modules that are at the logical breaks in the overall pipelines. Overview

7. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 7

8. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 8 FTS Pipeline Building Block Pipeline –Modify Timelines –Create Interferograms –Modify Interferograms –Transform Interferograms –Modify Spectra

9. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 9 Signal Timeline Product – Level-0.5 Product that is the input to the FTS pipeline – This product contains a timeline of the recorded signal for each detector (Voltage vs. Time).

10. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 10 Modify Timelines 1 st -Level Deglitching Glitch template and threshold V d-RMS (t) V2(t)V2(t) Clipping Correction V1(t)V1(t) V3(t)V3(t) Time Domain Phase Correction LPF Components Bolometer Time Constants Remove Electrical Crosstalk Electrical crosstalk matrix V5(t)V5(t) Non-Linearity Correction V4(t)V4(t) V6(t)V6(t) Temperature Drift Correction Reference Voltage Conversion Factors Thermal Fluctuation Timeline V th (t)

11. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 11 – Glitches are identified in the Detector Timelines by way of wavelet analysis (same as for Photometer 1 st -Level Deglitching) – Signal samples that are flagged as glitches are corrected within the wavelet analysis. First Level Deglitching V1(t)V1(t) 1 st -Level Deglitching Glitch template and threshold V d-RMS (t)

12. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 12 – Clipped signals are those whose raw ADC value = 0 or 2 16 -1 (65535). – These signal samples are corrected by way of an 8 th order polynomial fit to the neighbouring signal samples. Clipping Correction V2(t)V2(t) V1(t)V1(t)

13. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 13 – The combined effect of the low pass filters in the readout electronics and the thermal response of the bolometers gives rise to a delay in the measured signal samples. – The delay is quantified by a combination of the known filter parameters and empirically derived thermal time constants for each bolometer. – The delay per detector is removed by convolution. Time Domain Phase Correction V3(t)V3(t) LPF Components V2(t)V2(t) Bolometer Time Constants

14. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 14 Time Domain Phase Correction

15. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 15 – Electrical crosstalk assumptions: – Linear – Effects on primary detector are negligible – No x-talk between arrays Remove Electrical Crosstalk V4(t)V4(t) V3(t)V3(t) Electrical crosstalk matrix

16. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 16 – Similar correction as that in the Photometer pipelines. – No Flux Conversion at this point. Non-Linearity Correction V5(t)V5(t) Reference Voltage V4(t)V4(t) Conversion Factors

17. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 17 V6(t)V6(t) Temperature Drift Correction V5(t)V5(t) – V th (t), is derived from the thermometers (2 per BDA). Temperature Drift Correction Correlation Parameters

18. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 18 FTS Pipeline Building Block Pipeline –Modify Timelines –Create Interferograms Interferogram Creation –Modify Interferograms –Transform Interferograms –Modify Spectra

19. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 19 V6(x)V6(x) Interferogram Creation V6(t)V6(t) Create Interferograms Position of ZPD z(t),P(t)P(t) Obliquity Factor – The mechanism timeline is divided into a set of individual timelines – one per spectrometer scan for the building block. – The mechanism timelines are then regularized by way of interpolation and converted from MPD to OPD.

20. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 20 V7(x)V7(x) Interferogram Creation V6(t)V6(t) Create Interferograms Position of ZPD z(t),P(t)P(t) Obliquity Factor – The input signal timelines for each detector are then merged with the regularized mechanism timelines to create a set of interferograms (one interferogram per scan per detector).

21. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 21 Create Interferograms V

22. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 22 Interferogram Product Each Interferogram Product contains one interferogram (Voltage vs. Optical Path Difference) per scan per detector.

23. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 23 Modify Interferograms Telescope/SCAL Correction Reference Interferograms V8(x)V8(x) Baseline Removal V7(x)V7(x) Second Level Deglitching Glitch Threshold V6(x)V6(x) Phase Correction Apodization Optical Phase V9(x)V9(x) V 10 (x ) V 11 (x )

24. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 24 – The contributions to the derived interferograms from the telescope and from SCAL are removed from the measured interferograms by way of subtraction. Telescope/SCAL Removal V7(x)V7(x) Telescope/SCAL Correction V6(x)V6(x) Reference Interferograms

25. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 25 – The position-dependent baseline for each interferogram is first characterized – Low order polynomial – Low frequency components of the FT – The derived baseline is then removed from each interferogram by way of subtraction. Baseline Removal V8(x)V8(x) V7(x)V7(x)

26. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 26 Baseline Removal

27. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 27 – The interferograms for each detector are inspected and the statistical outliers are flagged as glitches. – Outliers are flagged on a positional basis using the MAD method. – Samples that have been deemed to be glitches are replaced by the average of the clean samples at that position for that detector. Second Level Deglitching V9(x)V9(x) V8(x)V8(x) Glitch threshold

28. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 28 – The symmetric portion of each interferogram is first transformed. – A low-order weighted fit is made to the measured in-band for each spectrum to quantify any phase that remains. – The phase is removed from the MR/LR spectra by multiplication and from the HR interferograms by convolution. Phase Correction V 10 (x) Phase Correction V9(x)V9(x) Optical Phase

29. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 29 – The phase is removed from the MR/LR spectra by multiplication – The phase is removed from the HR interferograms by convolution. Phase Correction V 10 (x) Phase Correction V9(x)V9(x) Optical Phase

30. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 30 – An apodization function is applied to each interferogram. – This reduces the effects of the Sinc ILS in the spectral to be derived. Apodization V 11 (x) Apodization V 10 (x)

31. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 31 Apodization

32. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 32 FTS Pipeline Building Block Pipeline –Modify Timelines –Create Interferograms –Modify Interferograms –Transform Interferograms Fourier Transform –Modify Spectra

33. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 33 – Each interferogram in the building block is transformed into a spectrum in this module. – The spectral sampling interval will be fixed – the value of which will depend on the requested resolution Transform Interferograms V 12 (σ) Fourier Transform V 11 (x)

34. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 34 Transform Interferograms V 12 (σ) Fourier Transform V 11 (x) Observation Type Spectral Sampling Interval [cm -1 ] Nyquist Frequency [cm -1 ] Low0.25200 Medium0.05200 High0.01200

35. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 35 Spectrum Product Each Spectrum Product contains one spectrum (Voltage vs. Wavenumber) per scan per detector.

36. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 36 Modify Spectra Spectral Response Spectral RSRF I 14 (σ ) Flux Conversion V 13 (σ) Optical Crosstalk Removal Optical Crosstalk Matrix V 12 (σ) Spectral Averaging Conversion Factors I 15 (σ ) I 16 (σ )

37. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 37 – The wavenumber- dependent RSRF for the Telescope→BDA path is removed from each of the measured spectra. Spectral Response V 13 (σ) Spectral Response V 12 (σ) Spectral RSRF

38. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 38 – Compare the derived spectra with those from a source with a known flux. – The ratio between the two gives the wavenumber-dependant factor by which the measured spectra must be multiplied to give flux- calibrated quantities. Flux Calibration I 14 (σ) Flux Calibration V 13 (σ) Flux Conversion Factors

39. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 39 – Now that the signal have been converted to units of optical power, an optical crosstalk correction matrix may be applied. Remove Optical Crosstalk I 15 (σ) Remove Optical Crosstalk I 14 (σ) Optical crosstalk matrix

40. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 40 – On a detector-by-detector and wavenumber-by-wavenumber basis, the spectra derived for all scans in the building block are averaged. – Outliers, flagged by MAD clipping, are optionally removed from the average (default setting is to remove outliers). Spectral Averaging I 15 (σ) Spectral Averaging I 15 (σ)

41. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 41 Average Spectrum Product One Average Spectrum Product per Observation Building Block Each product contains one spectrum (Flux vs. Wavenumber) per detector

42. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 42 – The average spectrum products from each building block are merged together to form a single spectral cube. – The cube is sampled on a regular grid in both spatial dimensions as well as in the spectral dimension. Create Spectral Cube Spectral Cube (x, y, σ) Spatial Regridding ASP 1 (σ)ASP 2 (σ)ASP n (σ)ASP 3 (σ) …

43. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 43 Create Spectral Cube

44. SPIRE Consortium Meeting La Palma 01 October 2008 SPIRE FTS Pipeline Trevor Fulton 44 Conclusions As for Photometer pipelines, the hard work is in producing good calibration files. Spectrometer Pipelines are currently undergoing phase one of the Scientific Validation. –This includes are review of the supporting documentation End-to-end pipeline test #4 (November 2008). Complete outstanding development/calibration products (Winter 2008/2009). Demonstration of the pipeline in its current implementation Thursday afternoon.