Herschel Open Time Cycle 1 DP workshop ESAC, March page 1 Spurs in HIFI data Colin Borys

- page 2 Artifacts in HIFI Data Generally, there are three major cleaning exercises with HIFI data. 1.Baseline subtraction 2.Standing Wave removal 3.Spur flagging In this talk/demo, we concentrate on spurious response in HIFI.

- page 3 Spur Definition and Detection For the purposes of this discussion, we define spurs as those features which are generally Gaussian in shape and narrow (<50MHz). They can look like astronomical lines, and have the potential to ‘fool’ users.

- page 4 Related artifacts Other artifacts due to saturated detectors or under pumped mixers can also corrupt HIFI spectra, but are much more obvious and easy to remove.

- page 5 Spur Detection in HIFI SPG Because spurs can look like real lines, we search for them only in the loads. The algorithm used in the standard HIFI pipeline is as follows: Loop through each WBS HTP at level 0.5 Loop through each cold load within this HTP Loop through each subband in the load dataset 1. Flag saturated pixels 2. Compute second derivative of the load spectrum 3. Mark points in the 2 nd derivative that exceed a threshold and fit as a spur. 4. Flag the spur, and record parameters in a table (trend analysis section of obs context). As part of the level 0.5->level 1 pipeline, flux calibration is performed and as a result flags in the loads are propagated naturally to the science data.

- page 6 The obs context spur table obs=getObservation(" ”,poolName=" _sscan_dbs_1b_r4")

- page 7 The obs context spur table obs=getObservation(" ”,poolName=" _sscan_dbs_1b_r4")

- page 8 HSPOT and Spur Reporting Based on repeated observations, we find that spur positions (LO vs IF) are generally stable and repeatable. Weak spurs (in particular band 4b) can come and go based on the conditions of the instrument at the time of observation. This means we warn users at the HSPOT planning stage if their observation is in danger of being corrupted by a spur.

- page 9 Diverse Spur Behavior Strong spurs, such as the (now fixed) spur in 1a, can knock out an entire dataset. Weak spurs can impact only a very narrow region (4b for example) Some spurs have a predictable position in the IF as a function of the LO frequency.

- page 10 More spur pathology 1.Some spurs can move in IF over time. If an observation goes on a long time before a LO retune, the spur can ‘migrate’ out of the flagged region and complicate data reduction (most notably for spectral scans). 2.Spurs can be missed by the algorithm if the load is not as ‘smooth’ as expected. Such data need to be flagged manually, and we discuss this in the demo.

- page 11 Spur impact on deconvolution Historically, the spur finder was developed for use in spectral surveys, where artifacts such as these can significantly corrupt the result from the deconvolution. It has now matured into a full featured, automated pipeline module for ALL observations, regardless of mode. Deconvolved output from data with an unflagged spur. The iterative nature of the algorithm propagates the spur over the entire spectral range. Same spectrum, this time with the spur masked out. The result reveals some baseline issues that now need to be addressed. Final result after simple median baseline subtraction.

Herschel Open Time Cycle 1 DP workshop ESAC, March page 12 Flagging data

- page 13 Overview Whether it be spurs, overtly bad data, or some other reason, there are two main ways to flag data for exclusion 1.Channel level flags (via HIFI.MASK) 2.Dataset level flags (via ROWFLAGS) Currently we do not have user friendly tools to let you flag data, but it is relatively straightforward via scripts, and one can (in some cases) use the SpectrumExplorer As with other scripts shown in this workshop, the general idea is to extract the data you want to manipulate, deal with it, then re-insert into the obs context.

- page 14 Tool dependent flag response Currently, HIPE tools respond differently to different flags. Deconvolution of SSCAN, for example, handles row flags and channel flags. RemoveFlaggedPixels is a task that lets users quickly remove flagged data via setting to NaN or interpolation HIPE help search on ‘rowflags’ will help you parse what flags correspond to.