HIFI Tutorial 1: Running the HIFI Pipelines and Adaptations A.P.Marston ESAC 27 June 2013

Document title | Author Name | Place | Data doc | Programme | Pag. 2 HIFI Pipelines: General  Pipelines for all observing modes run up to at least level 2 – some go beyond this, where there is further (originally not foreseen) processing. Maps – final map cubes provided Spectral scans – deconvolution is applied “blindly” based on automatic flagging of data settings (indicates where problem data should be ignored).  Level 2 get to the situation of final DSB spectra (includes reference spectra removal).  The idea is that we also have removed all instrumentatl effects by the time we get to level 2 data. However, this is an ideal and it is not always possible without some further human interactions.

HIFI Pipeline Process Description 1.Take data from the satellite and minimally manipulate it into time ordered Data Frames (a HifiTimelineProduct, or HTP, for each spectrometer). This is a Level 0 data Product, which is the least processed data available to the scientific community. 2.Remove backend instrumental effects - essentially a frequency calibration. There are separate pipelines for the WBS and HRS spectrometers, and the result is a Level 0.5 Product. You will not see this Product in the ObservationContext unless the generation of a Level 1 product fails. However, you can generate it for yourself when reprocessing. 3.Application of observing mode specific calibrations, i.e., subtraction of reference and off positions and intensity calibration using Hot/Cold loads. This is done by the Level 1 pipeline and resulting Level 1 Products are sets of frequency and intensity calibrated spectra. 4.The Level 2 pipeline removes further instrumental effects by converting to antenna temperature, applying side-band gain corrections, and converting to sky frequency. Spectra at Level 2 are averaged according to LO setting and position. 5.The Level 2.5 pipeline combines the Level 2 products into final products for each observing mode: averaged spectra for point mode (as for Level 2), cubes for mapping observations and deconvolved single sideband spectra for spectral scans. Document title | Author Name | Place | Data doc | Programme | Pag. 3

HIFI Pipelines: Re-pipelining (see HIFI Data Reduction Guide, chapter 5)  Why repipeline?: Using different calibration set. Use more up to date pipeline  Basic repipeline:  Need access to data in HIPE. Can be brought into session from a local disk storage. Or from archive (in one way or another).  Applicable task is hifiPipeline  Double-clicking on this task brings up a GUI interface that allows full reprocessing from the basic level 0 data. It requires information contained in the calibration part of the observation context. Be aware that some calibrations are regenerated in the pipeline when it is rerun – e.g. measurements of system temperature. Reprocessing this way does the same as original pipeline. Document title | Author Name | Place | Data doc | Programme | Pag. 4

Running pipeline with changes  Updated calibration.  If you want to make sure to use the most recent pipeline then you can get it from the HSA. cal = configureHifiPipeline(useHsa=True)  Then run with this calibration. For example obs = getObservation( ,useHsa=True) cal = configureHifiPipeline(useHsa=True) obs_1 = hifiPipeline(obs=obs, cal=True, palStore=cal) # gets latest cal in HSA  Or a specific calibration obs_1 = hifiPipeline(obs=obs, cal=True, palStore=cal, calVersion="HIFI_CAL_7_0")  Updated auxiliary products (e.g. pointing information) obs_1 = hifiPipeline(obs=obs, cal=True, aux=True, palStore=cal) Document title | Author Name | Place | Data doc | Programme | Pag. 5

Storing calibrations  You can store calibrations on your disk hifical = getHifiCal(useHsa=True) # gets HIFI calibration and puts it into “hifi-cal” on your local disk – WARNING: this takes some time.  To pass this pool for use in the pipeline (adding pipeline-out provides a place for anything written out in the reprocessing to be stored – put first in the list): store=ProductStorage(["pipeline-out","hifi-cal"]) obs_new = hifiPipeline(obs=obs, palStore=store, cal=True) Document title | Author Name | Place | Data doc | Programme | Pag. 6

Things you can choose to do in the pipelines (see HIFI Data Reduction Guide chapter 5.5)  To allow existing data processing tasks to be included as part of the pipeline. Include baseline or standing wave removal prior to gridding mapping data or deconvolving spectral scans, or you might want to add the doFold task at the end of the pipeline for Frequency Switch observations. Interactive level 2.5.  To omit steps from the pipeline or to use optional steps in the pipeline. For example, you may wish to omit the averaging step at the end of the Level 2 pipeline or to turn on the optional doFilterLoads Level 1 pipeline step to help mitigate standing waves coming from the calibration loads. Level 1 pipeline option. In the Level 2 pipeline you could choose to use doMainBeamTemp to convert to main beam temperature instead of the default antenna temperature. Level 2 pipeline option.  To use alternative options in the default steps of the pipeline. One example includes choosing a different order Polynomial to fit or a different interpolation method in the mkOffSmooth step of the Level 1 pipeline in order to improve baselines in the data. Document title | Author Name | Place | Data doc | Programme | Pag. 7

Example Options:  fitBaseline (see HIFI DRG chapter 11.3):  Adding a fitted baseline: level 2.5 interactive pipeline option.  Drag task into level 2.5 interactive area – adds to default 2.5 pipeline. Edit box to identify how you want to run the fitting – note can be interactive or “lights out”  doFilterLoads (in level 1 pipeline – smooth filter load calibration information): Very useful for reducing the effects seen in high continuum data – e.g. from planets (see Elena’s presentation).  doAvg (in level 2 pipeline – so combination of observations to create final product or not).  doMainBeamTemperature (add to level 2.5 pipeline).  doGridding (for maps up to level 2.5) Document title | Author Name | Place | Data doc | Programme | Pag. 8

Running parts of the pipeline  It is not always desirable to run the whole pipeline.  For instance just the configurable level 2.5 pipeline would be fine.  Easily achieved using hifiPipeline.  Set fromLevel to 2.0  Set upToLevel to 2.5 Document title | Author Name | Place | Data doc | Programme | Pag. 9

Saving the final products To FITS on disk.  You need to select spectra one by one to save as FITS. Or better (easier to retrieve) to “pool” (basically a directory) on your hard disk  Right-click on the variable and go to “Send to …” on the menu. Choose “local pool” Export to CLASS using hiClass.  Right-click on the variable and go to “Send to …” on the menu. Choose “hiClass” This produces an output that is in CLASS format FITS file. Document title | Author Name | Place | Data doc | Programme | Pag. 10