1. Reducing IFU Data ITSO DR Workshop, Thursday 5th May Nic Scott

2. Overview ›Part 1 – Overview of IFU data reduction -Useful resources, different kinds of IFU, “theory,” and overview of main steps ›Part 2 – Hands-on with WiFeS data -Understanding calibrations, reducing data and checking output ›Part 3 – Tips and tricks -Extra steps, common issues and solutions, and comments for specific IFUs 2

3. Part 1 - Overview

4. Useful websites ›The IFS wiki: http://ifs.wikidot.com/welcomehttp://ifs.wikidot.com/welcome ›Instrument specific sites: -OSIRIS: http://www2.keck.hawaii.edu/inst/osiris/ -WiFeS: http://rsaa.anu.edu.au/observatories/instruments/wide-field-spectrograph- wifes -KOALA: https://www.aao.gov.au/science/instruments/current/koala/overview -NIFS: https://www.gemini.edu/sciops/instruments/nifs/https://www.gemini.edu/sciops/instruments/nifs/ -GMOS IFU: https://www.gemini.edu/sciops/instruments/gmos/integral-field- spectroscopy ›IFS Surveys -Atlas3D: http://www-astro.physics.ox.ac.uk/atlas3d/ -CALIFA: http://califa.caha.es/ -SAMI: http://sami-survey.org/ -MANGA: http://www.sdss.org/surveys/manga/ 4

5. Useful resources ›Fits image/cube viewers: -QFitsView: http://www.mpe.mpg.de/~ott/QFitsView/ -ds9: http://ds9.si.edu/site/Home.html -Fv: http://heasarc.gsfc.nasa.gov/ftools/fv/ ›Data reduction tools: -Iraf -Python/IDL -p3d: http://p3d.sourceforge.net/http://p3d.sourceforge.net/ -Instrument-specific packages and pipelines ›Many tools developed for radio astronomy can be applied to optical IFS data – cube viewing software, disk fitting, line analysis etc. 5

6. Different kinds of IFU ›Fibre-fed -Much like a fibre-fed multi-object spectrograph ›Image slicer -Uses an array of mirrors to ‘slice’ the image into multiple long-slit spectra ›Lenselet array -Uses a lenslet array in the focal plane to divide up field, then carefully arranges spectra on CCD 6 MUSE SAURON SAMI

7. Fibre-feb IFUs ›Fibre-fed IFUs use multiple closely-spaced fibres to act as spaxels and capture individual spectra ›Pros: cheap, extremely flexible, easily reconfigurable, easy to arrange spectra on the CCD, can take advantage of fibre-MOS software and techniques ›Cons: low efficiency, fibres can’t be too close together, circular spaxels ›Can be coupled with a lenselet array to solve some of these problems, but with further efficiency loss ›Examples include: PPAK, KOALA, SAMI, GMOS 7 Croom et al. (2012)

8. Lenselet array IFUs ›Use an array of closely-packed lenses, a lenselet array, in the image plane to act as spaxels ›Pros: reasonable efficiency, square pixels, variable spatial scales, high fill-factor ›Cons: reasonable efficiency, arranging spectra on CCD can be challenging, crosstalk between spectra ›Examples: SAURON, OSIRIS 8 Bacon et al. (2000) Larkin et al. (2010)

9. Image slicer IFUs ›Use an array of mirrors to ‘slice’ the image plane into a series of long-slit spectra. Each slice acts as a single y-column, with the pixels along the slit providing the x-direction ›Pros: High efficiency (only reflections), can apply long-slit techniques and software, high fill- factor, fill CCD easily ›Cons: Expensive, complicated ›Examples: WiFeS, MUSE, SINFONI 9 Bacon et al. (2010/12)

10. Steps in IFS data reduction ›1: General calibrations -Standard steps to account for CCD and instrument effects: bias subtraction, flat fielding etc. Essentially the same as for non-IFS data ›2: Extraction of spectra -Identify and extract individual spectra from the CCD. This step is highly specific to different kinds of IFU ›3: Reassembly of spectra into a datacube -Identify the relative on-sky positions and wavelength calibration of the extracted spectra. Put all spectra on a common wavelength and spatial scale then assemble into a cube 10

11. General calibrations ›Effects to correct for: -CCD bias level -Dark current -CCD sensitivity variations -CCD illumination -Cosmic rays -Hot pixels and bad columns -Total telescope and instrument throughput ›Calibration frames required: -Bias -Dark -Flat lamp and/or twilight flat -Spectrophotometric standard star 11 ›Goal is to correct for instrumental effects i.e. go from electrons to photons emitted by the source ›Most of these steps can be done early on in the reduction process, and even benefit from being done before spectra extracted ›Throughput correction typically done as a later stage

12. Extraction of spectra ›The ‘trick’ of IFS is arranging three dimensions of data onto a two dimensional detector ›This can be done in very different ways depending on the IFS design: -Lenselet systems use a rectification matrix – a predetermined relationship between detector pixel and lenselets -Fibre-based systems typically use a tramline map combined with a known fibre profile -Slicer systems fit the slitlet profile, extracting each slitlet as an independent long-slit spectrum ›The end goal is a set of extracted spectra, either as Row Stacked Spectra (RSS), or multiple long-slit spectra 12 Childress et al (2014) Sharp et al (2015) Bacon et al (2000)

13. Assemble data cube ›The many one-dimensional (or two-dimensional for slicer IFUs) spectra need to be arranged into a single three-dimensional datacube (x,y,λ) ›All spectra need to be placed on a common wavelength scale – typically done by fitting an arc spectrum then interpolating to a constant Å /pixel scale ›Each spectrum needs to be assigned an (x,y) coordinate position in the datacube. This process depends on the IFU type: -For lenselet arrays, the position of each lenselet on the sky is known in advance and a simple matrix is used to arrange the extracted spectra into a cube -For fibre-based IFUs, spectra are arranged into a cube based on the known positions of each fibre -For image slicers, a trace or wire calibration is used to match the spatial position of pixels in each slitlet 13

14. Optional extra steps ›Sky subtraction -Many options for this e.g. nod-and-shuffle, offset sky, dedicated sky fibres or lenselets -Can be done before or after assembling data cube ›Flux calibration -Best done after cube creation -IFS data has very good flux calibration accuracy compared to fibre or long-slit data. Can fit model of star in 2d, therefore determine very accurate total flux ›Combining exposures -Can be done at or after cubing -Can be done to increase field-of-view (mosaicing), increase effective spatial resolution (dithering) or simply increase S/N 14

15. Part 2 – Hands on with WiFeS data

16. Preparation ›Checks that pywifes.py is in your python path: -Terminal -> ipython --pylab -> import pywifes ›If not: export PYTHONPATH = $PYTHONPATH:/path/to/code/folder ›Missing python dependencies? Install them with pip ›Make sure you have a fits viewer application installed 16

17. Overview of the pywifes pipeline ›Pywifes is a Python-based pipeline for reducing WiFeS data written mainly by Mike Childress – see Childress et al. (2014). ›Its really good! Easy to use, largely automated, produces high-quality data products ›You can download the latest version here: http://www.mso.anu.edu.au/pywifes/doku.php ›NB We are not using the latest version today. The WiFeS data provided was taken in 2012, instrument and pipeline upgrades since then mean an older pipeline version is required to reduce this data. Your own installation of pywifes will likely not work for this exercise. ›Pywifes can simply reduce your data in one go. However, we’re going to take it one step at a time so we can check the output at each stage. 17

18. Check your raw data! ›Before starting reduction visually inspect the raw data (this should probably have been done when observing) ›Things to look out for: -Saturated exposures -Files that aren’t what they claim to be -Exposures with no or very low flux -Excessive amounts of cosmic rays -Weird artifacts ›Calibration files with issues must be removed or fixed before starting reduction – including bad calibrations makes your final data worse 18

19. Setting up the reduction ›NB I’ve already done this step for you ›Need to tell the DR pipeline which files you want to use ›Pipeline scripts will automatically identify file types – but check that it has done this right ›Manually edit file to stop the pipeline using them in reduction – alternatively, simply move them out of the reduction folder 19

20. Running the reduction script ›Open reduce_blue_data.py in the reduction scripts folder ›Set the first step, ‘run’:True. All others should be False. ›Execute the reduction script: ›Reduced files will be created in the Data/Reduced/ folder ›*.p??.fits indicates which step in the reduction process a file is ›NB To disable a reduction step, comment it out using #s in the reduce_blue_data.py file 20

21. Bias subtraction ›Overscan_sub and bias_sub subtract the zero-point of the CCD from all reduction files ›Overscan_sub does this on a region of the CCD that is not observed. Bias does this based on a separate bias combined observation ›Check the combined bias frame for artifacts! 21

22. Flat creation ›These steps combine the lamp and twilight observations into single high- S/N combined frames ›Then extract the flat frame for each slitlet ›Inspect the super_domeflat and super_twiflat frames to check for artifacts ›Check one or two of the extensions in the super_domeflat_mef and super_twiflat_mef files to check extraction worked (use Open as… Multi-Extension Cube… in ds9) 22

23. Spectral extraction ›These steps determine the slitlet profile using the flat frames ›Once the profiles are determined, each slitlet is extracted and repackaged as a Mulit-Extension Fits (MEF) file ›This step is also applied to the super flat frames ›NB The slitlet_defs file claims to be.fits but is really a.pkl. Use a text editor to view it and modify if necessary (this is fixed in current versions of pywifes) 23

24. Wavelength calibration ›Using the arc frame, fits to the positions of peaks ›Matches these to a line list for the given lamp ›Fits a smooth polynomial to the list of peak positions vs line wavelengths and stores this as the wavelength solution for each slitlet ›Check terminal output has sensible values, and wave_soln.fits image varies smoothly 24

25. Wire solution ›Wire frame is a flat with a single-pixel width mask placed in the focal plane ›Purpose is to match corresponding y- pixels in different slitlets ›Position of the wire (a trough) is fit in each slitlet then recorded in wire_soln ›Output is slightly counterintuitive – it’s a n λ * n slitlets “image” where the value of each pixel is the fitted position of the wire at that position ›Check to see wire positions are consistent from slitlet to slitlet and don’t vary much with wavelength 25

26. Flat fielding ›Attempt to mask cosmic rays using the LACOSMIC routine ›Pretty good, but not perfect at finding cosmics ›If any frame has an excessive number of cosmics (very long exposure red/NIR frames for example) consider disabling them ›Apply the previously calculated flat response to each slitlet 26

27. Cube creation – part 1 ›Apply the wavelength and wire solutions to your science observations ›Interpolates each slitlet onto a common wavelength and spatial scale ›Aligns each slitlet using the wire solution ›Produces another MEF file 27

28. Cube creation – part2 ›For convenience, convert the MEF “cube” into a 3D datacube ›NB In later versions of pywifes this function is built in ›With this version: -ipython --pylab -import wifes_utilities -wifes_utilities.wifes_mef_to_cube(infile,outlife) ›Inspect your final data cube using QFitsview or other datacube viewer 28

29. Flux calibration ›To flux calibrate our data, we require observations of spectrophotometric standard stars ›These functions extract star spectra, compare those spectra to the known ‘standard’ spectrum and compute a polynomial transformation ›This polynomial is then applied to the science frames 29

30. Bits and bobs and summary

31. Variance ›Because IFS data reduction is complex, tracking variance is also complex ›The key is to treat your variance image exactly as you treat your flux -Dividing by a flat field? Divide your variance by the square of the same frame -Flux calibrating? Rescale your variance in the same way ›Not quite that simple – many steps add additional variance -Subtracting the bias? Bias frames have noise so add variance -Extracting spectra? Cross-talk and inexact extraction apertures add variance ›Good news is most IFS reduction packages do this for you ›Understanding your variance is critical for taking best advantage of your data! 31

32. Correlation and Covariance ›Not all flux pixels are independent. Not all variance pixels are independent either ›Spatial correlation: -Simplest example is seeing, happens in all IFUs -Crosstalk between spectra located close to one another on the CCD – common in fibre and lenselet IFUs -Resampling onto a common spatial scale ›Spectral correlation: -Resampling onto a common wavelength scale ›Covariance – or correlated noise: -Similar to correlated flux, but more complicated to deal with -Dealing with it properly requires a full covariance matrix – some DR packages will provide this -Often simpler to partially account for covariance by rescaling your variance by an appropriate factor – or when model fitting penalise for fewer independent datapoints 32

33. Atmospheric Refraction ›Light of different wavelengths is refracted by different amounts as it passes through the atmosphere ›This means the position of an object in IFS varies systematically as a function of wavelength ›Sometimes telescopes have an ADC (atmospheric dispersion corrector) built in ›If not, you can manually shift each wavelength to have a common centroid, either by fitting to the object centroid or using a DAR model ›This effect is worse at blue wavelengths, can be safely ignored in the NIR 33

34. Combining data ›Several reasons to combine data cubes: improve S/N, increase field-of- view, attempt to increase spatial resolution ›Can centre based on object centroid or just telescope pointing for faint objects ›Easy mode: use native sampling and combine to the nearest pixel – this is often good enough ›Hard mode: combine to sub-pixel accuracy by resampling data onto a finer grid – have to do this when trying to increase effective resolution via drizzling ›Caution – resampling correlates your data AND your noise. May be more trouble than its worth 34 Sharp et al. (2015)

35. Closing remarks ›IFS is an extremely powerful technique with a broad range of astrophysical applications. It has revolutionised optical astronomy in the last ~15 years ›Different IFUs have different wavelength ranges, fields-of-view, spectral and spatial resolutions, efficiencies, access to AO etc. Pick the best tool to suit your science goals ›However, the instruments and therefore the data are complex, and an understanding of how the data are produced is critical to get the most out of it ›Different IFU designs come with different pros and cons, as well as different issues affect their data products ›Many IFUs have an excellent pipeline – I highly recommend using these! ›If your IFU doesn’t have a pipeline, find a collaborator with experience reducing data from that instrument 35

36. Questions?