2. Beating Noise Observational Techniques ASTR 3010 Lecture 11 Textbook

3. From previous classes We learned that in the astronomical data pre-processing, We learned that in the astronomical data pre-processing, bias subtraction, dark subtraction, flat-fielding, bad pixel correction, cosmic ray hits removal are important. There are several observation techniques that can take care of the above correction nearly automatically. There are several observation techniques that can take care of the above correction nearly automatically.

4. Subtraction of two images taken with a small offset image1 – image2 image1 – image2 bad pixels, bias, and dark are removed! bad pixels, bias, and dark are removed! But, cosmic rays remain. But, cosmic rays remain. Need to apply flat-fielding Need to apply flat-fielding -=

5. ?Dither Observation? Dither Observation Dither Observation Series of images taken with telescope offsets. box5 box4

6. Dithering Observation dead pixels hot pixels cosmic rays Observation No 1 Detector Array

7. Dithering Observation dead pixels hot pixels cosmic rays Observation No 2

8. Dithering Observation dead pixels hot pixels cosmic rays Observation No 3

9. Dithering Observation dead pixels hot pixels cosmic rays Observation No 4

10. Dithering Observation dead pixels hot pixels cosmic rays Observation No 4

11. Dithering Observation Five observations Five observations obs1 … obs5 flux1 flux2 flux3 flux4 flux5 True Flux

12. Dithering Observation Five observations Five observations This will take care of bias, dark, cosmic ray hits, bad pixels! obs1 … obs5 flux4 < flux5 < flux2 < flux1 < flux3 Ordered list median value

13. Dozens of Dithered images with small offsets SuperResolution SuperResolution

14. With the P&A telescope Not Yet!

15. P&A Dithering by Hand

16. Real example : Keck NIRC2 AO observation of HIP88945

17. Shift and Add look at those hot pixels!!

18. # demonstrating the Shift and Add concept. # # import libraries from ds9 import * from numpy import * import pyfits im1=pyfits.getdata('n0002.fits',ignore_missing_end=True)im2=pyfits.getdata('n0003.fits',ignore_missing_end=True)im3=pyfits.getdata('n0004.fits',ignore_missing_end=True)im4=pyfits.getdata('n0005.fits',ignore_missing_end=True)im5=pyfits.getdata('n0006.fits',ignore_missing_end=True) # create a big enough array to cover any shifted images... BIG = zeros((1201,1201)) # Then, we will find the positions of the star in each image. # Shift the image in the BIG array such that the star appears # at the center of the array (601,601) posx_im1=276posy_im1=279 BIG[601-posx_im1:601-posx_im1+512,601-posy_im1:601-posy_im1+512] += im1 posx_im1=397posy_im1=162 BIG[601-posx_im1:601-posx_im1+512,601-posy_im1:601-posy_im1+512] += im2 posx_im1=155posy_im1=403 BIG[601-posx_im1:601-posx_im1+512,601-posy_im1:601-posy_im1+512] += im3 posx_im1=397posy_im1=401 BIG[601-posx_im1:601-posx_im1+512,601-posy_im1:601-posy_im1+512] += im4 posx_im1=154posy_im1=161 BIG[601-posx_im1:601-posx_im1+512,601-posy_im1:601-posy_im1+512] += im5 d=ds9() d.set_np2arr( transpose(BIG) )

19. Shift and Median is better!! Python HW2: Create a Python script doing this!

20. Median sky subtraction When your object is extended… or you need to make a pretty picture… When your object is extended… or you need to make a pretty picture… in near-IR: useful for 10-20 min depends on the quality of the night. in near-IR: useful for 10-20 min depends on the quality of the night.

21. In Astronomical observations… Three major noise sources… Three major noise sources… o Poisson noise (aka “shot noise”) o Sky noise o Electronic noise Sky noise limited observation Sky noise limited observation Dark current limited observation Dark current limited observation Readout noise limited observation(?) Readout noise limited observation(?)

22. Typical stellar sources in the Solar Neighborhood Noise versus signal at various wavelengths Opticalnear IRmid IR 10 1 10 9 Sky noise Detector noise Signal Strength Extremely bright objects High red shift galaxies 10 5

23. Coadds Because of the increased sky background, frames will get saturated in short exposures. Because of the increased sky background, frames will get saturated in short exposures.  variable sky is the limiting factor  sky limited observations. Take as many unsaturated frames as necessary and combined them later. Take as many unsaturated frames as necessary and combined them later.  too many files for a single object (several hundreds)  on-chip combine of multiple exposures = “coadd”

24. Near-IR observation Tips Need to sample the sky Need to sample the sky o through dithering o or from a dedicated sky observation in the nearby field  needs to be dithered also. why? Choose a right number of coadds. Choose a right number of coadds. o Choose the maximum frame exposure time which will stay in the linear regime. o Then, from the required total exposure time, calculate the necessary number of coadds. Sky level is varying faster at longer wavelengths! Sky level is varying faster at longer wavelengths! o at K-band (2.2 μm) : up to 20 minutes o at L-band (3.6 μm) : max is about a minute o at mid-IR (10 μm) : fraction of a second  different strategy Dither Dither Repeat the dither sequence as many times as necessary. Repeat the dither sequence as many times as necessary.

25. Observation Planning : near-IR You will be observing at the K-band (2.2 micron), and you expect that your source would have a count rate of about 1,000 counts per second where this signal will be spread over ~10 pixels due to the PSF structure. Some other relevant data are You will be observing at the K-band (2.2 micron), and you expect that your source would have a count rate of about 1,000 counts per second where this signal will be spread over ~10 pixels due to the PSF structure. Some other relevant data are o Detector linear regime: count < 10,000 counts o Sky brightness at K = 100 counts per second per pixel o Readout noise = 10 counts per readout o Dark current = 25 counts per second per pixel What would be the best observing strategy for this object if you want your final S/N~100?

26. Observation Planning : near-IR Source ~ 2,000 count per second over 4 pixels. Goal S/N >= 100 Source ~ 2,000 count per second over 4 pixels. Goal S/N >= 100 o Detector linear regime: count < 8,000 counts o Sky brightness at K = 1,000 counts per second per pixel o Readout noise = 10 counts per readout o Dark current = 100 counts per second per pixel Per pixel per second signal = 500 source count + 1,000 sky count + 100 dark current = 1,600 counts/pxl/sec  In 5 sec exposure, the source will get into a non-linear regime  max exposure time should be less than 5sec or we should “coadd” Per pixel per second signal = 500 source count + 1,000 sky count + 100 dark current = 1,600 counts/pxl/sec  In 5 sec exposure, the source will get into a non-linear regime  max exposure time should be less than 5sec or we should “coadd” In a single 5 second exposure : S/N = 500 / sqrt(1600)~12  if we coadd 16 frames, then a single exposure will be 40 second long and S/N=12*sqrt(16)=48 In a single 5 second exposure : S/N = 500 / sqrt(1600)~12  if we coadd 16 frames, then a single exposure will be 40 second long and S/N=12*sqrt(16)=48 For a box5 dither observation pattern, we will get five images  total readout noise = 10 * sqrt(5) = 22.4 counts. This is negligible to shot noises from source+sky+detector. For a box5 dither observation pattern, we will get five images  total readout noise = 10 * sqrt(5) = 22.4 counts. This is negligible to shot noises from source+sky+detector. If we take one image (40 sec long with 8 coadds) using a box5 pattern  final S/N=48 * sqrt(5) = 107 If we take one image (40 sec long with 8 coadds) using a box5 pattern  final S/N=48 * sqrt(5) = 107 The total duration of the observation is 5 x 40sec + total overhead (~1 min) = 4 min. At K-band, the sky will be invariable for up to ~20 min. So, this will be OK. The total duration of the observation is 5 x 40sec + total overhead (~1 min) = 4 min. At K-band, the sky will be invariable for up to ~20 min. So, this will be OK.

27. Nod & Shuffle "Nod and Shuffle" is a technique used to obtain very faint spectra "Nod and Shuffle" is a technique used to obtain very faint spectra the natural glow of the nighttime sky would overwhelm the extremely faint objects (e.g., far, far away galaxies) the natural glow of the nighttime sky would overwhelm the extremely faint objects (e.g., far, far away galaxies) N&S allows astronomers to effectively subtract away the bright spectral emission lines and fainter continuum of our atmosphere's nighttime glow while retaining the faint spectra of dim, red galaxies. This also significantly reduces readout noise. N&S allows astronomers to effectively subtract away the bright spectral emission lines and fainter continuum of our atmosphere's nighttime glow while retaining the faint spectra of dim, red galaxies. This also significantly reduces readout noise. 1.obtaining a spectrum of an object 2.moving the electrical charge built up by the image of the spectrum on the CCD to another location "buffer storage" on the CCD (Shuffling) 3.shifting the position of the telescope slightly (Nodding) so that the spectrum of the target object shifts to a different part of the CCD 4.obtaining another spectrum in the nodded position 5.shuffling the charges moved in step 1 back to the original position (while moving the charges from step 4 to the buffer storage) 6.nodding the telescope back to the position in step 1 7.repeating until enough light is collected.

28. Sky cancellation: ‘nod and shuffle’ Storage of ‘sky’ image next to object image via ‘charge shuffling’ Zero extra noise introduced, rapid switching (60s) A B ABAB Typically A=60s/15 cy: 1800s exposure  10  subtraction

29. Another example

30. Weird image artifacts Latent images : badly saturated pixels do not come back to normal right away!

31. Weird image artifacts ghost images : multiple reflections within an optical element (e.g., filter, beam splitter, entrance pupil) seen near a very bright star

32. Handling various problems at IR Non-uniform QEFlat-field High background level Point sources Extended sources Dithering and coadds Small dither offset Off-chip dither Non-linearity of detectorsStay within linear range Memory effects (latent)Do not saturate Bad pixelsMask or median out from dithers Cosmic raysShift and add Dark currentSubtracts out with sky Hot rows, hot pixels, amplifier glow Subtracts out well

33. In summary… Important Concepts Various noise sources at different wavelengths and their relation to the observing techniques. Median combining of images Important Terms Dither observation Shift and add Shift and median Coadd Nod-and-shuffle Chapter/sections covered in this lecture : 9