1. PESSTO
Data Reduction
PESSTO meeting
6/1/2018

2. PESSTO pipeline and Quality Control Team
S. Valenti B. Schmidt C. Inserra S. Mattila M. Fraser F. Patat M. Sullivan G. Pignata E. Cappellaro E. Kankare M. Ergon …..
Provide Data reduction Pipeline
(documentation, pipeline, testing)
The Server of data reduction
Provide support to the Data reduction Team
Check data quality of the reduced data
PESSTO meeting
6/1/2018

3. PESSTO meeting
6/1/2018

4. NTT large Program Data reduction QUBA PIPELINE:
pre-reduction images EFOSC2, 1.82 ekar, NOT, TNG
pre-reduction spectra EFOSC2, 1.82 ekar, NOT, TNG,ISIS
reduction spectra EFOSC2, 1.82 ekar, NOT, TNG,ISIS
Photometry (a python version of the snoopy-iraf package)
NTT (SOFI and EFOSC2), 1.82 ekar, NOT, TNG (NICS and DOLORES)
Liverpool Telescope, Calar Alto, FORS1, FORS2, …….etc.
Python with Pyraf, Pylab, Pyfits ……
Several (but not all) people in the collaboration are using it
Try to install the pipeline on the pc of each reducer interested in using the pipeline
New version released by me by and a link where to download the pipeline
PESSTO meeting
6/1/2018

5. NTT large Program Data reduction
RUN date Observer EFOSC2 SOFI
19-21/10/09 C. Inserra photometry(prered) NOT REDUCED
21-23/11/09 M. Ergon reduced (QUBA) NOT REDUCED
23-25/12/09 E. Kankare NOT REDUCED NOT REDUCED
22-24/01/10 G. Leloudas reduced NOT REDUCED
17-20/02/10 I. Agnoletto NOT REDUCED NOT REDUCED
04-07/03/10 Max/Salgado reduced (QUBA) NOT REDUCED
16-19/04/10 M. Turatto NOT REDUCED (some reduction at the telescope ?)
27-28/05/10 M. Miluzio no observations no observations
17-19/06/10 S. Valenti reduced (QUBA) no observations
14-15/09/10 S. Benetti reduced (gasgano) reduced
3-5 /10/10 R. Kotak reduced QUBA no observations
27-29/10/10 M. Ergon NOT REDUCED NOT REDUCED
30-01/12/10 M. Miluzio reduced (QUBA) NOT REDUCED
23-25/01/11 C. Inserra photometry(prered) NOT REDUCED
10-12/02/11 F. Bufano NOT REDUCED NOT REDUCED
24-26/03/11 E. Kankare photometry(prered) NOT REDUCED
9-11/04/ L.Tomasella photometry(prered) NOT REDUCED
07-09/05/ N. De la Rosa photometry(prered) reduced(photometry)
23-26/06/ S. Valenti reduced (QUBA) reduced(photometry)
PIs prefer to reduce the data of their objects more data have been reduced…. (single objects, not full nights) Waste of time and energy
PESSTO meeting
6/1/2018

6. NTT large Program PESSTO
Not all the data, (as I would like) are reduced and stored in the archive
I’m getting poor feedback on the bugs of the pipeline
For most of the reported bugs, my answer is: I fixed that in the last version, did you install it ? …………no I still have the old version.
New reducer are open to use the pipeline, but old reducer ….are not …………. Pipeline not user-friendly ? Or not better than the manual reduction ?
PESSTO meeting
6/1/2018

7. PESSTO pipeline PYTHON module NTT_pipeline/
Add this directory to your python site-package/
export PYTHONPATH=$PYTHONPATH:path_of_site-package/
Simple python module IRAF, PYRAF, numpy ... (pywcs)
Works from terminal no graphic interphase
PESSTO meeting
6/1/2018

8. But missing 2D-wavelengh calibrated image
PESSTO meeting
6/1/2018

9. PESSTO pipeline 7 python script in ~/site-pakage/NTT_pipeline/bin/
PESSTOFASTSPEC.py
etabase_file.py
PESSTOstart.py
PESSTOSOFIPHOT.py SOFI images pre-reduction
PESTOEFOSCPHOT.py EFOSC2 images pre-reduction
PESTOEFOSCPRESPEC.py EFOSC2 spectra pre-reduction
PESSTOEFOSCSPECRED.py EFOSC2 spectra reduction
PESSTOSOFISPEC.py to be done (SOFI spectra)
PESSTO meeting
6/1/2018

10. PESSTO pipeline ~/bin/PESSTOFASTSPEC.py @list_raw_spectra options:
-i = interactive mode
-d = interactive dispersion line (for nebular spectra)
-t = trace with a bright object (for faint objects)
-s = use specific sensitivity function
-a = use specific arc_ex file (already ext. and wave. calibrated)
Trim
overscan,
extraction (interactive with option –i )
Wavelength calibration (re-identification using data from archive)
Check Wavelength calibration (with skylines)
Flux calibration (using a median sensitivity curve from archive)
PESSTO meeting
6/1/2018

11. PESSTO pipeline ~/bin/PESSTOFASTSPEC.py @list_raw_spectra
Gr11+Free slit slit1.5
Gr13+Free slit slit1.5
Gr16+OG530 slit slit1.5
PESSTO meeting
6/1/2018

12. PESSTO pipeline ~/bin/PESSTOFASTSPEC.py @list_raw_spectra
Check Wavelength calibration (with skylines)
Rigid shift of sky lines to maximize the product of the two spectra
Please don’t use the fast reduction as a black box
PESSTO meeting
6/1/2018

13. PESSTO pipeline
Convert fits in asci Produce the asci-file with all the information needed for ETABASE Obj-name, Ascii-filename, Obs-date [YYYY-MM-DD], Obs-UT [HH:MM:SS], Inst-Id, Observer/s, Exptime, Slit, Reducer, Reduction-date, Spec-Remarks, Publish, Contrib, Airmass, Dichroic, Grism, Grating, Obj-RA, Obj-DEC, IAU-name, Obj-type-Id, Redshift, Host, FITS-filename
PESSTO meeting
6/1/2018

14. PESSTO pipeline ~/bin/PESSTOstart.py @listafiles Split files in:
EFOSC-SPECTRA
EFOSC-PHOTOMETRY
SOFI-SPECTRA
SOFI-PHOTOMETRY
PESSTO meeting
6/1/2018

15. PESSTO pipeline Combine the single sky-subtracted images
option:
-f = no flat field correction
-l = no illumination correction
-x = no crosstalk correction
-F = chose flat interactively
-L = chose illumination correction interactively
-i = interactive mode (not yet useful)
-c = chose combine param. for combining images [median]
-R = chose reject param for combining images [minmax]
-s = scale zeropoint before combining images
Combine the single sky-subtracted images
No Darks correction
Flat field correction SpecFlat_ _H.fits (quite stable)
Illumination correction Illum_ _H.fits (quite stable)
Crosstalk correction iraf script provided in the ESO website (included in the pipeline)
PESSTO meeting
6/1/2018

16. PESSTO pipeline ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files
Special Flat field To correct for the shading pattern. The effect for most observations is small, and manifests itself as a discontinuity of a few percent in the ZP across the center of the array
Illumination correction Illumination of the screen can never represent accurately the illumination of the sky. The illumination correction frame refers only to the flat field used to create it.
Crosstalk correction A bright source imaged on the array produces a “ghost” that affects all the lines where the source is and all the corresponding lines in the other half of the detector.
PESSTO meeting
6/1/2018

17. PESSTO pipeline
Small dither (<200 arcsec) N images >= 4
Sky with all (except A) - = A
Mask_A
Sky with all (except A) - = A
Mask_B
Mask_C
Mask_D
PESSTO meeting
6/1/2018

18. PESSTO pipeline
Small dither (<200 arcsec) N images >= 4
Sky with all (except A) - = A
Mask_A
Sky with all (except A) - = A
Mask_B
Mask_C
Mask_D
PESSTO meeting
6/1/2018

19. PESSTO pipeline
ON – OFF (extended galaxies) dither >=200 arcsec N(off) >=3 N (on) >=3 - = D F D
Sky with all OFF D E F E
Sky with all OFF - = A D E F
Mask_D
Mask_D
PESSTO meeting
6/1/2018
Mask_F

20. PESSTO pipeline ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files raw
Using dither information in the header
… we need interactivity ? …….
raw
Flat field, Illumination, Cross Talks
sky-subtracted
Providing: single sky-subtracted images
PESSTO meeting
6/1/2018

21. PESSTO pipeline 7 python script in ~/site-pakage/NTT_pipeline/bin/
PESSTOFASTSPEC.py
etabase_file.py
PESSTOstart.py
PESSTOSOFIPHOT.py SOFI images pre-reduction
PESTOEFOSCPHOT.py EFOSC2 images pre-reduction
PESTOEFOSCPRESPEC.py EFOSC2 spectra pre-reduction
PESSTOEFOSCSPECRED.py EFOSC2 spectra reduction
PESSTOSOFISPEC.py to be done (SOFI spectra)
PESSTO meeting
6/1/2018

22. PESSTO pipeline ~/bin/PESSTOEFOSCPHOT.py @lista_EFOSC_raw_phot_files
Option:
-f = no flat field correction
-b = no bias correction
-F = chose flat interactively
-B = chose bias interactively
-c = chose combine param. for flatcombine
-R = chose reject param for flatcombine
-i = interactive mode (making the flat field, fringing correction)
-m mask.fits (mask for fringing correction)
PESSTO meeting
6/1/2018

23. PESSTO pipeline ~/bin/PESSTOEFOSCPHOT.py @lista_EFOSC_raw_phot_files
PESSTO meeting
6/1/2018

24. PESSTO pipeline ~/bin/PESTOEFOSCPRESPEC.py @lista_EFOSC_raw_spec_files
Option:
-f = no flat field correction
-b = no bias correction
-i = interactive mode
(qubaprespec_v3.py)
Triming
Overscan
bias
Flat field (normalized)
For each spectrum associate an arc (same pre-reduction)
PESSTO meeting
6/1/2018

25. PESSTO pipeline ~/bin/PESSTOEFOSCSPECRED.py @lista_preredu_spec_files
Option:
-i = interactive mode
-d = interactive dispersion line
-t = trace with a bright object
-s = use specific sensitivity function (choose sens interactively )
-a = use specific arc file (choose the arc interactively )
Standard
Objects
Sensitivity curve
Flux calibrated spectrum
Atmosferic features correction
(atm. features in the standard,
scaling an atmospheric model to the SN atm. features)
PESSTO meeting
6/1/2018

26. PESSTO pipeline 2D-wavelength calibrated spectra
Check Wavelength calibration for the standards(with atmospheric features)
2D-wavelength calibrated spectra
PESSTO meeting
6/1/2018

27. PESSTO pipeline 7 python script in ~/site-pakage/NTT_pipeline/bin/
PESSTOFASTSPEC.py
etabase_file.py
PESSTOstart.py
PESSTOSOFIPHOT.py SOFI images pre-reduction
PESTOEFOSCPHOT.py EFOSC2 images pre-reduction
PESTOEFOSCPRESPEC.py EFOSC2 spectra pre-reduction
PESSTOEFOSCSPECRED.py EFOSC2 spectra reduction
PESSTOSOFISPEC.py to be done (SOFI spectra)
2D-w-calib
PESSTO meeting
6/1/2018

28. PESSTO SOFI SPECTRA reduction (SS) Spectra Flux calibration (FC)
Automatic flat SOFI (FS)
Automatic fringing (AF)
TESTING (T)
Documentation (D)
Server for data reductions (Serv)
GUI interface (GUI)
2D-w-calibrated spectra (2D)
Astrometry ? (A)
Modify the code (M)
Configuration Management (CM)
Quality Control (QC)
S. Valenti (M,T,Serv,….) B. Schmidt (FC)
C. Inserra
S. Mattila
M. Fraser
F. Patat
M. Sullivan
G. Pignata
E. Cappellaro
E. Kankare
M. Ergon (CM)
PESSTO meeting
6/1/2018

29. PESSTO EFOSC2 Gr11+Free slit1.0 slit1.5 Gr13+Free slit1.0 slit1.5
Gr16+OG530 slit slit1.5
SOFI
< 200 arcsec images
< 200 arcsec images
(Different Objects name)
PESSTO meeting
6/1/2018

30. PESSTO
PESSTO meeting
6/1/2018

31. PESSTO
PESSTO meeting
6/1/2018

32. PESSTO
PESSTO meeting
6/1/2018