PESSTO Data Reduction PESSTO meeting 6/1/2018

PESSTO pipeline and Quality Control Team S. Valenti B. Schmidt C. Inserra S. Mattila M. Fraser F. Patat M. Sullivan G. Pignata E. Cappellaro E. Kankare M. Ergon ….. Provide Data reduction Pipeline (documentation, pipeline, testing) The Server of data reduction Provide support to the Data reduction Team Check data quality of the reduced data PESSTO meeting 6/1/2018

PESSTO meeting 6/1/2018

NTT large Program Data reduction QUBA PIPELINE: pre-reduction images EFOSC2, 1.82 ekar, NOT, TNG pre-reduction spectra EFOSC2, 1.82 ekar, NOT, TNG,ISIS reduction spectra EFOSC2, 1.82 ekar, NOT, TNG,ISIS Photometry (a python version of the snoopy-iraf package) NTT (SOFI and EFOSC2), 1.82 ekar, NOT, TNG (NICS and DOLORES) Liverpool Telescope, Calar Alto, FORS1, FORS2, …….etc. Python with Pyraf, Pylab, Pyfits …… Several (but not all) people in the collaboration are using it Try to install the pipeline on the pc of each reducer interested in using the pipeline New version released by me by email and a link where to download the pipeline PESSTO meeting 6/1/2018

NTT large Program Data reduction RUN date Observer EFOSC2 SOFI 19-21/10/09 C. Inserra photometry(prered) NOT REDUCED 21-23/11/09 M. Ergon reduced (QUBA) NOT REDUCED 23-25/12/09 E. Kankare NOT REDUCED NOT REDUCED 22-24/01/10 G. Leloudas reduced NOT REDUCED 17-20/02/10 I. Agnoletto NOT REDUCED NOT REDUCED 04-07/03/10 Max/Salgado reduced (QUBA) NOT REDUCED 16-19/04/10 M. Turatto NOT REDUCED (some reduction at the telescope ?) 27-28/05/10 M. Miluzio no observations no observations 17-19/06/10 S. Valenti reduced (QUBA) no observations 14-15/09/10 S. Benetti reduced (gasgano) reduced 3-5 /10/10 R. Kotak reduced QUBA no observations 27-29/10/10 M. Ergon NOT REDUCED NOT REDUCED 30-01/12/10 M. Miluzio reduced (QUBA) NOT REDUCED 23-25/01/11 C. Inserra photometry(prered) NOT REDUCED 10-12/02/11 F. Bufano NOT REDUCED NOT REDUCED 24-26/03/11 E. Kankare photometry(prered) NOT REDUCED 9-11/04/11 L.Tomasella photometry(prered) NOT REDUCED 07-09/05/11 N. De la Rosa photometry(prered) reduced(photometry) 23-26/06/11 S. Valenti reduced (QUBA) reduced(photometry) PIs prefer to reduce the data of their objects more data have been reduced…. (single objects, not full nights) Waste of time and energy PESSTO meeting 6/1/2018

NTT large Program PESSTO Not all the data, (as I would like) are reduced and stored in the archive I’m getting poor feedback on the bugs of the pipeline For most of the reported bugs, my answer is: I fixed that in the last version, did you install it ? …………no I still have the old version. New reducer are open to use the pipeline, but old reducer ….are not …………. Pipeline not user-friendly ? Or not better than the manual reduction ? PESSTO meeting 6/1/2018

PESSTO pipeline PYTHON module NTT_pipeline/ Add this directory to your python site-package/ export PYTHONPATH=$PYTHONPATH:path_of_site-package/ Simple python module IRAF, PYRAF, numpy ... (pywcs) Works from terminal no graphic interphase PESSTO meeting 6/1/2018

But missing 2D-wavelengh calibrated image PESSTO meeting 6/1/2018

PESSTO pipeline 7 python script in ~/site-pakage/NTT_pipeline/bin/ PESSTOFASTSPEC.py etabase_file.py PESSTOstart.py PESSTOSOFIPHOT.py SOFI images pre-reduction PESTOEFOSCPHOT.py EFOSC2 images pre-reduction PESTOEFOSCPRESPEC.py EFOSC2 spectra pre-reduction PESSTOEFOSCSPECRED.py EFOSC2 spectra reduction PESSTOSOFISPEC.py to be done (SOFI spectra) PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOFASTSPEC.py @list_raw_spectra options: -i = interactive mode -d = interactive dispersion line (for nebular spectra) -t = trace with a bright object (for faint objects) -s = use specific sensitivity function -a = use specific arc_ex file (already ext. and wave. calibrated) Trim overscan, extraction (interactive with option –i ) Wavelength calibration (re-identification using data from archive) Check Wavelength calibration (with skylines) Flux calibration (using a median sensitivity curve from archive) PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOFASTSPEC.py @list_raw_spectra Gr11+Free slit1.0 slit1.5 Gr13+Free slit1.0 slit1.5 Gr16+OG530 slit1.0 slit1.5 PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOFASTSPEC.py @list_raw_spectra Check Wavelength calibration (with skylines) Rigid shift of sky lines to maximize the product of the two spectra Please don’t use the fast reduction as a black box PESSTO meeting 6/1/2018

PESSTO pipeline ~bin/etabase_file.py @list_reduced_spectra Convert fits in asci Produce the asci-file with all the information needed for ETABASE Obj-name, Ascii-filename, Obs-date [YYYY-MM-DD], Obs-UT [HH:MM:SS], Inst-Id, Observer/s, Exptime, Slit, Reducer, Reduction-date, Spec-Remarks, Publish, Contrib, Airmass, Dichroic, Grism, Grating, Obj-RA, Obj-DEC, IAU-name, Obj-type-Id, Redshift, Host, FITS-filename PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOstart.py @listafiles Split files in: EFOSC-SPECTRA EFOSC-PHOTOMETRY SOFI-SPECTRA SOFI-PHOTOMETRY PESSTO meeting 6/1/2018

PESSTO pipeline Combine the single sky-subtracted images ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files option: -f = no flat field correction -l = no illumination correction -x = no crosstalk correction -F = chose flat interactively -L = chose illumination correction interactively -i = interactive mode (not yet useful) -c = chose combine param. for combining images [median] -R = chose reject param for combining images [minmax] -s = scale zeropoint before combining images Combine the single sky-subtracted images No Darks correction Flat field correction SpecFlat_20110516_H.fits (quite stable) Illumination correction Illum_20110516_H.fits (quite stable) Crosstalk correction iraf script provided in the ESO website (included in the pipeline) PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files Special Flat field To correct for the shading pattern. The effect for most observations is small, and manifests itself as a discontinuity of a few percent in the ZP across the center of the array Illumination correction Illumination of the screen can never represent accurately the illumination of the sky. The illumination correction frame refers only to the flat field used to create it. Crosstalk correction A bright source imaged on the array produces a “ghost” that affects all the lines where the source is and all the corresponding lines in the other half of the detector. PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files Small dither (<200 arcsec) N images >= 4 Sky with all (except A) - = A Mask_A Sky with all (except A) - = A Mask_B Mask_C Mask_D PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files Small dither (<200 arcsec) N images >= 4 Sky with all (except A) - = A Mask_A Sky with all (except A) - = A Mask_B Mask_C Mask_D PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files ON – OFF (extended galaxies) dither >=200 arcsec N(off) >=3 N (on) >=3 - = D F D Sky with all OFF D E F E Sky with all OFF - = A D E F Mask_D Mask_D PESSTO meeting 6/1/2018 Mask_F

PESSTO pipeline ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files raw Using dither information in the header … we need interactivity ? ……. ~/bin/PESSTOSOFIPHOT.py @lista_SOFI_raw_phot_files raw Flat field, Illumination, Cross Talks sky-subtracted Providing: single sky-subtracted images PESSTO meeting 6/1/2018

PESSTO pipeline 7 python script in ~/site-pakage/NTT_pipeline/bin/ PESSTOFASTSPEC.py etabase_file.py PESSTOstart.py PESSTOSOFIPHOT.py SOFI images pre-reduction PESTOEFOSCPHOT.py EFOSC2 images pre-reduction PESTOEFOSCPRESPEC.py EFOSC2 spectra pre-reduction PESSTOEFOSCSPECRED.py EFOSC2 spectra reduction PESSTOSOFISPEC.py to be done (SOFI spectra) PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOEFOSCPHOT.py @lista_EFOSC_raw_phot_files Option: -f = no flat field correction -b = no bias correction -F = chose flat interactively -B = chose bias interactively -c = chose combine param. for flatcombine -R = chose reject param for flatcombine -i = interactive mode (making the flat field, fringing correction) -m mask.fits (mask for fringing correction) PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOEFOSCPHOT.py @lista_EFOSC_raw_phot_files PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESTOEFOSCPRESPEC.py @lista_EFOSC_raw_spec_files Option: -f = no flat field correction -b = no bias correction -i = interactive mode (qubaprespec_v3.py) Triming Overscan bias Flat field (normalized) For each spectrum associate an arc (same pre-reduction) PESSTO meeting 6/1/2018

PESSTO pipeline ~/bin/PESSTOEFOSCSPECRED.py @lista_preredu_spec_files Option: -i = interactive mode -d = interactive dispersion line -t = trace with a bright object -s = use specific sensitivity function (choose sens interactively ) -a = use specific arc file (choose the arc interactively ) Standard Objects Sensitivity curve Flux calibrated spectrum Atmosferic features correction (atm. features in the standard, scaling an atmospheric model to the SN atm. features) PESSTO meeting 6/1/2018

PESSTO pipeline 2D-wavelength calibrated spectra ~/bin/PESSTOEFOSCSPECRED.py @lista_preredu_spec_files Check Wavelength calibration for the standards(with atmospheric features) 2D-wavelength calibrated spectra PESSTO meeting 6/1/2018

PESSTO pipeline 7 python script in ~/site-pakage/NTT_pipeline/bin/ PESSTOFASTSPEC.py etabase_file.py PESSTOstart.py PESSTOSOFIPHOT.py SOFI images pre-reduction PESTOEFOSCPHOT.py EFOSC2 images pre-reduction PESTOEFOSCPRESPEC.py EFOSC2 spectra pre-reduction PESSTOEFOSCSPECRED.py EFOSC2 spectra reduction PESSTOSOFISPEC.py to be done (SOFI spectra) 2D-w-calib PESSTO meeting 6/1/2018

PESSTO SOFI SPECTRA reduction (SS) Spectra Flux calibration (FC) Automatic flat SOFI (FS) Automatic fringing (AF) TESTING (T) Documentation (D) Server for data reductions (Serv) GUI interface (GUI) 2D-w-calibrated spectra (2D) Astrometry ? (A) Modify the code (M) Configuration Management (CM) Quality Control (QC) S. Valenti (M,T,Serv,….) B. Schmidt (FC) C. Inserra S. Mattila M. Fraser F. Patat M. Sullivan G. Pignata E. Cappellaro E. Kankare M. Ergon (CM) PESSTO meeting 6/1/2018

PESSTO EFOSC2 Gr11+Free slit1.0 slit1.5 Gr13+Free slit1.0 slit1.5 Gr16+OG530 slit1.0 slit1.5 SOFI < 200 arcsec 4 images < 200 arcsec 6 images (Different Objects name) PESSTO meeting 6/1/2018

PESSTO PESSTO meeting 6/1/2018

PESSTO PESSTO meeting 6/1/2018

PESSTO PESSTO meeting 6/1/2018