Data Inspection Tutorial Hiroshi Nagai

Scope of this tutorial This tutorial demonstrates the step-by-step procedure how to see/check/understand the ALMA data using an archive dataset. How to check the data products and investigate the data quality How to reproduce the calibration/imaging results Some tips which should be checked by users Data: Pipeline-reduced data Level: Beginners - Intermediate

References ALMA Data Products Document https://almascience.nao.ac.jp/documents-and-tools/cycle3/ALMAQA2Products3.0.pdf ALMA Technical handbook https://almascience.nao.ac.jp/documents-and-tools/cycle4/alma-technical-handbook Presentation at Mitaka Town Meeting 2016 https://www.youtube.com/watch?v=1PxshXA7eNo http://alma-intweb.mtk.nao.ac.jp/~nagai/tutorial/BasicsInterferometry.pdf Data Reduction Tips by H. Nagai (mainly for manually- calibdated dataset, only in Japanese) http://alma-intweb.mtk.nao.ac.jp/~nagai/ALMAdata/DataAnalysis.html

Preparation Install CASA 4.7.0-1 Download the presentation file and data package from tutorial webpage Install analysisUtils

What is the ALMA Archive Data? ALMA archives (a) raw data (b) calibration scripts, final image products (FITS image), and supporting materials. Part (a) consists of visibility datasets in ALMA's native format, the ASDM, which are only needed if the PI would like to perform custom data calibration and imaging. Part (b), the “QA2 Products”, consists of the QA2 data reduction scripts, log files, calibration, and flagging tables, and the imaging products on which the quality assurance decision was based. The calibration/imaging is done at ARCs and JAO by either manual or ALMA Pipeline (Quality Assurance) Pipeline calibration + manual imaging : Standard mode observation Manual calibration and imaging: Non-standard mode observation, standard mode observation which needs significant manual intervention (low SNR, too many flagging, temporal pipeline bugs, etc) (What’s standard and non-standard? See PG/TH.) Roughly speaking, PL:Manual = 6:4

Recent update NAOJ provides calibrated Measurement Sets to PIs at Japanese institutes as an optional service. ALMA Imaging Pipeline has been in operation. Roughly 15 MOUSs imaged with PL has been delivered to PIs. Note the size of delivery package can be very large (>100GB). Dear XXXX XXXX, As you received a notification email, the data of the member ObsUnitSet uid://XXXX/XXXX/XXXX of your project 2015.1.0XXXX.S are now available to you for download. EA-ARC has also prepared calibrated Measurement Sets (MSs) for this SB. If you would like to download the calibrated MSs, please visit following ftp site. ftp location: ftp://AF0XXX@alma-dl.mtk.nao.ac.jp/calibrated.tar.gz user name: AF0XXX password: XXXXXXXXXXXX Available until:2017-01-12 02:54:47 JST

QA Philosophy The goal of ALMA Quality Assurance (QA) is to ensure that a reliable final data product is delivered to the PI that has reached the desired control parameters outlined in the science goals, is calibrated to the desired accuracy and contains no significant calibration or imaging artifacts. For Cycle 4, as for all previous Cycles, QA will be done on a “best effort” basis, covering all the major issues affecting data quality. The ALMA QA2 data reduction team performs for each ALMA science data set a detailed analysis to confirm that the observations have achieved the science goals requested by the PI.

QA Philosophy If the requirements are met within the cycle- dependent tolerances (see chapter 11 of the ALMA Technical Handbook), the data are declared “QA2 pass”, packaged in a standardized way, and delivered to the PI. If the requirements are not met but additional observations of the SB are no longer possible within the cycle, the data are delivered as “QA2 semipass”.

Example Data (public data) Project: 2013.1.00063.S (PI: R. Indebetouw) MOUS id: uid://A001/X12e/X280 Scheduling Block: sn1987a_a_07_TE

Data Search Go to Science Portal https://almascience.nao.ac.jp/alma-data/archive Click Archive Query

Archive Query & Request Handler Search by Project code

Project Structure Tree Project code Science Goal OUS Group OUS Member OUS Scheduling block Product (tar package containing image products and supplement materials) ASDMs

Data Package Package name should be in the format of ProjectCode_MOUSuid_001_of_001.tar e.g., 2013.1.00063.S_uid___A001_X12e_X280_001_of_001.tar Untar the package, i.e., tar xvf 2013.1.00063.S_uid___A001_X12e_X280_001_of_001.t ar 2013.1.00063.S/science_goal.uid___A001_X12e_X27e/group. uid___A001_X12e_X27f/member.uid___A001_X12e_X280 Project code MOUS uid

Check README There also should be raw/ directory if you downloaded asdm packages and untar them. [nagai@alma-proc01 member.uid___A001_X12e_X280]$ ls calibrated calibration log product qa README script [nagai@alma-proc01 member.uid___A001_X12e_X280]$ less README NB: calibrated directory can be created by running scriptForPI.py. This was already done for this tutorial.. Use same (or newer) version of CASA and pipeline

For this data, there are two execution blocks. Check Weblog Weblog is a system of webpages containing all the diagnostic plots and other information generated by the pipeline For this data, there are two execution blocks.

QA2 Report for Manual Calibration Instead of weblog, several png images and a text file for each EB, so-called “QA2 report”, is provided under qa/ directory.

QA2 report (png images)

QA2 report (text file)

Let’s check weblog together [nagai@alma-proc01 member.uid___A001_X12e_X280]$ ls calibrated calibration log product qa README script [nagai@alma-proc01 member.uid___A001_X12e_X280]$ firefox qa/pipeline-20151016T113145/html/index.html & Check observation summary Check flagging summaries Check calibrated data If any suspicion, check relevant calibrations Check flux consistency Search calibrator fluxes in ALMA database using aU.getALMAFlux

Observation Summary Click here Click here

Check flagging summaries Go to “By Topic” This page shows how much fraction of the data are flagged by pipeline per antenna.

Check calibrated data Go to “By Task”, click 15. hifa_applycal If any suspicious feature, check relevant calibrations. e.g., strange bandpass shape -> check bandpass/Tsys calibration Time variable phase -> check gain/wvr calibration Click here

Check flux consistency Go to 13.hfa_fluxscale For this data, a quasar J0519-454 was chosen as a flux calibrator. The pipeline obtained the flux mode of this calibrator from the measurement in ALMA database close in time. See 10.hifa_setjy for the flux model. This page shows the flux densities of other calibrators derived from the flux scaling using the model of J0519-454.

Retrieve Calibrator Flux in database CASA <2>: aU.getALMAFlux(sourcename='J0538-4405',date='20150629',frequency='301.861GHz') Using Band 3 measurement: 2.090 +- 0.050 (age=1 days) 103.5 GHz Using Band 7 measurement: 1.030 +- 0.100 (age=2 days) 343.5 GHz exact value: -0.589841, 1-sigma extrema: -0.492419, -0.694681, mean unc=0.101131 Median Monte-Carlo result for 301.861000 = 1.116127 +- 0.226341 (scaled MAD = 0.222051) Result using spectral index of -0.589841 for 301.861000 GHz from 103.490000 GHz = 1.111537 +- 0.226341 Jy Out[2]: {'ageDifference': 1.0, 'fluxDensity': 1.1115372967972896, 'fluxDensityUncertainty': 0.22634072524386856, 'meanAge': 1.5, 'monteCarloFluxDensity': 1.1161274327138342, 'spectralIndex': -0.58984117504330225, 'spectralIndexUncertainty': 0.11320293251116684}

Produce Calibrated MS Download tarred asdms Once untar them, asdms will be placed under raw/ directory Go to script/ directory and run sciprtForPI.py It can take a few - several hours to finish In this tutorial, we already created calibrated MSs and provide them. Please download from the tutorial webpage. Check the calibrated amplitude and phase as a function of time, freq., uv-distance for calibrators (and targets if SN is enough) using plotms For this data set, we see a dip in the spectrum of phase calibrator shown in weblog. Please confirm if you see the same feature in the calibrated data using plotms.

Check bad data [nagai@alma-proc01 calibrated]$ pwd /work/nagai/Tutorial/2013.1.00063.S/science_goal.uid___A001_X12e_X27e/group.uid___A001_X12e_X27f/member.uid___A001_X12e_X280/calibrated [nagai@alma-proc01 calibrated]$ ls casa-20161214-051732.log products uid___A002_Xa48b1f_X1b0.ms.split.cal working ipython-20161214-051744.log rawdata uid___A002_Xaa96da_Xb77.ms.split.cal [nagai@alma-proc01 calibrated]$ casa CASA <9>: plotms(vis='uid___A002_Xa48b1f_X1b0.ms.split.cal',xaxis='chan',yaxis='amp',avgtime='1e9',avgscan=T,avgantenna=T,spw='1',coloraxis='ant1',field='2')

sciprtForPI.py vs. casa_pipescript.py Running scriptForPI.py reproduces the pipeline calibration done at JAO/ARCs. Rerun the pipeline calibration with additional flagging can be performed running casa_pipescript.py Copy casa_pipescript.py from the script directory to the raw directory Copy flux.csv and *flagtemplate.txt from the calibration directory to the raw directory

Introducing Additional Flagging Additional manual flagging can be introduced to any Pipeline reduction by editing the *flagtemplate.txt files under calibration/ directory. For this data, one may want to flag channels affected by the atmospheric absorption. Add following line: mode=manual spw=’19’ channel=’34~38’ reason=‘bad_channel’ Rerun casa_pipescript.py The data of DA56 in spw=17 was flagged by data reducer because of amplitude outlier. # User flagging commands file # # Examples # Note: Do not put spaces inside the reason string ! # mode=manual correlation=YY antenna=DV01;DV08;DA43;DA48&DV23 spw=21:1920~2880 autocorr=False reason='bad_channels' # mode=manual spw='25:0~3;122~127' reason='stage8_2' # mode=manual antenna='DV07' timerange='2013/01/31/08:09:55.248~2013/01/31/08:10:01.296' reason='quack' mode=manual antenna='DA56' spw='17' reason='timegaincal_outlier_amp'

Editing Calibrator Fluxes For some reason, you may want to redo the flux calibration Referenced calibrator flux is old Quasar which is used for the flux calibrator is time variable Check Calibrator database i) Edit flux.csv and rerun casa_pipescript.py or ii) give flux rescaling task in sciptForImagingPrep.py and run it on the calibrated MSs. Note QA2 data reducers normally checked the flux consistency of calibrators and did the flux rescaling if necessary at EA-ARC. However, about 20% of EA data is processed at JAO. Those may not be checked as careful as what we do. The criteria to apply/un-apply flux rescaling could be different among ARCs.

Example of scriptForImagingPrep.py setjy(vis = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal', standard = 'manual', field = 'J1427-4206', fluxdensity = [1.89, 0, 0, 0], spix = -0.43, reffreq = '233.0GHz') os.system('rm -rf ../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf') gaincal(vis = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal', caltable = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf', solint = 'inf', combine = 'scan', refant = 'DV16', gaintype = 'T', calmode = 'a') applycal(vis = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal', field = '0,3', # J1427-4206,NGC5064 gaintable = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf', gainfield = '0', # J1427-4206 calwt = F, flagbackup = F) print "# Concatenating the data." concat(vis = ['../calibrated/uid___A002_Xb020f7_X32f5.ms.split.cal', '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal'], concatvis = '../calibrated/calibrated.ms') Give a “correct” flux density to the phase calibrator Run amplitude gain calibration using the given flux model Apply new gain calibration to the phase calibrator and target

Imaging Products Imaging products (FITS of primary-beam corrected image namely *.pbcor.fits and primary beam coverage namely *.flux.fits) are found under product/ directory Imaging script used for produce these products can be found under script/ directory (scriptForImaging.py). [nagai@alma-proc01 product]$ pwd /work/nagai/Tutorial/2013.1.00063.S/science_goal.uid___A001_X12e_X27e/group.uid___A001_X12e_X27f/member.uid___A001_X12e_X280/product [nagai@alma-proc01 product]$ ls calibrated_final_cont.flux.fits SN1987A_SiO76R2.pbcor.fits calibrated_final_cont.mask uid___A001_X12e_X280.J0538-4405_bp.spw17.I.fits calibrated_final_cont.pbcor.fits uid___A001_X12e_X280.J0538-4405_bp.spw19.I.fits cont.mask uid___A001_X12e_X280.J0538-4405_bp.spw21.I.fits SN1987A_SiO76R1.flux.fits uid___A001_X12e_X280.J0538-4405_bp.spw23.I.fits SN1987A_SiO76R1.mask uid___A001_X12e_X280.J0635-7516_ph.spw17.I.fits SN1987A_SiO76R1.pbcor.fits uid___A001_X12e_X280.J0635-7516_ph.spw19.I.fits SN1987A_SiO76R2.flux.fits uid___A001_X12e_X280.J0635-7516_ph.spw21.I.fits SN1987A_SiO76R2.mask uid___A001_X12e_X280.J0635-7516_ph.spw23.I.fits

Check Image CH=0 CH=2 CH=4 Click here to zoom-up CASA <2>: viewer('calibrated_final_cont.pbcor.fits') CASA <4>: viewer('SN1987A_SiO76R2.pbcor.fits') CH=0 CH=2 CH=4 Will explain how to display - beam size and other parameters - integrated spectrum in a given region - image statistics - source size - Gaussian model fitting

Parameter control in scriptForImaging.py # imaging control # ---------------- # The cleaning below is done interactively, so niter and threshold can # be controlled within clean. weighting = 'briggs' robust=0.5 niter=1000 threshold = '0.0mJy' ############################################# # Imaging the Continuuum # Set the ms and continuum image name. contvis = 'calibrated_final_cont.ms' contimagename = 'calibrated_final_cont' # If necessary, run the following commands to get rid of older clean # data. #clearcal(vis=contvis) #delmod(vis=contvis) for ext in ['.flux','.image','.mask','.model','.pbcor','.psf','.residual','.flux.pbcoverage']: rmtables(contimagename+ext) clean(vis=contvis, imagename=contimagename, field=field, mode='mfs', psfmode='clark', imsize = imsize, cell= cell, weighting = weighting, robust = robust, niter = niter, threshold = threshold, interactive = True, imagermode = imagermode) weighting = 'briggs' robust=0.5 niter=1000 threshold = '0.0mJy' Choice of robust parameter changes the beam size and image quality. Smaller value, higher resolution Larger value, less sidelobe (if the uv-coverage is very bad) But, optimal parameter always depends on the uv- coverage

Trial of different robust parameter execfile(‘scriptForContImaging_rb2.0.py’)

Want to do manual calibration? If you would like to do calibration manually instead of the pipeline, try Eric’s script generator (developed by Eric Villard in JAO). Type es.generateReducScript(‘asdm_name’) on CASA term analysisUtils should be installed. This will return a set of calibration tasks An example script uid___A002_Xa48b1f_X1b0.ms.scriptForCalibration.py is found under calibrated/ directory