1. Data Inspection Tutorial
Hiroshi Nagai

2. Scope of this tutorial
This tutorial demonstrates the step-by-step procedure how to see/check/understand the ALMA data using an archive dataset.
How to check the data products and investigate the data quality
How to reproduce the calibration/imaging results
Some tips which should be checked by users
Data: Pipeline-reduced data
Level: Beginners - Intermediate

3. References
ALMA Data Products Document
ALMA Technical handbook
Presentation at Mitaka Town Meeting
Data Reduction Tips by H. Nagai (mainly for manually- calibdated dataset, only in Japanese)

4. Preparation Install CASA 4.7.0-1
Download the presentation file and data package from tutorial webpage
Install analysisUtils

5. What is the ALMA Archive Data?
ALMA archives
(a) raw data
(b) calibration scripts, final image products (FITS image), and supporting materials.
Part (a) consists of visibility datasets in ALMA's native format, the ASDM, which are only needed if the PI would like to perform custom data calibration and imaging. Part (b), the “QA2 Products”, consists of the QA2 data reduction scripts, log files, calibration, and flagging tables, and the imaging products on which the quality assurance decision was based.
The calibration/imaging is done at ARCs and JAO by either manual or ALMA Pipeline (Quality Assurance)
Pipeline calibration + manual imaging : Standard mode observation
Manual calibration and imaging: Non-standard mode observation, standard mode observation which needs significant manual intervention (low SNR, too many flagging, temporal pipeline bugs, etc) (What’s standard and non-standard? See PG/TH.)
Roughly speaking, PL:Manual = 6:4

6. Recent update
NAOJ provides calibrated Measurement Sets to PIs at Japanese institutes as an optional service.
ALMA Imaging Pipeline has been in operation.
Roughly 15 MOUSs imaged with PL has been delivered to PIs.
Note the size of delivery package can be very large (>100GB).
Dear XXXX XXXX,
As you received a notification , the data of the member ObsUnitSet uid://XXXX/XXXX/XXXX
of your project XXXX.S are now available to you for download.
EA-ARC has also prepared calibrated Measurement Sets (MSs) for this SB.
If you would like to download the calibrated MSs, please visit following ftp site.
ftp location:
user name: AF0XXX
password: XXXXXXXXXXXX
Available until: :54:47 JST

7. QA Philosophy
The goal of ALMA Quality Assurance (QA) is to ensure that a reliable final data product is delivered to the PI that has reached the desired control parameters outlined in the science goals, is calibrated to the desired accuracy and contains no significant calibration or imaging artifacts. For Cycle 4, as for all previous Cycles, QA will be done on a “best effort” basis, covering all the major issues affecting data quality.
The ALMA QA2 data reduction team performs for each ALMA science data set a detailed analysis to confirm that the observations have achieved the science goals requested by the PI.

8. QA Philosophy
If the requirements are met within the cycle- dependent tolerances (see chapter 11 of the ALMA Technical Handbook), the data are declared “QA2 pass”, packaged in a standardized way, and delivered to the PI.
If the requirements are not met but additional observations of the SB are no longer possible within the cycle, the data are delivered as “QA2 semipass”.

9. Example Data (public data)
Project: S (PI: R. Indebetouw)
MOUS id: uid://A001/X12e/X280
Scheduling Block: sn1987a_a_07_TE

10. Data Search
Go to Science Portal
Click Archive Query

11. Archive Query & Request Handler
Search by Project code

12. Project Structure Tree
Project code
Science Goal OUS
Group OUS
Member OUS
Scheduling block
Product (tar package containing image products and supplement materials)
ASDMs

13. Data Package
Package name should be in the format of ProjectCode_MOUSuid_001_of_001.tar e.g., S_uid___A001_X12e_X280_001_of_001.tar
Untar the package, i.e., tar xvf S_uid___A001_X12e_X280_001_of_001.t ar
S/science_goal.uid___A001_X12e_X27e/group. uid___A001_X12e_X27f/member.uid___A001_X12e_X280
Project code
MOUS uid

14. Check README
There also should be raw/ directory if you downloaded asdm packages and untar them.
member.uid___A001_X12e_X280]$ ls calibrated calibration log product qa README script
member.uid___A001_X12e_X280]$ less README
NB: calibrated directory can be created by running scriptForPI.py. This was already done for this tutorial..
Use same (or newer) version of CASA and pipeline

15. For this data, there are two execution blocks.
Check Weblog
Weblog is a system of webpages containing all the diagnostic plots and other information generated by the pipeline
For this data, there are two execution blocks.

16. QA2 Report for Manual Calibration
Instead of weblog, several png images and a text file for each EB, so-called “QA2 report”, is provided under qa/ directory.

17. QA2 report (png images)

18. QA2 report (text file)

19. Let’s check weblog together
member.uid___A001_X12e_X280]$ ls
calibrated calibration log product qa README script
member.uid___A001_X12e_X280]$ firefox qa/pipeline T113145/html/index.html &
Check observation summary
Check flagging summaries
Check calibrated data
If any suspicion, check relevant calibrations
Check flux consistency
Search calibrator fluxes in ALMA database using aU.getALMAFlux

20. Observation Summary
Click here
Click here

21. Check flagging summaries
Go to “By Topic”
This page shows how much fraction of the data are flagged by pipeline per antenna.

22. Check calibrated data Go to “By Task”, click 15. hifa_applycal
If any suspicious feature, check relevant calibrations.
e.g.,
strange bandpass shape -> check bandpass/Tsys calibration
Time variable phase -> check gain/wvr calibration
Click here

23. Check flux consistency
Go to 13.hfa_fluxscale
For this data, a quasar J was chosen as a flux calibrator. The pipeline obtained the flux mode of this calibrator from the measurement in ALMA database close in time. See 10.hifa_setjy for the flux model.
This page shows the flux densities of other calibrators derived from the flux scaling using the model of J

24. Retrieve Calibrator Flux in database
CASA <2>: aU.getALMAFlux(sourcename='J ',date=' ',frequency=' GHz')
Using Band 3 measurement: (age=1 days) GHz
Using Band 7 measurement: (age=2 days) GHz
exact value: , 1-sigma extrema: , , mean unc=
Median Monte-Carlo result for = (scaled MAD = )
Result using spectral index of for GHz from GHz = Jy
Out[2]:
{'ageDifference': 1.0,
'fluxDensity': ,
'fluxDensityUncertainty': ,
'meanAge': 1.5,
'monteCarloFluxDensity': ,
'spectralIndex': ,
'spectralIndexUncertainty': }

25. Produce Calibrated MS Download tarred asdms
Once untar them, asdms will be placed under raw/ directory
Go to script/ directory and run sciprtForPI.py
It can take a few - several hours to finish
In this tutorial, we already created calibrated MSs and provide them. Please download from the tutorial webpage.
Check the calibrated amplitude and phase as a function of time, freq., uv-distance for calibrators (and targets if SN is enough) using plotms
For this data set, we see a dip in the spectrum of phase calibrator shown in weblog. Please confirm if you see the same feature in the calibrated data using plotms.

26. Check bad data [nagai@alma-proc01 calibrated]$ pwd
/work/nagai/Tutorial/ S/science_goal.uid___A001_X12e_X27e/group.uid___A001_X12e_X27f/member.uid___A001_X12e_X280/calibrated
calibrated]$ ls
casa log products uid___A002_Xa48b1f_X1b0.ms.split.cal working
ipython log rawdata uid___A002_Xaa96da_Xb77.ms.split.cal
calibrated]$ casa
CASA <9>: plotms(vis='uid___A002_Xa48b1f_X1b0.ms.split.cal',xaxis='chan',yaxis='amp',avgtime='1e9',avgscan=T,avgantenna=T,spw='1',coloraxis='ant1',field='2')

27. sciprtForPI.py vs. casa_pipescript.py
Running scriptForPI.py reproduces the pipeline calibration done at JAO/ARCs.
Rerun the pipeline calibration with additional flagging can be performed running casa_pipescript.py
Copy casa_pipescript.py from the script directory to the raw directory
Copy flux.csv and *flagtemplate.txt from the calibration directory to the raw directory

28. Introducing Additional Flagging
Additional manual flagging can be introduced to any Pipeline reduction by editing the *flagtemplate.txt files under calibration/ directory.
For this data, one may want to flag channels affected by the atmospheric absorption. Add following line:
mode=manual spw=’19’ channel=’34~38’ reason=‘bad_channel’
Rerun casa_pipescript.py
The data of DA56 in spw=17 was flagged by data reducer because of amplitude outlier.
# User flagging commands file #
# Examples
# Note: Do not put spaces inside the reason string !
# mode=manual correlation=YY antenna=DV01;DV08;DA43;DA48&DV23 spw=21:1920~2880 autocorr=False reason='bad_channels'
# mode=manual spw='25:0~3;122~127' reason='stage8_2'
# mode=manual antenna='DV07' timerange='2013/01/31/08:09:55.248~2013/01/31/08:10:01.296' reason='quack'
mode=manual antenna='DA56' spw='17' reason='timegaincal_outlier_amp'

29. Editing Calibrator Fluxes
For some reason, you may want to redo the flux calibration
Referenced calibrator flux is old
Quasar which is used for the flux calibrator is time variable
Check Calibrator database
i) Edit flux.csv and rerun casa_pipescript.py or ii) give flux rescaling task in sciptForImagingPrep.py and run it on the calibrated MSs.
Note QA2 data reducers normally checked the flux consistency of calibrators and did the flux rescaling if necessary at EA-ARC. However, about 20% of EA data is processed at JAO. Those may not be checked as careful as what we do. The criteria to apply/un-apply flux rescaling could be different among ARCs.

30. Example of scriptForImagingPrep.py
setjy(vis = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal',
standard = 'manual',
field = 'J ',
fluxdensity = [1.89, 0, 0, 0],
spix = -0.43,
reffreq = '233.0GHz')
os.system('rm -rf ../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf')
gaincal(vis = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal',
caltable = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf',
solint = 'inf',
combine = 'scan',
refant = 'DV16',
gaintype = 'T',
calmode = 'a')
applycal(vis = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal',
field = '0,3', # J ,NGC5064
gaintable = '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf',
gainfield = '0', # J
calwt = F,
flagbackup = F)
print "# Concatenating the data."
concat(vis = ['../calibrated/uid___A002_Xb020f7_X32f5.ms.split.cal', '../calibrated/uid___A002_Xb0ebd1_X6594.ms.split.cal'],
concatvis = '../calibrated/calibrated.ms')
Give a “correct” flux density to the phase calibrator
Run amplitude gain calibration using the given flux model
Apply new gain calibration to the phase calibrator and target

31. Imaging Products
Imaging products (FITS of primary-beam corrected image namely *.pbcor.fits and primary beam coverage namely *.flux.fits) are found under product/ directory
Imaging script used for produce these products can be found under script/ directory (scriptForImaging.py).
product]$ pwd
/work/nagai/Tutorial/ S/science_goal.uid___A001_X12e_X27e/group.uid___A001_X12e_X27f/member.uid___A001_X12e_X280/product
product]$ ls
calibrated_final_cont.flux.fits SN1987A_SiO76R2.pbcor.fits
calibrated_final_cont.mask uid___A001_X12e_X280.J _bp.spw17.I.fits
calibrated_final_cont.pbcor.fits uid___A001_X12e_X280.J _bp.spw19.I.fits
cont.mask uid___A001_X12e_X280.J _bp.spw21.I.fits
SN1987A_SiO76R1.flux.fits uid___A001_X12e_X280.J _bp.spw23.I.fits
SN1987A_SiO76R1.mask uid___A001_X12e_X280.J _ph.spw17.I.fits
SN1987A_SiO76R1.pbcor.fits uid___A001_X12e_X280.J _ph.spw19.I.fits
SN1987A_SiO76R2.flux.fits uid___A001_X12e_X280.J _ph.spw21.I.fits
SN1987A_SiO76R2.mask uid___A001_X12e_X280.J _ph.spw23.I.fits

32. Check Image CH=0 CH=2 CH=4 Click here to zoom-up
CASA <2>: viewer('calibrated_final_cont.pbcor.fits')
CASA <4>: viewer('SN1987A_SiO76R2.pbcor.fits')
CH=0
CH=2
CH=4
Will explain how to display - beam size and other parameters
- integrated spectrum in a given region
- image statistics
- source size
- Gaussian model fitting

33. Parameter control in scriptForImaging.py
# imaging control #
# The cleaning below is done interactively, so niter and threshold can
# be controlled within clean.
weighting = 'briggs'
robust=0.5
niter=1000
threshold = '0.0mJy'
#############################################
# Imaging the Continuuum
# Set the ms and continuum image name.
contvis = 'calibrated_final_cont.ms'
contimagename = 'calibrated_final_cont'
# If necessary, run the following commands to get rid of older clean
# data.
#clearcal(vis=contvis)
#delmod(vis=contvis)
for ext in ['.flux','.image','.mask','.model','.pbcor','.psf','.residual','.flux.pbcoverage']:
rmtables(contimagename+ext)
clean(vis=contvis,
imagename=contimagename,
field=field,
mode='mfs',
psfmode='clark',
imsize = imsize,
cell= cell,
weighting = weighting,
robust = robust,
niter = niter,
threshold = threshold,
interactive = True,
imagermode = imagermode)
weighting = 'briggs'
robust=0.5
niter=1000
threshold = '0.0mJy'
Choice of robust parameter changes the beam size and image quality.
Smaller value, higher resolution
Larger value, less sidelobe (if the uv-coverage is very bad)
But, optimal parameter always depends on the uv- coverage

34. Trial of different robust parameter
execfile(‘scriptForContImaging_rb2.0.py’)

35. Want to do manual calibration?
If you would like to do calibration manually instead of the pipeline, try Eric’s script generator (developed by Eric Villard in JAO).
Type es.generateReducScript(‘asdm_name’) on CASA term
analysisUtils should be installed.
This will return a set of calibration tasks
An example script uid___A002_Xa48b1f_X1b0.ms.scriptForCalibration.py is found under calibrated/ directory