1. XMM data reduction: part II SAS command-line analysis and scripting Andy Read

2. Weds 14th Dec 2005 XMM data reduction: part II 2 Overview Command-line examples/help Why use command-line and scripts? Example scripts : SASmakecalev & SASprepareXMM Imaging scripts & spectral scripts Scripts within scripts Final remarks

3. Weds 14th Dec 2005 XMM data reduction: part II 3 Setting up the user environment To run on XROA system: > sas-setup-6.5.0 initialises software > setenv SAS_ODF (path_to_ODF_directory) e.g. /data/71/tro/xmm_data/0106860101/odf/ > setenv SAS_CCFPATH /data/rb2/sas_ccf > cifbuild fullpath=yes >& cifbuild.log builds ccf.cif – calibration index file > setenv SAS_CCF (path_to_ccf.cif_file) > odfingest odfdir=$SAS_ODF outdir=$SAS_ODF >& odfingest.log builds ***SUM.SAS file in ODF directory – ODF summary file necessary for reprocessing (from Tim’s talk) Command line examples

4. Weds 14th Dec 2005 XMM data reduction: part II 4 Command line help - Simple help – parameters and default values

5. Weds 14th Dec 2005 XMM data reduction: part II 5 - More - Parameters, parameter types, default values and ‘one-line’ help

6. Weds 14th Dec 2005 XMM data reduction: part II 6 - Full help – Links to SAS web pages - Cut and paste link into browser

7. Weds 14th Dec 2005 XMM data reduction: part II 7 Why use scripts? Useful scripting cookbook at http://starlink.rl.ac.uk/star/docs/sc4.htx/sc4.html http://starlink.rl.ac.uk/star/docs/sc4.htx/sc4.html - Scripts are just a list of command-line calls/commands run sequentially - There are many powerful command line calls - Can run intensive analysis on many datasets, sources etc. - Can be sure of doing exactly the same thing every time, e.g. running exactly the same procedures on many sources/datasets - No need for interactive use - can run overnight (no need to be there) - can run as cron jobs (no need to be even logged in) - Often need to redo entire analysis, e.g. - New calibration files - New ODF files - Mistake, change in e.g. source positions, detection threshold etc.

8. Weds 14th Dec 2005 XMM data reduction: part II 8 Example script : SASmakecalev

9. Weds 14th Dec 2005 XMM data reduction: part II 9 Executable t-shell script Comment lines Runs at low priority On-screen help – appears whenever incorrect number of arguments is given (e.g. zero) Number of arguments If correct number of arguments is given (7) then input arguments (1-7) converted to parameters used in script, and we move into the main body of the script

10. Weds 14th Dec 2005 XMM data reduction: part II 10 Check that ODF exists (otherwise exit) Get SAS version (for directory naming) Create (if necessary) output directory, and (if allowed) enter

11. Weds 14th Dec 2005 XMM data reduction: part II 11 Check ODF Set SAS_ODF environment variable Check for ***SUM.SAS file (Check for ***SUM.ASC file) Create ccf.cif – calibration index file Set SAS_CCF environment variable Create ***SUM.SAS file in ODF directory – ODF summary file necessary for reprocessing Output all SAS environment variables to screen – useful check

12. Weds 14th Dec 2005 XMM data reduction: part II 12 Run emproc (if required) Run emchain (if required) Run epproc (if required) Run epchain (if required) List created final calibrated event lists and exit

13. Weds 14th Dec 2005 XMM data reduction: part II 13 Run script

14. Weds 14th Dec 2005 XMM data reduction: part II 14 Example script : SASprepareXMM

15. Weds 14th Dec 2005 XMM data reduction: part II 15 Soft Proton flaring in an observation of a Galaxy Group

16. Weds 14th Dec 2005 XMM data reduction: part II 16 Executable t-shell script Comment lines Runs at low priority On-screen help – appears whenever incorrect number of arguments is given (e.g. none) Number of arguments

17. Weds 14th Dec 2005 XMM data reduction: part II 17 If correct number of arguments is given (9) then input arguments (1-9) converted to parameters used in script If requested : Check ODF Check for ***SUM.SAS file Create ***SUM.SAS file in ODF Set SAS_ODF environment variable

18. Weds 14th Dec 2005 XMM data reduction: part II 18 If requested : Create ccf.cif – calibration index file Set SAS_CCF environment variable Output all SAS environment variables to screen – useful check

19. Weds 14th Dec 2005 XMM data reduction: part II 19 Convert patcode parameter into an expression for evselect

20. Weds 14th Dec 2005 XMM data reduction: part II 20 MOS1 example (comment)

21. Weds 14th Dec 2005 XMM data reduction: part II 21 If $m1 does not equal 0, OK – continue to next line If $m1 equals 0, ignore this whole section

22. Weds 14th Dec 2005 XMM data reduction: part II 22 If $m1 equals F, then we attempt to find the file Else, the file name is given by the parameters

23. Weds 14th Dec 2005 XMM data reduction: part II 23 Remove a temporary file if it already exists

24. Weds 14th Dec 2005 XMM data reduction: part II 24 Long list all the MOS1 event files in the directory to temporary file

25. Weds 14th Dec 2005 XMM data reduction: part II 25 Count the number of entries (MOS1 event files) in the temporary file

26. Weds 14th Dec 2005 XMM data reduction: part II 26 If only one MOS1 event file, use this

27. Weds 14th Dec 2005 XMM data reduction: part II 27 If more than one MOS1 event file, try to find the largest

28. Weds 14th Dec 2005 XMM data reduction: part II 28 Initiate loop through ‘i’ (i=0)

29. Weds 14th Dec 2005 XMM data reduction: part II 29 loop through ‘i’ until ‘i’ equals number of event files, then end

30. Weds 14th Dec 2005 XMM data reduction: part II 30 Increment ‘i’ by +1 (i=1)

31. Weds 14th Dec 2005 XMM data reduction: part II 31 Get size of ‘i’th file : Type the file, extract the top (head) ‘i’ lines, then extract the bottom (tail) line – file size is the 5 th string in long listing

32. Weds 14th Dec 2005 XMM data reduction: part II 32 If size of ‘i’th file is larger than largest recorded, note ‘i’ (‘ibig’) and size as largest recorded

33. Weds 14th Dec 2005 XMM data reduction: part II 33 Loop round to next ‘i’

34. Weds 14th Dec 2005 XMM data reduction: part II 34 Get name of largest file (i.e. file corresponding to ‘ibig’) : Type the file, extract the top (head) ‘ibig’ lines, then extract the bottom (tail) line – file name is the 9 th string in long listing Set $m1 to this filename

35. Weds 14th Dec 2005 XMM data reduction: part II 35 Create (evselect) lightcurve : High-energy, single events, ‘good’ flags, 100s bins

36. Weds 14th Dec 2005 XMM data reduction: part II 36 Output help to screen: What is being done, what the user should do

37. Weds 14th Dec 2005 XMM data reduction: part II 37 Plot (dsplot) the light curve to the screen for user to examine

38. Weds 14th Dec 2005 XMM data reduction: part II 38 Request upper and lower threshold values from user

39. Weds 14th Dec 2005 XMM data reduction: part II 39 Use threshold values to create (tabgtigen) good times GTI file

40. Weds 14th Dec 2005 XMM data reduction: part II 40 Create (evselect) intermediate event file : filter on low-E threshold, pattern, flags, etc.

41. Weds 14th Dec 2005 XMM data reduction: part II 41 Create (evselect) final event file : filter intermediate file for good times using GTI file

42. Weds 14th Dec 2005 XMM data reduction: part II 42 Same for MOS2

43. Weds 14th Dec 2005 XMM data reduction: part II 43 Same for pn

44. Weds 14th Dec 2005 XMM data reduction: part II 44

45. Weds 14th Dec 2005 XMM data reduction: part II 45 Enter MOS1 thresholds Same for MOS2 Same for pn New filtered event files (r1***EV***) smaller than originals

46. Weds 14th Dec 2005 XMM data reduction: part II 46 Already seen xmmselect GUI - clean data, and produce science products Only handles one dataset at a time Essentially a handy GUI ‘wrapper’ to the SAS-task ‘evselect’ evselect is the command-line workhorse of SAS analysis tasks Many scripts are command-line ‘wrappers’ of evselect and other SAS-tasks

47. Weds 14th Dec 2005 XMM data reduction: part II 47 Script to create images - Essentially an evselect wrapper, plus other tasks (e.g. asmooth)

48. Weds 14th Dec 2005 XMM data reduction: part II 48 MOS1 MOS1 smoothed MOS2 MOS2 smoothed pn pn smoothed e.g. 300-2000eV sky (X/Y) 2′′ images (raw and smoothed) [MOS-sdtq, pn-sd]

49. Weds 14th Dec 2005 XMM data reduction: part II 49 Script to create spectra + ARF, RMF and do grouping etc. Again, evselect wrapper, plus arfgen, rmfgen, grppha etc… SAS-task equivalent especget now much improved

50. Weds 14th Dec 2005 XMM data reduction: part II 50 createspectrum to create BG spectrum Note: can set parameters (here, co-ordinates from Tim’s region files) on the command line Run script (using parameters)

51. Weds 14th Dec 2005 XMM data reduction: part II 51 createspectrum to create source spectrum, ARF, RMF and group final spectrum

52. Weds 14th Dec 2005 XMM data reduction: part II 52

53. Weds 14th Dec 2005 XMM data reduction: part II 53 Can put each step/script into a file (e.g. analysis) – A script to call other scripts Can then run whole ‘analysis’: > source analysis

54. Weds 14th Dec 2005 XMM data reduction: part II 54 Homework (2005) Extract NGC 1313 data (Oct 2000) from XSA Set up analysis environment Filter data for background flares Extract 0.3 – 10 keV images for all three detectors Extract spectra of NGC 1313 X-1 Compare your results to  /data/71/tro/xmm_data/0106860101/work/ Homework (2006) – use scripts (and write a script of your scripts) to do all the analysis

55. Weds 14th Dec 2005 XMM data reduction: part II 55 Resources This talk (and others):  http://www.star.le.ac.uk/~sav2/stats/ http://www.star.le.ac.uk/~sav2/stats/ The scripts (and others):  http://www.star.le.ac.uk/~amr30/Scripts/ http://www.star.le.ac.uk/~amr30/Scripts/ XMM-Newton SAS web pages  Via http://xmm.vilspa.esa.eshttp://xmm.vilspa.esa.es  Particularly useful documentation: HEASARC ABC guide SAS user’s guide Experienced users!