1. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Science Data Systems (SDS) CUC Presentation Jan 2005

2. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Science Data Systems - supporting user science with software and data products Work closely with Data Systems to provide science overview of software and detailed specs CIAO - the software package (and documentation!) for user data analysis Pipelines - ensure data products created with correct algortihms and contain the needed metadata CALDB - figure out how raw calibration results from CAL team must be packaged to support user analysis and data processing From CUC point of view, we mediate between users and Data Systems teams to deliver improved user software

3. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Science Data Systems - under new management Jonathan McDowell, SAO - group leader: jcm@cfa.harvard.edu Mike Wise, MIT - deputy leader: wise@space.mit.edu SDS is about 50/50 SAO and MIT: an integrated team Close coordination with Pepi Fabbiano's Data Systems groups, especially Janet DePonte Evans' software development teams: CIAO and the pipelines are joint SDS/DS products Let us know how we can improve your ability to do science with CIAO and with CXC data products

4. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Science Data Systems: summary Recent releases –CIAO 3.2 –APPHOT and DS9 interaction Pileup Report – Documentation – Estimation – Correction What else is coming –Improvements –Enhancements

5. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 CIAO 3.2 Highlights New 'How does the new release affect your analysis' web page New mkacisrmf tool released Spatial model of the ACIS contamination New ACIS hot pixel tools CALDB 3.0

6. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 CIAO 3.2 Web Pages

7. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 mkacisrmf infile=acisD2000-01-29p2_respN0001.fits outfile=source_pi.wrmf \ energy=0.1:10.0:0.05 channel=1:1024:1 chantype=pi \ wmap=spectrum_pi.fits gain=acisD2000-01-29p2_gain_ctiN0001.fits New tool in CIAO 3.2 to support changing ACIS calibration Implements algorithm of A. Vikhlinin ( calcrmf2 ) Separates RMF calculation into position dependent and independent pieces “Ideal” response computed at readout prior to the effects of CTI (one per CCD) Scatter matrix computed automatically from simulations including CTI Faster execution, better interpolation, and retains all existing mkrmf functionality mkacisrmf Ideal Response Total Response Scatter matrix  = Energy PHA

8. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 New RMFs provide small improvements over mkrmf products 10-15% more accurate at energies below 1 keV Currently calibrated for ACIS chips 0-3, 5, 6, and 7 at -120  C In the future, only mkacisrmf Cal. products will be produced Released mkrmf RMFNew mkacisrmf RMF

9. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Spatial Model for the ACIS Contaminant The ACIS QE degradation model has been enhanced to account for spatial variations in the contamination on the ACIS optical blocking filters. The contamination is now expressed as a function of time, energy, and ACIS chip coordinates. Temperature structure of ACIS OBF Instrument maps ( mkinstmap ) ACIS-I2ACIS-S3

10. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Spatial model implemented in all CIAO 3.2 tools Default model in CIAO 3.2 / CALDB 3. 0 mkarf, mkgarf, mkwarf, mkinstmap ~20% variation across chip Extended or off-axis sources most effected Affects grating ARFs as well

11. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Status of APPHOT Package Frank Primini's X-ray aperture photometry scripts demoed at last CUC as an example of scripting You asked for more details on their availability They are now real software and users can have them! And you asked how we did those cool things with ds9 I'll tell you how, and we'll soon put this up as a thread for users Working with Data Systems to formalize how we develop and release interactive scripts as we increase their use. Stay tuned.

12. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Status of APPHOT Package Version 1.1.0 available for interested users athttp://hea- www.harvard.edu/~fap/APPHOT/. Will be added to CIAO contributed software after additional testing. – Installs under CIAO 3.2; no additional installation of XPA or SLxpa modules required – Can now determine actual ECF of aperture defined by user – Faster computation of Theta, Phi (now uses S-Lang pixlib module) Version 1.2.0 in development. – Choice of method for accessing ECF data file ● Interpolation in theta, phi, energy ● Nearest theta, phi, energy in ECF data cube – Plots of ECF vs. aperture radius

13. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Status of APPHOT Package Additional development suspended pending evaluation of Penn State's ACIS Extract S/W package – ACIS Extract richer, more sophisticated. We are considering porting its IDL-based functionality to CIAO's open source environment – APPHOT could still be used to provide quick assessment of aperture sizes, e.g., for proposal planning

14. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Launching CIAO Tools from ds9 The ds9 image display program can be used to launch user-defined analysis tasks which can obtain input parameters from and display results to ds9. This capability is accomplished by means of ASCII “analysis commands files”, loaded into ds9 via the Analysis Menu, and is described in detail in the ds9 Reference Manual. Typically, this makes sense when one is examining an image and wishes to define regions or image locations at which, say, counts or PI spectrum are desired. These can be obtained and displayed quickly using CIAO tools. In the following pages we present some examples of analysis command files that launch CIAO tools.

15. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 A Simple Analysis Commands File The analysis commands file (ACF) is a structured ASCII file that contains command lines plus keywords that tell ds9 how and when to run the command. The command file can take advantage of a number of ds9 internal variables, or macros, that expand to arguments containing information on the current ds9 display, prior to command execution. The full list of macros is provided in the ds9 Reference Manual. Of particular importance are the following: ● $filename expands to the filename of the displayed image or event list; ● $regions expands to the list of regions (in various formats) defined in the display; ● $text displays any text output from the command to a text window. The ACF can also define parameters, which are set in a parameter window, and which may be passed as arguments to the command line. The following page contains a simple ACF that can be used to display some ds9 macros and run the CIAO tool dmlist to display header or data. Results are displayed on subsequent pages.

16. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 The ds9_ciao.ans ACF

17. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Loading the ACF Clicking the Analysis button allows users to load ACFs via the Load Analysis Commands submenu

18. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Running dmlist Clicking dmlist invokes the parameter window Clicking on dmlist invokes the parameter window, in which values of the “option” parameter may be set Clicking OK runs the dmlist command line

19. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Results for dmlist Clicking dmlist invokes the parameter window Clicking on dmlist invokes the parameter window, in which values of the “option” parameter may be set Clicking OK runs the dmlist command line When dmlist finishes, the results are displayed in a pop-up text window

20. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 A More Complicated (but still simple) Example It may be difficult to build complicated CIAO command lines that can be successfully parsed by ds9's command line parser. Rather, the user may wish to pass the arguments obtained from parameter windows or ds9 macros to a shell script with sufficient string handling capabilities to build and execute a valid CIAO command line. The following page illustrates how that can be done in a script that runs the CIAO tool dmcoords to determine the off-axis angle of a point selected in a ds9 display.

21. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Launching dmcoords from ds9_ciao.ans

22. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Running dmcoords from a Script

23. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Results for dmcoords Clicking dmlist invokes the parameter window Clicking on dmlist invokes the parameter window, in which values of the “option” parameter may be set Clicking OK runs the dmlist command line When dmlist finishes, the results are displayed in a pop-up text window Clicking dmcoords runs the program to determine off-axis angle for the image location specified by this region. Results are displayed in the pop-up dmcoords window.

24. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Advanced Use The previous examples were only intended to illustrate the basics of setting up an ACF, defining parameters, and passing arguments to CIAO tools. In practice, the most flexible way to launch CIAO tools from ds9 is via scripts that communicate directly with ds9 via the XPA Messaging System. This can be accomplished either by running the various XPA programs (xpaget, xpaset, etc.) from the command line within the script, and then parsing the results for the desired input, or by using a scripting language, such as S-Lang, which provides bindings to the XPA library. Users who wish to use these advanced features should examine the scripts in the APPHOT package.

25. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 SDS Efforts for Pileup Helping users diagnose and deal with ACIS pileup is an SDS top priority for 2005 I'll present the current status

26. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 SDS Efforts for Pileup Comprehensive ABC Guide (status: draft in work) –One Stop Shopping for Information and Advice New Diagnostic Tool For Detecting Piled Sources (research) –Concept: Detects Piled PSFs –Potentially can Help Calibrate MARX’s Pileup Module Tools for Dealing with Piled Gratings Spectra –Simple Model Soon Available –Sophisticated Model in Development Future Work –Models of Piled Lightcurves

27. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 “The Chandra ABC Guide to Pileup”: Outline Introduction –Basic Definitions, Pileup Fractions and Their Effects, Methods for Avoiding Pileup, When Not to Avoid Pileup Estimation –PIMMS, Simulation with Spectral Fitting Packages (Sherpa, ISIS, XSPEC), MARX Detection –Simple Methods, Future New Detection Tool Mitigation –Spectral Fitting Packages (Sherpa, ISIS, XSPEC), Simple Gratings Models, (Future) Advanced Gratings Models

28. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 “The Chandra ABC Guide to Pileup”: Goals Consolidate Information: –Proposers Guide, PIMMS Ahelp, Sherpa Ahelp, Threads, Journal Articles, etc. Provide ‘Practical’ Advice: –Pros & Cons of Avoidance Strategies (Greater Detail than Proposer’s Guide) –Subtleties of Detection Methods –Subtleties of Spectral Fitting Mitigation Techniques

29. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 New Detection Tool in Research/Development Compares PSF from 2-pixel split events to other events Provides Statistical Assessment of Whether the Two Distributions are Consistent (i.e., Not Piled) Does not Directly Rely Upon Spectra Potentially sensitive to Even Mild Pileup Side Benefit: Statistics Module for CIAO (e.g., K-S Test, Mann-Whitney Test, etc.)

30. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 PSFs Distort Under Pileup (In Complex Ways!) Working Concept: ‘Split’ Events PSF Distorts Differently than that for Other Events

31. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Low (Unpiled) Count Rate: PSF from Split Events and Other Grades Appear the Same p-value > 0.05 Indicates Normalized PSF Distributions are Indistinguishable (Off Axis Source)

32. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Example: Radio Galaxy Nuclear Emission, Difficult to Assess Pileup via Spectra Observation of Gambrill et al. 2003, A&A, 401 Source was Thought to be Unpiled

33. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Normalized Grade Selected PSFs Differ p-value < 0.05 Indicates Differences are Statistically Significant: Good Indication of Mild Pileup?

34. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 PSF Can be Assessed with the MARX Pileup Module: Consistent with Mild Pileup

35. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Spectral Fit Function, Based on Observed Counts/Frame/Pixel, for Simple Correction 10% Pileup Correction Simple Spectral Model, Only Applicable to Sources with Peak Pileup Approximately < 10%. (Here Applied to an X-ray Binary Spectrum)

36. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 More Sophisticated Correction in Development Will Incorporate Detector Efficiency Variations, Deadtime Effects, Energy Dependence of PSF/LSF, etc.

37. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Spectra, with Peak Pileup of ~20%, Corrected

38. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Ongoing Work Better Afterglow Correction: acis_run_hotpix The Most Recent Gratings LSF: mkgrmf In Cooperation with Calibration, Need to Incorporate Better Models of: –Cosmic Ray Dead Area Correction –Backside/Frontside Correction –Iridium Edge Correction Exploring Possibility of Creating a Chandra Lightcurve Simulation, which Would Include Pileup Effects – Adapt Existing XEUS Lightcurve Simulator for Chandra Use – MARX Provides Accurate Mean (White Noise) Piled Lightcurve, this Could Provide Piled Lightcurves with Realistic (Red Noise, QPO) Variability Features Requirements for Gratings Pileup Correction: Future Pileup Work:

39. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 What’s coming in CIAO 4.0 Infrastructure improvements: must keep up with users' environment –Increased platform support (under review - newer Linux builds) –64 bit and large file support –Performance improvements (pixlib ~ 10 times faster) –Ability to build from source Enhancements: filling gaps in user science capabilities –New CHIPS plotting package; Sherpa improvements –Improved Backgrounds (ABC guide, scripts, tools) –Multi-observation support (mosaicing) Schedule and other enhancements still under review

40. SDS Presentation Jonathan McDowell CXC 1 Chandra Users Committee Meeting, 25 January 2005 Sherpa status Already in CIAO 3.2: bug fixes for improved stability, projection error method improvements, and revised manual We are now conducting a major review Reworking the optimization methods (including evaluating adding the LMDIF algorithm) for faster and more accurate convergence Expect to rework Sherpa to modularize it - make separate libraries available for models, optimization methods, etc. with defined interfaces to let us write small tools and scripts using parts of the Sherpa functionality. Will also greatly improve Sherpa code maintainability and stability Must add support for new CHIPS plotting