CUC Oct 2005 Science Data Systems Jonathan McDowell

SDS – Oct 2005 ● CIAO Evolution Overview ● PSFs ● Chandra Source Catalog

Staffing ● Shrinking science team: – Martin Elvis back to full time research (2004) – Mike Wise leaving for Amsterdam (Dec 2005) – Eric Schlegel to U.Tex/San Antonio (Aug 2005) – Dan Harris to “retirement” (aka science) (Sep 05) – No replacements

CIAO Releases ● CIAO Feb 05 bugfix – acis_build_badpix header problems, support for parameter block input ● CIAO – Jun 05 – Repro3 support – Asphist fixes for multiple observations – Mkacisrmf bugfixes and support for new CALDB – New algorithm for HRC degap: will improve HETG wavelengths – Change to bad pixel handling in VFAINT mode

CIAO Schedule ● CIAO 3.3 – November (Tool release) – New science tools and bug fixes – reproject_image, reproject_aspect ● CIAO 4.0 – 2006 (Infrastructure release) – Sherpa, Chips: major functionality improvements – Data Model improvements ● CIAO script releases – TBD ● SAOSAC release - TBD

CIAO 3.3 ● reproject_image, reproject_aspect ● DM ability to handle data cubes (filter on regions, make slices, etc.) ● Proposal tool support for Cycle 8 ● Fixes to region area bugs ● Fix problem with FTOOLS/CIAO clash (renamed cfitsio library)

CIAO Future ● Areas needing improvement for users: – PSFs – Merging observations – Background modelling – Easy fluxes and upper limits – Publication quality plots – Robust, fast and programmable spectral and spatial fitting (improved Sherpa)

PSFs: User needs and our plans ● User need: – Ability to make custom PSFs ● Plan: Distribute portable SAOSAC/MARX combo ● User need: – Estimate PSF fraction easily ● Plan: encircled-energy fraction tables in CALDB and software to use them; some numerical issues at present

PSFs: ECF table status ● Normalization issue in wings: tracking down some discrepancies ● HRC-only (flat plane) values: characterizing error introduced wrt ACIS tilted-chip, appears significant

PSFs: SAOSAC status ● SAOSAC (CXC Optics Group) consists of a heterogeneous group of 40 programs and associated scripts: C,C++, Fortran, Perl, Lua. Designed as engineering tool ● Available to users via CHART web interface ● Moving to portable version isolated from references to local disks, new build system ● Modifying to handle numerical differences between Sparc and x86 architecture

PSFs: SAOSAC and users ● SAOSAC outputs ray files. Need to project to focal plane and intersect with instrument ● Will model instrument effects on PSF (but should be optional, since PSU has its own detector model) ● Will use MARX for both projection and default instrument model (may replace/enhance instrument model later)

PSFs: SAOSAC/MARX driver script ● We will write a driver script with a simple interface. For a first release: – Single point source only, no grating – Specify spectrum ( via PHA file or Sherpa model syntax?) – Specify off-axis angle and azimuth – Specify SIM position and fiducial correction – Specify date? (for time-dependent instrument cal?) – Specify instrument model (MARX or none)

Implementation Plan ● CXC Optics to complete SAOSAC portable infrastructure by December ● Perform SDS testing and resolve Linux issues ● Identify resources for driver script work ● Documentation effort to guide users ● Identify release schedule by end January ● Release asynchronously with respect to CIAO?

Chandra Source Catalog ● Chandra Level 3 Pipeline ● Make a definitive catalog of sources useful for individual and statistical studies ● “Blind”, automatic processing capable of handling different instrument configurations and different kinds of field (crowded, extended, etc.) ● Runs over full field (handle large off-axis PSFs) ● Good for stars, galaxies, AGN, clusters, etc.. not just optimized for one kind of source

Chandra Source Catalog ● Not just a static catalog: ● Access to extracted data products for each source via the catalog ● Dynamic interaction to produce user-configured views with on-the-fly processing ● Continuous updating as mission continues and data becomes public

Chandra Source Catalog ● In early years of mission, steadily improved L1 and L2 calibrated products (event lists) ● We ran detect and made source lists, but these products were provided as a guide only. ● Our understanding of the instruments is now at the point where automatic processing can go further along the analysis chain – this was always part of our plan.

Chandra Source Catalog ● New standard data products: extracted spectra, ARF, RMF and 4-band postage-stamp images for each source. ● New standard data product: high quality source list ● New standard data product: source properties table for each source, with accurate positions and fluxes from 2D PSF fitting as well as extent and variability flags

Chandra Source Catalog ● Energy bands for ACIS: – Broad keV – Soft keV – Medium keV – Hard 2.0 – 7.5 keV ● Only one broad band for HRC ● Bands selected after comparison with those in use in the community; compatibility with XMM

Chandra Source Pipeline - 1 ● Reprocess event lists with latest cal, and perform high background (flare) cleaning ● Make exposure maps in each band for false edge source suppression and construction of sensitivity map ● Run multi-scale WAVDETECT in each band – Scales 1,2,4,8,16 ● Identify sources across scales and bands ● Define source and (annular) background regions for each source

Overlapping sources

Chandra Source Pipeline - 2 ● Make postage stamp fluxed image in each band ● Use SAOSAC to make PSF for source ● 2D gaussian (PSF-convolved) fits to image, attempt to identify source as single, double or complex ● Derive source fluxes and position(s). Use power law spectral fit with alpha=1 and galactic NH. (Free alpha if enough counts, currently n=300). ● Correct with ECF (PSF fraction) tables ● Variability test (currently KS, soon Gregory-Loredo)

Near term enhancements ● Merge pipeline: combine source data from multiple observations and instruments, get best fluxes and long-term variability estimate ● Remove readout streaks (“out of time events”) ● Revise wavdetect parameters based on pilot studies and characterize completeness, sensitivity, false source rate ● Complication: 2D fitting can discover extra sources, so threshold is not simple to define.

Proposed enhancements ● PSF fraction correction (as fn. of energy) for sources. Plan to use fraction within given contour, rather than PSU method of selecting contour based on given fraction. (Will be useful user tool). Method more appropriate from blind pipeline with avoidance of nearby sources (interactive region tweaking not possible in pipeline) ● Background maps

Longer term enhancements ● Handle very extended sources (SNRs, clusters) using Voronoi tesselation detect ● Mosaic regions (M31, Orion..) ● Gratings data (some day...)

Reviews of other processing pipes ● ACIS_EXTRACT reviewed by M. McCullogh, F. Primini for functionality; further tests planned. We are adopting the important approach of energy-dependent PSF fraction corrections to the ARF. ● AE Currently uses mkpsf for PSF generation, which we don't have confidence in. ● XASSIST (A. Ptak) also under review; has no PSF fitting yet ● XMM Catalog: Plan is to do simple celldetect with no position refinement, and bright threshold (fewer sources) ● CHAMP: Used in initial evaluation; no HRC support.

Operations plan ● Initial evaluation based on 6 test sequences ● Run larger number of sequences for science testing over a period of months, while merge pipeline in development ● Start production processing circa Fall '06 ● Release initial public product when about 1/5 of mission has been processed, catalog will evolve as more is processed. ● 1 year to process first 5 years ● Access to “live” evolving catalog and to fixed snapshots

User interface ● Web-based GUI interface ● Links to data, source and background events, images, spectra and responses; searches on all columns ● VO-aware interface with ability to define on-the-fly catalogs in spreadsheet-like way; iterative searches ● Later enhance to allow workflows (run queries which involve processing on the actual data)