Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Carlos GABRIEL / Aitor IBARRA XMM-Newton Science Operations Centre – ESAC / ESA RISA: Remote Interface for Science Analysis Work done in collaboration with: Ignacio de la CALLE – Jesús SALGADO – Pedro OSUNA - Daniel TAPIADOR

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Main questions of this talk: 1) “how to enable observers to perform good (the best?) XMM-Newton data analysis?” 2) “how to keep that throughout a long mission and after it ?” Our answers: 1) SAS 2) RISA Motivation + outline

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Main requirements to analysis system when starting the XMM-Newton development: >> basis for the official XMM-Newton products >> interactive analysis system to be used by observers >> capable of dealing in a similar way with all the data from all the XMM-Newton instruments The requirements for an analysis system >> Also behind the 350 ref.pub / year Some additional (by now almost basic) requirements for an observatory’s data analysis system in the XXIth century: * for more and less experienced, occasional and dedicated users, >> several alternatives for data reduction * attracting instead of repelling >> user friendly * developed with the general user in mind, >> react quickly to new trying to pass to her/him best knowledge developments in calibration /processing * should not add unnecessary costs to the users >> no licensing S/W behind * should be used all over the world >> distributed + run on different platforms >> Scientific Analysis System SAS

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS SAS What is the SAS? freely distributed suite of programs (“tasks”) developed by ESA + SSC Consortium

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS - from raw data to calibrated final products, including diagnostic data, cross correlation products... - PPS consisting of SAS tasks running grouped in modules under a dedicated control system Pipeline products

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS * Survey Science Centre UoLeicester finished in July 2007 the reprocessing of all the XMM-Newton data on behalf of ESA >> uniform archive in terms of processing and calibration >> 2XMM catalogue = largest catalogue of X-ray sources = 250k detections >> 200k unique sources Full reprocessing + 2XMM >> D1 + P4.9

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Data from the EPIC cameras

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Extracting an image with xmmselect

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS High resolution spectrometry Dispersion equation: cos β = cos α + mλ / d source calibration

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Optical Monitor- Examples A): OM UV & MOS Images (M81) C): Optical grism ( BPM16274) B): UV & X-ray light curves (X ) D): Supernova 2002p

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Continuous calibration + algorithmic progress SAS version 7.1 released in July 2007 >> main improvements in the area of source detection (2XMM Catalogue), together with * upgrades in EPIC CTI corrections * RGS light curve background subtraction and exposure correction * OM I/A photometry handling >> catalogue production part of full XMM-Newton data reprocessing exercise (complete re-population of XSA in August, just in time for AO7) XMM-Newton has a concept of continuous calibration (instead of versions) permitting immediate delivery of new calibration to observers, so far this does not imply an algorithmic change >> automatic servicing of new CCFs through rsync or mirror >> download of S/W + calibration files necessary whenever something new

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Supported platforms Supported platforms (SAS 7.1.0) Linux Red Hat 9.0: Built on 9. Tested to work on Fedora Core 1. EL3: Built on RHEL 3. Ttw on FC1 and FC2. EL4: Built on RHEL 4. FC3: Built on Fedora Core 3. SuSE 7.3: Built on 7.3. Ttw on Red Hat : Built on 8.2. Ttw on SuSE 9 and 9.1. Solaris 2.8: Built on : Built on 2.9. Mac OS X 10.2: Built on 10.2 (Darwin 6.8 / Jaguar) 10.3: Built on 10.3 (Darwin 7.X/Panther) 10.4: Built on (Darwin 8.2/Tiger) Windows: VM4SAS = virtual machine running Fedora Core 4 on MS Windows

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS SAS 7.0 download statistics Download questionnaire helping us to get customers’ number and profile ~ 1685 registered downloads >> ~ 3000 users (double counting mainly due to Windows VM) Our estimation from former versions: >> actual users

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Reprocessing PPS Maturity of calibration and data reduction capabilities >> reprocessing of all XMM-Newton observations with most recent SAS/CCF leading to much improved datasets, uniformity, additional products + 2XMM Catalogue >> serendipitous X-ray source catalogue from EPIC data (bigger and better than 1XMM) Reprocessing AND 2XMM Catalogue compilation are a single exercise ! (final run in this very moment >> ~ September-October XSA repopulated) Additions to former pipeline: - a much improved source detection (extended sources search and parameterisation, better background estimation, reliability + flux accuracy at faint fluxes) - spectra & time series for brighter sources (> 500 counts ~ 10% of ALL EPIC sources)

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Merging SAS into the XSA The XMM-Newton scientific archive (XSA) belongs to the most advanced, state-of-the-art astronomical archives in the world - playing a key role in the VO initiative We are integrating SAS processing and interactive capabilities into the XSA: - reprocessing selected data using the latest calibration - filtering processed event lists prior to retrieval - deriving interactively spectra and light curves, including filtering, regions selection, etc

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Shaping the future of SAS: RISA So far, SAS runs locally on user’s machine: Integration on several different platforms + distribution Large maintenance due to need of compatibility with new libraries in new versions SOC SAS download + installation + setup necessary Data + Calibration DB download User Running SAS through Web Services? reduction of maintenance due to the limitation to few platforms (1 ?) easy to be fully “frozen” from a certain point in time neither SAS installation nor data download needs by single user automatic access to large H/W and S/W resources (ESAC Grid + VO tools) full data access (close to XSA and central CCF repository) Not expected to render a distributed SAS unnecessary at first, but reducing requirements on maintenance significantly in the long term, at the same time more efficient and comfortable

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS What is RISA? RISA = SAS remote interface allowing users to reduce and analyse XMM-Newton data making use of all of the SAS functionalities, running the processes on the GRID * With a client application, an end-point access and input VOTable files, a user can execute in a Grid environment a SAS workflow. * The workflow starts with the selection of the desired SIAP search (Source Ra-Dec or Name, XMM-Newton Observation ID or Revolution number). * In a second step, the user can select either a standard workflow (“Thread” as provided by the SOC) or a “Users’ Defined Workflow” (with all the SAS tasks available)

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Working with RISA 18

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS RISA - Client side

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS RISA - Client side output The client serializes the RISA input workflow together with the parameters corresponding to each workflow. >>VOtables

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS RISA - Server side SERVER The server app creates Grid template files that are sent to each of the available Grid resources. GRID TEMPLATE The server creates the templates using the GridWay DRMAA and submits the jobs to the GRID for execution. SAS + GRID PERFORMANCE This architecture allows to analyse large amounts of data using remote resources. User free of updating S/W, CalDB, etc. GRID The Workflow is executed in available Grid nodes. Products are stored in a Data Product Storage unit. XSA Each node in the GRID makes a request to the XMM-Newton Science Archive to retrieve the requested data sets.

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS ESAC Grid * HW configuration: 10 x Dell PE 1855 (blades). More in the near future. 2 x Intel Xeon 3.2 Ghz (1 GB Cache) 6 GB RAM 140 GB internal disk (RAID 1) * Grid Middleware: EGEE – gLite 3.0. OS: Scientific Linux 3. WNs will be updated to SLC4.5 x86_64 soon. Computing Element: LCG (globus based). Metascheduler: GridWay (against LCG for the time being) * In production since February 2007 within the SWE (South West Europe) Federation. * Supports Planck VO (and of course dteam and ops). # Starting a collaboration with IFCA (Santander) and soon also INAF-OAT (Trieste) >> communication, scalability

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS The pre-defined Workflows * epic_event_thread >> epproc up to calibrated event list * epic_edetect_thread >> basic data reduction + full source detection

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS The pre-defined Workflows * epic_lightcurve_thread >> basic data reduction (epproc) + coordinates driven light curve extraction

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS The pre-defined Workflows * epic_spectrum_thread >> basic data reduction (epproc) + coordinates driven spectrum extraction

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS The pre-defined Workflows +: * epic_analysis_thread >> combination of “image” + “lightcurve” + “spectra” threads * epic_slew_thread >> full slew analysis (a la RDS) * rgs_thread >> full rgsproc processing soon to appear: - epic full analysis thread (different maximization of S/N) - om threads open to suggestions

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Roadmap - α-version ready and working >> several pre-defined workflows >> access to all individual SAS tasks, including access to all parameters >> access to XSA, including authentication for data download >> ESAC grid working (gLite/EGEE middleware - 12 nodes each of them with 1 CPU and 3GB of RAM) >> large storage area available for products - In the plan: * further standardization of SAS workflows * upgrading of parameters interface (SAS I/F like) * SAS error handling * intermediate + info exchange data concept >> full I/A capability * merging system with an advanced workflow generator (eg. Taverna)Taverna Internal version v0.8 released Public v ~ 12/07 - 3/08 Public v2.0 ~ 09/08

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Conclusions We are: - definitively helping observers to maximize the scientific return from the XMM-Newton mission with a mature, solid, everywhere running analysis system - intending to maintain the high standards throughout the next ~ 15 years through a novel implementation of web services with full scientific analysis capabilities running in the GRID

Carlos Gabriel Science Operations & Data Systems Division Research & Scientific Support Department ADASS London 2007 XMM-Newton ESAC SAS Taverna Language and S/W tools (developed by BioInformatics community) to facilitate easy use of workflow and distributed compute technology. Already in use by ESO for its Reflex workflow system AstroGrid VO Desktop (version to be released in 2008) partly based on Taverna