1. Web Based Data Quality Monitoring for the Fermi Gamma-Ray Space Telescope Tony Johnson tonyj@slac.stanford.edu

2. T. JohnsonFERMI Data Quality Monitoring2/19CHEP 2010, Taipei, Taiwan Launched 11 June 2008 – LAT activated 25 June

3. T. JohnsonFERMI Data Quality Monitoring3/19CHEP 2010, Taipei, Taiwan

4. T. JohnsonFERMI Data Quality Monitoring4/19CHEP 2010, Taipei, Taiwan In Orbit: Single Events in the LAT The green crosses show the detected positions of the charged particles, the blue lines show the reconstructed track trajectories, and the yellow line shows the candidate gamma-ray estimated direction. The red crosses show the detected energy depositions in the calorimeter.

5. T. JohnsonFERMI Data Quality Monitoring5/19CHEP 2010, Taipei, Taiwan Fermi One Year All Sky Map

6. T. JohnsonFERMI Data Quality Monitoring6/19CHEP 2010, Taipei, Taiwan GN HEASARC - - DELTA 7920H White Sands TDRSS SN S & Ku LAT Instrument Science Operations Center (SLAC) GBM Instrument Operations Center GRB Coordinates Network Telemetry 1 kbps - S Alerts Data, Command Loads Schedules Mission Operations Center (MOC) Fermi Science Support Center  sec Fermi Spacecraft Large Area Telescope & GBM GPS Fermi MISSION ELEMENTS

7. T. JohnsonFERMI Data Quality Monitoring7/19CHEP 2010, Taipei, Taiwan Data Processing Flow at SLAC Downlink from Goddard Space Flight Center –~8 downloads per day –15 GB total daily Level 0 Processing –Automatically launched as data arrives –Decode & repackage incoming data Split science data from telemetry data Level 1 Processing –Full event reconstruction: 750 GB/day –Data Quality Monitoring –Transfer science summary files to Goddard Science Support Center - 200 MB/day Immediately available to the public ASP (Automated Science Processing) –GRB and Flare detection –Spectral analysis 120,000 quantities to be monitored –Mixture of Oracle, Root, Fits data Level 0 Level 1 ASP Trending Database (Oracle) Data Quality (Root) ASP Results (Fits)

8. T. JohnsonFERMI Data Quality Monitoring8/19CHEP 2010, Taipei, Taiwan Level 1 Processing Task Example Reconstruction Digitization ~2000 batch jobs every 3 hours – 800 simultaneous cores All batch automatically submitted using “Data Pipeline” Very low rate of manual intervention required (<.01% of jobs) Data verified and available to public (much) less than 24 hours after acquisition

9. T. JohnsonFERMI Data Quality Monitoring9/19CHEP 2010, Taipei, Taiwan ISOC Control Room “Duty Scientists” monitoring data quality daily All of the data processing and data quality monitoring can be done from the web

10. T. JohnsonFERMI Data Quality Monitoring10/19CHEP 2010, Taipei, Taiwan Technologies Used Web: –Many independent web applications Allows independent development Shared application framework provides –authentication, authorization »CAS single sign-on –page decoration, sitewide menus –Database utilities –Apache/Tomcat servers Multiple servers for redundancy Monitored using Nagios, JMX –Java Server Pages (JSP) Open Source and Custom tag libraries simplify development –DisplayTag for tabular data »Sorting, filtering, pagination –JAIDA tag library for plotting »Images generated dynamically on server Data Access Tools –Oracle, Partitioning –Java Fits Library –FreeHEP Root IO library

11. T. JohnsonFERMI Data Quality Monitoring11/19CHEP 2010, Taipei, Taiwan Monitoring Data Processing Web interface allows –Quick overview of data processing –Flags runs requiring further attention –Allows “drill-down” to isolate/identify problems

12. T. JohnsonFERMI Data Quality Monitoring12/19CHEP 2010, Taipei, Taiwan Processing Pipeline Web Interface Pipeline web interface allows –Many views of data processing, down to log files of individual jobs –If jobs do fail they can be “rolled back” directly from the web interface

13. T. JohnsonFERMI Data Quality Monitoring13/19CHEP 2010, Taipei, Taiwan Data Quality Monitoring Web interface allows –Show data from single run or aggregate set of runs –View description of each plot –View/Print multiple plots –Customized tree to draw attention to important plots Can be customized for individuals or groups

14. T. JohnsonFERMI Data Quality Monitoring14/19CHEP 2010, Taipei, Taiwan Telemetry Trending Web interface allows –Dynamic selection of time period –Dynamic overlay of quantities –Customized tree to draw attention to important plots Can be customized for individuals or groups Cross trending of housekeeping and level 1 data

15. T. JohnsonFERMI Data Quality Monitoring15/19CHEP 2010, Taipei, Taiwan Automated Alarms Automated alarms are used to alert duty scientists to anomalies Use fixed limits and reference histograms Many quantities are highly orbit dependent, so particle fluxes, geomagnetic variables must be taken into account –20 different alarm algorithms Error limit Warning limit

16. T. JohnsonFERMI Data Quality Monitoring16/19CHEP 2010, Taipei, Taiwan Data Quality Monitoring The trending graphs below show some rate summary plots for the 24 hours around GRB. Strong correlation with orbital period (~90 minutes) can clearly be seen This burst was so bright that it can be seen even in the global rate plots.

17. T. JohnsonFERMI Data Quality Monitoring17/19CHEP 2010, Taipei, Taiwan Automated Science Processing Used to rapidly detect Gamma Ray Bursts or other flaring events Enabled timely notification of interesting events to external astrophysical community

18. T. JohnsonFERMI Data Quality Monitoring18/19CHEP 2010, Taipei, Taiwan Technology reuse New Scientific Computing Applications (SCA) group formed at SLAC –Part of Particle and Particle Astrophysics (PPA) Division –Aims to encourage reuse of software between experiments Much of the Fermi monitoring web framework being reused for Enriched Xenon Observatory (EXO) experiment Starting use for Large Synoptic Survey Telescope (LSST) Others… Other related talks at CHEP 2010 –21 October: 14:30~16:00 parallel 47 -- Building Interactive Web Applications for HEP Using the Google Web Toolkit (GWT) –21 October: 16:30~18:00 parallel 48 -- Fermi Gamma-Ray Space Telescope Processing Pipeline and Data Catalog

19. T. JohnsonFERMI Data Quality Monitoring19/19CHEP 2010, Taipei, Taiwan Conclusions The Fermi Data Quality Monitoring system has now been used successfully for >2 years –Much of it is publically viewable at: http://glast-ground.slac.stanford.edu/ Acknowledgments –The LAT Data Quality Monitoring system was designed and implemented by: Anders Borgland, Eric Charles, Warren Focke, Martin Kocian, Maria Elena Monzani, David Paneque, Massimiliano Turri (SLAC), Luca Baldini, Johan Bregeon, Melissa Pesce- Rollins and Carmelo Sgrò (INFN and University of Pisa). –Maria Elena Monzani provided many of the plots used in this presentation.