Glast Collaboration Data Server and Data Catalog Tony Johnson DC2 Planning Meeting June 2005

Contents What Exists Ntuple Pruner/Peeler Data Server (for Internal Collaboration Use) What is Planned What is Wanted?

Data Server Portal Web Portal Provides access to existing data server functionality Currently: NTuple Pruner (Tom Glanzman) Selection of Data via Cuts on Merit Tuple Works with datasets in pipeline data catalog Download of Data via FTP after submission of batch job Allows access to Root Merit Tuple Event Peeler (coming very soon) (Tom Glanzman) Selection of Data via run/event number (uploaded file) Access to Root Merit tuple and/or full Root tuple “Data Server” (Jean-Paul LeFevre) Allows rapid selection of events based on Energy, Origin (decl, ra), Time, Gamma Quality Stored in “meta-data” database Additional MeritTuple cuts Supports adding cuts to personal “favorites” list Currently configured to work with DC1 Root merit tuple only

Screen Shots http://glast-ground.slac.stanford.edu/DataServer/

Data Server Issues Currently trying both Oracle (10g) and MySQL (5) using “spatial” extensions Do not fully support spherical geometry Forced to make rectangular selections rather than circular Problems at poles Performance seems OK – at least for 50 million events Selection performance scales by number of events selected, rather than total events in database Indexes seem very slow to build many hours to add 1,000,000 events – and this seems to scale by total database size Still under investigation, maybe can be improved by tuning Need to decide very soon how much effort to put into database vs. a custom solution

Data Catalog Plans Working on new “Glast Data Catalog” Less tightly coupled to pipeline than current catalog Allows domain specific, user-defined, hierarchical “meta-data” to be associated with each dataset, e.g. Simulation physics, test setup parameters Pointers to pipeline task Pointers to e-logbook entries Web interface will allow browsing data hierarchy or searching based on meta-data Implementation based on earlier “Grid” data catalog developed at SLAC. Uses XML for import/export of data (stored in Oracle XML database)

DC2 Data Catalog?

Data Server Plans Continue to enhance pruner/peeler Add TCut capability to peeler Add access to other data types (SVAC tuple) Data Server Enhance ergonomics of web interface Support search using sky catalog Work on handling larger data volumes Add ability to download events in different formats FITS, run/event #, Different Root tuples Add ability to browse events using event display Use xrootd server to stream data Eliminate waiting for batch job and FTP transfer Experiment with SLAC “Peta-Cache” system Initially use xrootd to serve existing Root tuples Highest performance may require storing tuples in some other format

Data Pump – Streaming data directly to users Data Server TCut Format Converter TCut Format Converter TCut Format Converter Multiple Threads xrootd Root Files

Conclusions Initial data server available Would like some people to try it and give feedback Lots of work to do Need to set goals/priorities for DC2 work Understand timescales Understand what data volume will be Understand what typical queries will be

Hierarchical Data Catalog