Magda Distributed Data Manager Prototype Torre Wenaus BNL September 2001

Sep 2001 Torre Wenaus, BNL 2 ATLAS PPDG Program  Principal ATLAS Particle Physics Data Grid deliverables:  Year 1: Production distributed data service deployed to users. Will exist between CERN, BNL, and at least four US grid testbed sites (ANL, LBNL, Boston U, Indiana, Michigan, Oklahoma, Arlington)  Year 2: Production distributed job management service  Year 3: Integration of all distributed services into ‘transparent’ distributed processing capability integrated into ATLAS software  This work is focused on the principal PPDG year 1 deliverable.  Enables us to participate in and benefit from grid middleware development while delivering immediately useful capability to ATLAS  Looking beyond data storage to the larger issue of data management has received little attention in ATLAS to date.

Sep 2001 Torre Wenaus, BNL 3  DBYA rapid prototyping tool for distributed data cataloging initiated 3/01 to jump start data manager development  Stable operation cataloging ATLAS files since 5/01  Globus integration: gsiftp in use, replica catalog in progress  Deployed to ANL, LBNL in addition to BNL, CERN 8/01  Good basis for developing the distributed data manager to fulfill the main ATLAS PPDG year 1 deliverable  DBYA now branching into Magda (MAnager for Grid-based DAta) to be developed as this production data manager  DBYA itself reverts to a rapid prototyping tool (sandbox)  Developers are currently T. Wenaus, W. Deng (BNL) DBYA  Magda See

Sep 2001 Torre Wenaus, BNL 4 Architecture & Schema  Partly based on NOVA distributed analysis project used in STAR  MySQL database at the core of the system  DB interaction via perl, C++, java, cgi (perl) scripts  C++ and Java APIs autogenerated off the MySQL DB schema  User interaction via web interface and unix commands  Principal components  File catalog covering arbitrary range of file types  Data repositories organized into sites and locations  Computers with repository access: a host can access a set of sites  Logical files can optionally be organized into collections  Replication, file access operations organized into tasks  To serve environments from production (DCs) to personal (laptops)

Sep 2001 Torre Wenaus, BNL 5 Architecture Diagram Location Site Location Site Location Site Host 2 Location Cache Disk Site Location Mass Store Site Source to cache stagein Source to dest transfer MySQL Synch via DB Host 1 Replication task Collection of logical files to replicate Spider scp, gsiftp Register replicas Catalog updates

Sep 2001 Torre Wenaus, BNL 6 Files and Collections  Files  Logical name is filename, without path except for CVS-based files (code) and web files for which logical name includes path within repository  Logical name plus virtual organization defines unique logical file in system  File instances include a replica number  Zero for the master instance; N=locationID for other instances  Notion of master instance is essential for cases where replication must be done off of a specific (trusted or assured current) instance  Collections  Several types supported:  Logical collections: arbitrary user-defined set of logical files  Location collections: all files at a given location  Key collections: files associated with a key or SQL query

Sep 2001 Torre Wenaus, BNL 7 Distributed Catalog  Catalog of ATLAS data at CERN, BNL  Supported data stores: CERN Castor, CERN stage, BNL HPSS (rftp service), AFS and NFS disk, code repositories, web sites  Current content: TDR data, test beam data, ntuples, code, ATLAS and US ATLAS web content, …  Some content (code, web) added more to jack up the file count to test scalability and file type diversity than because they represent useful content at present  About 300k files cataloged representing >2TB data  Has also run without problems with ~1.5M files cataloged  Running stably since May ’01  Coverage recently extended to ANL, LBNL  Other US ATLAS testbed sites should follow soon  ‘Spiders’ at all sites crawl the data stores to populate and validate catalogs  ‘MySQL accelerator’ implemented to improve catalog loading performance between CERN and BNL by >2 orders of magnitude; 2k files cataloged in 2 orders of magnitude; 2k files cataloged in <1sec

Sep 2001 Torre Wenaus, BNL 8 Data (distinct from file) Metadata  Keys  Arbitrary user-defined attributes (strings) associated with a logical file  Used to tag physics channels, histogram files, etc.  Logical file versions  Version string can be associated with logical file to distinguish updated versions of a file  Currently in use only for source code (version is the CVS version number of the file)  Data signature, object cataloging  Coming R&D; nothing yet

Sep 2001 Torre Wenaus, BNL 9 File Replication  Supports multiple replication tools as needed and available  Automated CERN-BNL replication incorporated 7/01  CERN stage  cache  scp  cache  BNL HPSS  stagein, transfer, archive scripts coordinated via database  Transfers user-defined collections keyed by (e.g.) physics channel  Just extended to US ATLAS testbed using Globus gsiftp  Currently supported testbed sites are ANL, LBNL, Boston U  BNL HPSS  cache  gsiftp  testbed disk  BNL or testbed disk  gsiftp  testbed disk  gsiftp not usable to CERN; no available grid node there (til ’02?!)  Will try GridFTP, probably also GDMP (flat files) when available

Sep 2001 Torre Wenaus, BNL 10 Data Access Services  Command line tools usable in production jobs: under test  getfile  Retrieve file via catalog lookup and (as necessary) staging or (still to come) remote replication  Local soft link to cataloged file instance in a cache or location  Usage count maintained in catalog to manage deletion  releasefile  Removes local soft link, decrements usage count in catalog, deletes instance (optionally) if usage count goes to zero  Callable APIs for catalog usage and update: to come  Collaboration with David Malon on Athena integration

Sep 2001 Torre Wenaus, BNL 11 Near Term Schedule  Complete and deploy data access services (2 wks)  Globus integration and feedback  Remote command execution (2 wks)  Test Globus replica catalog integration (1 mo)  Incorporate into Saul Youssef’s pacman package manager (1 mo)  ATLAS framework (Athena) integration (with D.Malon) (2 mo)  Look at GDMP for replication (requires flat file version) (2 mo)  Application and testing in coming DCs, at least in US  Offered to intl ATLAS (Gilbert, Norman) as a DC data management tool  Design and development of metadata and data signature (6 mo)  Study scalability of single-DB version, and investigate scalability via multiple databases (9 mo)