Dzero Data Handling and Databases Lee Lueking Dzero Computing Review May 9, 2002

outline Overview Data Handling Software Mass Storage System Databases Conclusions 11/30/2018 Lee Lueking - Dzero

Dzero Data Handling and Processing Architecture Reconstruction Farm Station Enstore Mover Nodes Central Analysis Station STK Silo High Capacity Switch CLuED0 Analysis Cluster Station STK Silo ADIC Tape Robot Of-Site Analysis and Processing Stations Online Data Logger Misc. Analysis Stations Misc. Analysis Stations Misc. Analysis Stations 11/30/2018 Lee Lueking - Dzero

The Anatomy of The D0 Data Handling, DataBase, Data Grid Core Lead: Lee Lueking Maintenance Production operation User support for D0 experiment Add needed D0 features Coordinate remote analysis sites Shift help from D0 experiment Physics Analysis Tools Lead:Wyatt Merritt Luminosity Physics Data Streams Interfaces for D0 framework GRID Lead:Igor Terekhov Evaluate and implement standard Grid components Create and collect use cases Plan and Design Job control, Monitoring, Info system, etc. Coordinate D0GRID and PPDG efforts Database Lead:Ruth Pordes Support and coordination for D0 database applications Includes SAM core, Analysis tools and GRID support 11/30/2018 Lee Lueking - Dzero

Data Handling Software

Components of a SAM Station Producers/ /Consumers Project Managers Temp Disk Cache Disk MSS or Other Station MSS or Other Station File Storage Server Station & Cache Manager File Storage Clients File Stager(s) Data flow Control eworkers 11/30/2018 Lee Lueking - Dzero

SAM as a Distributed System Name Server Database Server(s) (Central Database) Global Resource Manager(s) Log server Shared Globally Station 1 Servers Station 3 Servers Local To Site Station n Servers Station 2 Servers Mass Storage System(s) Arrows indicate Control and data flow Shared Locally 11/30/2018 Lee Lueking - Dzero

Data to and from Remote Sites Station Configuration Replica location Prefer Avoid Forwarding File stores can be forwarded through other stations “Routing” (now) Parasitic Stagers on D0mino Working for Imperial, Lancaster, BU, Arizona, Wuppertal, Columbia SAM Station 1 SAM Station 2 SAM Station 6 Remote SAM Station MSS SAM Station 3 SAM Station 5 SAM Station 4 Extra-domain transfers Currently use bbftp (parallel transfer protocol) 11/30/2018 Lee Lueking - Dzero

Dzero SAM Deployment Map Processing Center Analysis site 11/30/2018 Lee Lueking - Dzero

Future SAM and D0 Grid The station cache manager and File storage Server components will be merged to improve functionality of cache management and streamline routing behavior. Transition toward less centralized operation in which stations have site autonomy. The station can loose contact with the central db server and still function. Distributed naming service too. There will be a natural progression to “standard grid middleware” components. Current Grid work involves understanding job scheduling components with Condor-G, and the use of Globus tools including Grid Security Infrastructure (GSI), Grid Resource Allocation and Management (GRAM), Meta Directory Service (MDS), and GridFTP. Grid work done in conjunction with Particle Physics Data Grid (PPDG) collaboration. 11/30/2018 Lee Lueking - Dzero

SAM as Part of The Grid Monitoring and Job Management Information Service Job Management JDL Job Request Broker Condor MMS Resource Info MDS Condor Class Ads Job Submission Condor-G Site Gatekeeper Data Handling GRAM DH Resource Manager Local batch system SAM Condor 11/30/2018 Lee Lueking - Dzero

Data Handling, Data Grid, Analysis Tools Manpower Work Area FTE Description SAM Core 4 Maintenance and improvements Operations 2-3 Ongoing operations (incl d0 SAM shifters) Data Grid 3-4 Research and development Analysis Tools 3 Design and development 11/30/2018 Lee Lueking - Dzero

Mass Storage System

Enstore System and ISD Enstore is the principal MSS used with SAM. Integrated Systems Dept. (ISD) responsible for maintenance and upgrades to the system. ISD also responsible for FNAL MSS hardware including robotics, drives and tapes. Security enhancements are planned both at the admin and file transfer levels. Modernize code as needs and practices evolve. Continue to investigate replacing tape with disk as the technology becomes cost effective and maintainable. 11/30/2018 Lee Lueking - Dzero

dCache ISD is working with DESY to provide a disk cache buffering system called dCache. The system “front-ends” the current tape systems and is consistent with our computing model. Dzero will benefit from this in a couple of obvious ways: An interface to standard file transfer protocol (like GridFTP) will reduce the need to rout data through FNAL stations to remote sites. The intermediate cache will reduce access to tape for “hot” data coming from online and/or reco farms. For small cost, should represent significant performance improvements. 11/30/2018 Lee Lueking - Dzero

Tape Robot Enstore Data Rates 450 GB/day 1.6 TB/day ADIC Robot/LTO Drives STK Robot/ 9940 Drives 11/30/2018 Lee Lueking - Dzero

Summary of Tape Technologies Type Number written Bad tapes Total stored Comment 9940 795 1 (file) 43 TB Problem files recovered LTO 198 17 TB m2 567 dozens 26 TB Retired m1 1050 14 TB 11/30/2018 Lee Lueking - Dzero

Data Stored in SAM last 12 Months Number of Files 450,000 Data Size 100 TB Number of Events 400M (~120M RAW) 11/30/2018 Lee Lueking - Dzero

Storage Roadmap Current tape costs are about $1/GB and Commodity off the Shelf (COTS) IDE disk about $2/GB. Price trends indicate that tape will drop in price/GB by a factor of 2 every two years. COTS disk prices will reduce by a factor of 2 every year. Operation and deployment costs may be high for large storage facilities. Even if COTS disk is not the primary location for data, there will be very large disk caches. We will continue to use/buy STK or ADIC robots as needed. Older data will be shelved or copied to newer more dense media. 11/30/2018 Lee Lueking - Dzero

Cartridge Size and Media Cost   2003 GB $/GB 2005 2007 GB $/GB 2009 STK 120 0.65 250 0.30 500 0.15 1000 0.07 LTO 200 0.50 400 0.25 800 0.12 1600 0.06 Disk (COTS) 200 1.00 800 0.25 3200 0.06 12800 0.015 11/30/2018 Lee Lueking - Dzero

Drive Rates and Cost 2003 MB/s cost 2005 2007 2009 STK 20 $40k 40 $40k   2003 MB/s cost 2005 2007 2009 STK 20 $40k 40 $40k 80 $40k 160 $40k LTO 20 $10k 40 $10k 80 $10k 160 $10k Disk (COTS) 10 $200 40 $200 160 $200 640 $200 11/30/2018 Lee Lueking - Dzero

Robot Capacities Tape Slots/unit Drive Slots/unit Mounts /hour k$/unit   Tape Slots/unit Drive Slots/unit Mounts /hour k$/unit STK-Powderhorn 5500 20 N/A 75 ADIC- AML/2 3500 150 300   2003 TB MB/s 2005 2007 2009 STK 660 400 1320 800 2640 1600 5280 3200 ADIC AML/2 700 400 1400 800 2800 1600 5600 3200 11/30/2018 Lee Lueking - Dzero

Database

Database Approach We rely on a central database model with Highly Available Oracle server located at FNAL serving all offline onsite and offsite needs. An 8 processor SUN 4500 is employed with 1.2 TB RAID disk array. This has been very reliable. A three tier software architecture is employed with the middle tier called the “DB server”. 11/30/2018 Lee Lueking - Dzero

Database Applications Name Responsible Person Status & Plans Calibration Taka Yasuda About ½ the applications are in production. Taka is coordinating with the individual application developers on delivery, testing etc. We expect work here as more data validation is done, and as the access profile increases. Offline Luminosity and Streams Michael Begal, Jeremy Simmons, Greg Landsberg, (Analysis Tools Group) The offline luminosity application is not written. Streaming is not started, neither is the application to access the streaming information. This is a fairly significant project that needs the experiment requirements and design work to be completed. L1,L2,L3 Trigger Elizabeth Gallas Development will likely continue for the next 9 months. Reporting and physicist friendly interfaces will be needed after that time. It can be expected that a further round of work is needed for the Run IIB upgrades 11/30/2018 Lee Lueking - Dzero

Database Applications (cont.) Name Responsible Person Status & Plans SAM File and Event Catalog Lee Lueking General maintenance and updates Run Summary and Configuration Vladimir Sirotenko Jeremy Simmons An upgrade to this application is currently underway in support of data quality and validation. It is anticipated it will take another few months to complete VPC Calibration Volker Buescher This is now the responsibility of the online database group. RCP Steve White Marc Paterno The database side of this project is in abeyance. It is anticipated that no further work will be needed here. Speakers Bureau Elizabeth Gallas Completed Release Request Harry Melanson Completed. May need some tweaking as release procedures change 11/30/2018 Lee Lueking - Dzero

Database Application Storage Requirements Application Name Estimated Size 2 Years RIIa Offline Calibration Top Level 40 MB Offline Calibration 90 GB Offline Muon 30 GB Offline CFT 14 GB Offline CPS 2 GB Offline FPS 8 GB Offline FPD Unknown Offline Luminosity and Streams 200 GB L1, L2, L3 Trigger SAM File and Event 700 GB Speakers Bureau 800 MB VLPC Calibration 7 GB Run Configuration 105 GB Total 1.15 TB 11/30/2018 Lee Lueking - Dzero

Dzero DB Server Client Client Client CORBA DB Server Python SQLnet The DB server represents the middle tier of a three tier architecture. Reference material available at http://d0db-dev.fnal.gov/db_server_gen Client Client Client CORBA DB Server Python SQLnet Database Oracle 11/30/2018 Lee Lueking - Dzero

What Are Pros and Cons of 3 Tier Architecture? Advantages Significantly reduces the required number of concurrent client licenses required (significant savings when using commercial db). Enables any internet connected node to run client processes. Insulates client from changes in DB schema, and visa versa. Enables client applications to be coded in any language. Much more scalable approach: 1) Allows customized caching and other services to be built into middle tier, 2) Dbservers can be operated on “cheap” hardware. Disadvantages Complex environment Debugging at all tiers more complicated. Adds additional level of abstraction for developers and users. Reduces effectiveness of Oracle server caching and optimization 11/30/2018 Lee Lueking - Dzero

Where Do We Use DB Serves? SAM specific servers SAM user (serves SAM users) SAM prd (serves most SAM stations) SAM web (dedicated to web-based dataset editor) DLSAM (dedicated to online station) Central-Analysis (dedicated to CA station) D0 Farm (dedicated to D0 FNAL processing farm) Offline Trigger DB Offline Luminosity Generalized servers Offline Calibration (many) Offline Run Config 11/30/2018 Lee Lueking - Dzero

TNG DB Server Project This fully functional DB server is extremely True multi-threading supports multiple simultaneous clients. Number of active logins to database is configurable (database pool), drops idle connections. Memory (L1) and persistent (L2) server side object caching. Additional monitoring. Dynamic configuration. Simplification of D0om interface, pass data only (no CORBA objects). Provide user identification and tracking. More object oriented. More easily maintainable. Proxy feature will allow servers to be connected together to provide maximum reliability and scalability of the overall system. Jim Kowalkowski, and Steve White have done full evaluation and design. Now working on implementation. Expect to have first working version May 31 (no L2 cache or Proxy). All features in July. This fully functional DB server is extremely important to maintain our Central DB Model ! 11/30/2018 Lee Lueking - Dzero

Database Hardware and Software Ongoing Oracle licensing and support Oracle Layered products; OEM, Designer, etc. Hardware Servers – Load on DB server machines will increase as RIIa picks up. Probably require ~$60k/yr upgrades. Replace servers at beginning of RIIb for ~$300k, including software. Disk – Estimate growth as high as 1 TB/yr in RIIb. Probably ~$50k/yr. Backup – Included in server upgrade. 11/30/2018 Lee Lueking - Dzero

Database Manpower Task Effort Comments Database administration 1.5 FTEs Increase to 2-3 FTEs for several months for significant upgrade in infrastructure Application infrastructure 1 FTEs 2-3 FTEs til end of 2002, shared with SAM and remote access projects. Monitoring of the information .2FTEs   11/30/2018 Lee Lueking - Dzero

Conclusion The Dzero data handling system is used heavily, and has been reliable. The distributed data handling model is is proven sound and we expect to scale to meet most RIIa needs. SAM is being used to manage several kinds of compute hardware for various processing needs including processing farms and analysis clusters. This will continue and be expanded in RIIb. The SAM DH system will become part of an overall computing grid that is being developed and will enable easy and efficient use of compute resources around the world. This system is being built with “standard Grid Middleware” components for universal compatibality with other experiments. The current tape technologies are working well and should be adequate for RIIa and evolve to meet the needs of Run IIb We will continue to employ Oracle for critical and High Availability Database applications. The Database server technology we have developed is a key component to making this a scalable and robust solution for a system with worldwide deployment. 11/30/2018 Lee Lueking - Dzero