U.S. ATLAS Computing Facilities DOE/NFS Review of US LHC Software & Computing Projects Bruce G. Gibbard, BNL January 2000

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 2 US ATLAS Computing Facilities Facilities procured, installed and operated –…to meet U.S. ‘MOU’ Obligations Direct IT responsibility (Monte Carlo, for example) Support for detector construction, testing, & calib. Support for software development and testing –…to enable effective participation by US physicists in ATLAS physics program ! Direct access to and analysis of physics data sets Support simulation, re-reconstruction, and reorganization of data associated with that analysis

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 3 Setting the Scale Uncertainties in Defining Facilities Scale –Five years of detector, algorithm & software development –Five years of computer technology evolution Start from ATLAS Estimate & Regional Center Guidelines Adjust for US ATLAS perspective (experience, priorities and facilities model)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 4 ATLAS Estimate & Guidelines Tier 1 Center in ‘05 should include... –30,000 SPECint95 for Analysis –20,000 SPECint95 for Simulation –100 TBytes/year of On-line (Disk) Storage –200 TBytes/year of Near-line (Robotic Tape) Storage –100 Mbit/sec connectivity to CERN Assume no major raw data processing or handling outside of CERN

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 5 US ATLAS Perspective US ATLAS facilities must be adequate to meet any reasonable U.S. ATLAS computing needs ( U.S. role in ATLAS should not be constrained by a computing shortfall, rather the U.S. role should be enhanced by computing strength ) There must be significant capacity beyond that formally committed to International ATLAS which can be allocated at the discretion of U.S. ATLAS

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 6 Facilities Architecture Consists of Transparent Hierarchically Distributed Computing Resources Connected into a GRID –Primary ATLAS Computing Centre at CERN –US ATLAS Tier 1 Computing Center at BNL National in scope at ~20% of CERN –US ATLAS Tier 2 Computing Centers Six, each regional in scope at ~20% of Tier 1 Likely one of them at CERN –US ATLAS Institutional Computing Facilities Institutional in scope, not project supported –US ATLAS Individual Desk Top Systems

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 7 Schematic of Model

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 8 Distributed Model Rationale (benefits) –Improved user access to computing resources Higher performance regional networks Local geographic travel –Enable local autonomy Less widely shared resources More locally managed –Increased capacities Encourage integration of other equipment & expertise –Institutional, base program Additional funding options –Com Sci, NSF

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 9 Distributed Model (2) But increase vulnerability (Risk) –Increased dependence on network –Increased dependence on GRID infrastructure software and hence R&D efforts –Increased dependence on facility modeling tools –More complex management Risk / benefit analysis must yield positive result

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 10 Adjusted For U.S. ATLAS Perspective Total US ATLAS facilities in ‘05 should include... –10,000 SPECint95 for Re-reconstruction –85,000 SPECint95 for Analysis –35,000 SPECint95 for Simulation –190 TBytes/year of On-line (Disk) Storage –300 TBytes/year of Near-line (Robotic Tape) Storage –Dedicated OC Mbit/sec Tier 1 connectivity to each Tier 2 –Dedicated OC Mbit/sec to CERN 

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 11 GRID Infrastructure GRID infrastructure software must supply –Efficiency (optimizing hardware use) –Transparency (optimizing user effectiveness) Projects –PPDG : Distributed data services - Common Day talk by D. Malon –APOGEE: Complete GRID infrastructure including: distributed resources management, modeling, instrumentation, etc. –GriPhyN: Staged development toward delivery of a production system Alternative to success with these projects is a cumbersome to use and/or reduce efficiency overall set of facilities U.S. ATLAS involvement includes - ANL, BNL, LBNL

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 12 Facility Modeling Performance of Complex Distribute System is Difficult but Necessary to Predict MONARC - LHC centered project –Provide toolset for modeling such systems –Develop guidelines for designing such systems –Currently capable of relevant analyses –Common Day talk by K. Sliwa

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 13 Technology Trends CPU –Range: Commodity processors -> SMP servers –Factor 2 decrease in price/performance in 1.5 years Disk –Range: Commodity disk -> RAID disk –Factor 2 decrease in price/performance in 1.5 years Tape Storage –Range: Desktop storage -> High-end storage –Factor 2 decrease in price/performance in years

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 14 Technology Trends & Choices For Costing Purpose –Start with familiar established technologies –Project by observed exponential slopes Conservative Approach –There are no known near term show stoppers to evolution of these established technologies –A new technology would have to be more cost effective to supplant projection of an established technology

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 15 Technology Choices CPU Intensive processing –Farms of commodity processors - Intel/Linux I/O Intensive Processing and Serving –Mid-scale SMP’s (SUN, IBM, etc.) Online Storage (Disk) –Fibre Channel Connected RAID Nearline Storage (Robotic Tape System) –STK / 9840 / HPSS LAN –Gigabit Ethernet

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 16 Requirements Profile Facilities Ramp-up Driven by… –Core software needs ODBMS scalability tests in ‘01-’02 time frame –Subdetector needs Modest for next few years –Mock Data Exercises - not officially schedule so… Assume MDC I at 10% scale in 2003 and MDC II at 30% scale in 2004 –Facilities model validation

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 17 Tier 1 Full Function Facility Including... –Dedicated Connectivity to CERN –Primary Site for Storage/Serving Cache/Replicate CERN & other data needed by US ATLAS –Computation Primary Site for Re-reconstruction (perhaps only site) Major Site for Simulation & Analysis (~2 x Tier 2) –Regional support plus catchall for those without a region –Repository of Technical Expertise and Support Hardware, OS’s, utilities, other standard elements of U.S. ATLAS Network, AFS, GRID, & other infrastructure elements of WAN model

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 18 Tier 1 (2) Commodity processor farms (Intel/Linux) Mid-scale SMP servers (SUN) Fibre Channel connected RAID disk Robotic tape / HSM system (STK / HPSS)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 19 Current Tier 1 Status U.S. ATLAS Tier 1 facility now operating as a ~5 % adjunct to the RHIC Computing Facility including –Intel/Linux farms (28 CPU’s) –Sun E450 server (2 CPU’s) –200 GBytes of Fibre Channel RAID Disk –Intel/Linux web server –Archiving via low priority HPSS Class of Service –Shared use of an AFS server (10 GBytes)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 20

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 21 RAID Disk Subsystem

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 22 Intel/Linux Processor Farm

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 23 Intel/Linux Nodes

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 24 Tier 1 Staffing Estimate

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 25 Tier 2 Ramp-up Assume 2 years for Tier 2 to fully establish –Initiate first Tier 2 in 2001 True Tier 2 prototype Demonstrate Tier 1 - Tier 2 interaction –Second Tier 2 initiated in 2002 (CERN?) –Four remaining initiated in 2003 All fully operational by 2005 Six are to be identical (CERN exception?)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 26 Tier 2 Limit Personnel and Maintenance Support Costs Focused Function Facility –Excellent connectivity to Tier 1 (Network + GRID) –Tertiary storage via Network at Tier 1 (none local) –Primary Analysis site for its region –Major Simulation capabilities –Major online storage cache for its region Leverage Local Expertise and Other Resources –Part of site selection criteria – For example: ~1 FTE contributed,

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 27 Tier 2 (2) Commodity processor farms (Intel/Linux) Mid-scale SMP servers Fibre Channel connected RAID disk

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 28 Tier 1 / Tier 2 Staffing (In Pseudo Detail)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 29 Staff Evolution

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 30 Network Tier 1 Connectivity to CERN and to Tier 2’s is Critical to Facilities Model –Must be adequate –Must be guaranteed and allocable (dedicated and differentiate) –Should grow with need; OC12 should be practical by 2005 –While estimate is highly uncertain this cost must be covered in a distributed facilities plan

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 31 WAN Configurations and Cost (FY 2000 k$)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 32 Capacities by Year

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 33 Annual Equipment Costs at Tier 1 Center (FY 2000 k$)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 34 Annual Equipment Costs at Tier 2 Center (FY 2000 k$)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 35 US ATLAS Facilities Annual Costs (FY2000 k$)

19 January, 2000 DOE/NSF Review of US LHC Software & Computing Projects B. Gibbard 36 Major Milestones