1. The Texas High Energy Grid (THEGrid) A Proposal to Build a Cooperative Data and Computing Grid for High Energy Physics and Astrophysics in Texas Alan Sill, Texas Tech University Jae Yu, Univ. of Texas, Arlington Representing HiPCAT and the members of this workshop

2. Outline High Energy Physics and Astrophysics in Texas Work up to this workshop CDF, DØ, ATLAS, CMS experiments Problems A solution Implementation of the solution –DØ and CDF Grid status –ATLAS, CMS –Etc. Status Summary and Plans

3. High Energy Physics in Texas Several Universities –UT, UH, Rice, TTU, TAMU, UTA, UTB, UTEP, SMU, UTD, etc Many different research facilities used –Fermi National Accelerator Laboratory –CERN, Switzerland, DESY, Germany, and KEK, Japan –Jefferson Lab –Brookhaven National Lab –SLAC, CA and Cornell –Natural sources and underground labs Sizable community, variety of experiments and needs Very large data sets now! Even larger ones coming!!

4. High Energy Physics and Astrophysics data sets are huge –Total expected data size is over 5 PB for CDF and DØ, 10x larger for CERN experiments –Detectors are complicated, need many people to construct and make them work –Software is equally complicated –Collaboration is large and scattered all over the world Solution: Use the opportunity of having large data set in furthering grid computing technology –Allow software development and use at remote institutions –Optimize resource management, job scheduling, monitoring tools, use of resources –Efficient and transparent data delivery and sharing –Improve computational capability for education –Improve quality of life for researchers and students The Problem

5. Work up to this point HiPCAT: –What is HiPCAT? High Performance Computing Across Texas - a network and organization of computing centers and their directors at many Texas universities –Other projects (TIGRE, cooperative education, etc.) –Natural forum for this proposal –First presentation April 2003 –Many discussions since then –Led to this workshop

6. DØ and CDF at Fermilab Tevatron World’s Highest Energy proton-anti-proton collider –E cm =1.96 TeV (=6.3x10 -7 J/p  13M Joules on 10 -6 m 2 )  Equivalent to the kinetic energy of a 20t truck at a speed 80 mi/hr Chicago  Tevatron pp p CDF Dzero Currently generating data at over a petabyte per year

7. Large-scale cluster computing duplicated all over the world CDF Data Analysis Flow: CDF Level 3 Trigger (~250 duals) Production Farm (~150 duals) Central Analysis Farm (CAF) (~500 duals) User Desktops Robotic Tape Storage Data Analysis 7MHz beam Xing 0.75 Million channels 300 Hz L1 ↓ L2 ↓ 75 Hz 20 MB/s Read/write Data Reconstruction Simulation and Distributed clusters in Italy, Germany, Japan, Taiwan, Spain, Korea, several places in the US, the UK, and Canada (more coming).

8. Tevatron Current Grid Framework (JIM) UTA TTU

9. CDF-GRID: Example of a working practical grid CDF-GRID based on DCAF clusters is a de-facto working high energy physics distributed computing environment Built / developed to be clonable, Deployment led by TTU Large effort on tools usable both on- and off-site –Data access (SAM, dCache) –Remote / multi-level DB servers –Store from remote sites to tape/disk at FNAL User MC jobs at remote sites = reality now Analysis on remote data samples being developed using SAM –Up and working, already used for physics ! –Many pieces borrowed from / developed with / shared with Dzero This effort is making HEP remote analysis possible -> practical -> working -> easy for physicists to adopt

10. Basic tools Sequential Access via Metadata (SAM) –Data replication and cataloging system Batch Systems –FBSNG: Fermilab’s own batch system –Condor Three of the DØSAR farms consists of desktop machines under Condor CDF: Most central resources already based on Condor –PBS More general than FBSNG; most dedicated DØSAR farms use this manager Part of popular Rocks cluster configuration environment Grid framework: JIM = Job Inventory Management –Provide framework for grid operation  Job submission, match making and scheduling –Built upon Condor-G and Globus –MonALISA, Ganglia, user monitoring tools –Everyone has an account (with suitable controls), so everyone can submit!

11. Data Handling: Operation of a SAM Station /Consumers Producers/ Station & Cache Manager File Storage Server File Stager(s) Project Managers eworkers File Storage Clients MSS or Other Station MSS or Other Station Data flow Control Cache Disk Temp Disk

12. The tools cont’d Local Task management –CDF Grid (http://cdfkits.fnal.gov/grid/)http://cdfkits.fnal.gov/grid/ Decentralized CDF Analysis Farm = DCAF Develop code anywhere (laptop is supported) Submit to FNAL or TTU or CNAF or Taiwan or SanDiego or… Get output ~everywhere (most desktops OK) User monitoring system including Ganglia, info by queue/user per cluster –DØSAR (Dzero Southern Analysis Region) Monte Carlo Farm (McFarm) management (cloned to other institutions) DØSAR Grid: Submit requests onto a local machine and the requests gets transferred to a submission site and executed at an execution site Various Monitoring Software –Ganglia resource –McFarmGraph: MC Job status monitoring –McPerM: Farm performance monitor

13. Background Statistics on CDF Grid Data acquisition and data logging rate increased –More data = more physicists –Approved by FNAL’s Physics Advisor Committee and Director Computing needs grow, but DOE/Fnal-CD budget flat CDF proposal: do 50% of analysis work offsite –CDF-GRID: Planned at Fermilab, deployment effort led by TTU Have a plan on how to do it Have most tools in place and in use Already in deployment status at several locations throughout the world

14. Hardware resources in CDF-GRID site GHz now TB now GHz Summer TB Summer Notes INFN250595030 Priority to INFN users; Pinned data sets exist Taiwan1002.51502.5 Pinned data sets exist Korea120- - Running MC only now UCSD2805 5 Pools resources from several US groups. Min guaranteed from x2 larger farm (CDF+CMS) Rutgers10044004 In-kind, will do MC production TTU62604 2 DCAFs, test site + CDF+CMS cluster Germany GridKa ~20016~24018 Min. guaranteed CPU from x8 larger pool. Open to all by ~Dec (JIM) Canada240+- - In-kind, doing MC production, + common pool Japan--1506 Just being deployed (07/2004) Cantabria301602 ~1 months away MIT--200- ~1 month away UK--400- Open to all by ~Dec (JIM), + common pool

15. D0 Grid/Remote Computing April 2004 Joel Snow Langston University DØSAR MC Delivery Stat. (as of May 10, 2004) InstitutionInceptionN MC (TMB) x10 6 LTU6/20030.4 LU7/20032.3 OU4/20031.6 Tata, India6/20032.2 Sao Paulo, Brazil4/20040.6 UTA-HEP1/20033.6 UTA–RAC12/20038.2 D0SAR TotalAs of 5/10/0418.9

16. DØSAR Computing & Human Resources InstitutionsCPU(GHz) [future]Storage (TB)People Cinvestav131.11F+? Langston221.31F+1GA LTU25+[12]1.01F+1PD+2GA KU12??1F+1PD KSU401.21F+2GA OU19+270 (OSCER)1.8 + 120(tape)4F+3PD+2GA Sao Paulo60+[120]4.52F+Many Tata Institute521.61F+1Sys UTA430742.5F+1sys+1.5PD+3G A Total943 [1075]85.5 + 120(tape) 14.5F+2sys+6.5PD+10 GA

17. Current Texas Grid Status DØSAR-Grid –At the recent workshop in Louisiana Tech Univ. 6 clusters form a regional computational grid for MC production Simulated data production on grid in progress –Institutions are paired to bring up new sites quicker –Collaboration between DØSAR consortium and the JIM team at Fermilab begun for further software development CDF Grid –Less functionality than more ambitious HEP efforts, such as the LHC-Grid, but Works now! Already in use!! Deployment led by TTU Tuned on user’s needs Object Goal Oriented software! Based on working models and spare use of standards Costs little to get started Large amount of documents and expertise in grid computing accumulated between TTU and UTA already Comparable experience probably available at other Texas institutions

18. Actual DØ Data Re-processing at UTA

19. Network Bandwidth Needs

20. Also have Sloan Digital Sky Survey and other astrophysics work –TTU SDSS DR1 mirror copy (first in the world) –Locally hosted MySQL DB. –Image files stored on university NAS storage –Submitted proposal w/ Astronomy and CS colleagues for nationally- oriented database storage model based on local new observatory. –Virtual Observatory (VO) storage methods -- international standards under development. –Astrophysics is increasingly moving towards Grid methods.

21. Summary and Plans Significant progress has been made within Texas in implementing grid computing technologies for current and future HEP experiments UTA and TTU are playing leading roles in Tevatron grid effort for the currently running DØ and CDF as well as in LHC – ATLAS and CMS experiments All HEP experiments building operating grids for MC data production Large amount of documents and expertise exist within Texas! Already doing MC; moving toward data re-processing and analysis –Different level of complexities can be handled by emerging framework Improvements to infrastructure necessary, especially with respect to network bandwidths –THEGrid will boost the stature of Texas in HEP grid computing world –Regional plans: Started working with AMPATH, Oklahoma, Louisiana, Brazilian Consortia (tentatively named the BOLT Network) –Need Texas-based consortium to make progress in HEP and astrophysics computing

22. Summary and Plans cont’d Many shared pieces with between DØ and CDF experiment for global grid development: Provides a template for THEGrid work Near-term goals: –Involve other institutions, including those in Texas –Implement and use an analysis grid 4 years before LHC –Work in close relation but not as part of LHC-Grid (so far) –Other experiments will benefit from feedback and use cases –Lead the development of these technologies for HEP –Involve other experiments and disciplines; expand grid –Complete the THEGrid document THEGrid will provide ample opportunity to increase inter- disciplinary research and education activities