1. U.S. ATLAS Grid Testbed Status & Plans Kaushik De University of Texas at Arlington PCAP Review LBNL, November 2002

2. November 14, 2002 Kaushik De PCAP Review 2 Outline  Testbed introduction  Recent accomplishments  Software distribution  Application toolkit  Monte Carlo production  Monitoring  Integration issues  Since summer 2002, U.S. Grid Testbed has concentrated on rapid software distribution, grid-based Monte Carlo production, and SC2002 demonstrations  This talk will focus primarily on the grid- based production experience

3. November 14, 2002 Kaushik De PCAP Review 3 Testbed Goals  Demonstrate success of grid computing model for High Energy Physics  in data production  in data access  in data analysis  Develop, deploy and test grid middleware and applications  integrate middleware with applications  simplify deployment - robust, rapid & scalable  inter-operate with other testbeds & grid organizations (iVDGL, DataTag…)  provide single point-of-service for grid users  Evolve into fully functioning scalable distributed tiered grid

4. November 14, 2002 Kaushik De PCAP Review 4 Lawrence Berkeley National Laboratory Brookhaven National Laboratory Indiana University Boston University Argonne National Laboratory U Michigan University of Texas at Arlington Oklahoma University US -ATLAS testbed launched February 2001 Two new sites joining soon - UNM, SMU Grid Testbed Sites

5. November 14, 2002 Kaushik De PCAP Review 5 Testbed Fabric  >8 production gatekeepers at ANL, BNL, LBNL, BU, IU, UM, OU, UTA  http://heppc1.uta.edu/atlas/grid-testbed/testbed-sites.htm http://heppc1.uta.edu/atlas/grid-testbed/testbed-sites.htm  Large clusters at BNL, LBNL, IU, UTA, BU  Heterogeneous system - Condor, LSF, PBS  Currently > 100 nodes available  Could double capacity quickly, if needed  + Multiple R&D gatekeepers  gremlin@bnl - iVDGL GIIS  heppc5@uta - ATLAS hierarchical GIIS  atlas10/14@anl - EDG testing  heppc6@uta+gremlin@bnl - glue schema  heppc17/19@uta - GRAT development  few sites - Grappa portal  bnl - VO server  few sites - iVDGL testbed

6. November 14, 2002 Kaushik De PCAP Review 6 Testbed Tools  Many tools developed by the U.S. ATLAS testbed group during past 2 years   GridView - simple tool to monitor status of testbed Kaushik De, Patrick McGuigan   Gripe - unified user accounts Rob Gardner   Magda - MAnager for Grid DAta Torre Wenaus, Wensheng Deng (see Gardner & Wenaus talks)   Pacman - package management and distribution tool Saul Youssef   Grappa - web portal using active notebook technology Shava Smallen, Dan Engh   GRAT - GRid Application Toolkit   Gridsearcher - MDS browser Jennifer Schopf   GridExpert - Knowledge Database Mark Sosebee   VO Toolkit - Site AA Rich Baker   This talk will not describe all tools - only some examples of tools used for production

7. November 14, 2002 Kaushik De PCAP Review 7 Recent Accomplishments  May 2002  Globus 2.0 beta RPM developed at BNL  Athena-atlfast grid package developed at UTA  Software installation - Pacman developed at BU  GRAT toolkit for job submission on grid, developed at UTA & OU  June 2002  Tested interoperability - successfully ran ATLAS MC jobs on CMS & D0 grid sites  ANL demonstrated that U.S. testbed package can run successfully at EDG sites  July 2002  New production software released & deployed  2 week Athena-atlfast MC production run using GRAT & GRAPPA  Generated 10 million events, thousand files catalogued in Magda, all sites participating

8. November 14, 2002 Kaushik De PCAP Review 8 Accomplishments contd.  August/September 2002  New production software package deployed  3 week dc1 production run using GRAT  Generated 200,000 events, using ~ 30,000 CPU hours, 2000 files, 100 GB storage  Primary files stored at BNL HPSS  Replicas stored at grid testbed sites  Deployed VO server at BNL Tier I facility  October/November 2002  Prepare demos for SC2002  Deploy new VDT 1.1.5 on testbed  Deploy iVDGL packages Worldgrid, Sciencegrid  Interoperability tests with EDG  Prepare software for DC1 Phase 2

9. November 14, 2002 Kaushik De PCAP Review 9 Software Distribution  Goals:  Easy installation  Uniform software versions  Pacman perfect for this task  Deployment in 3 steps  VDT Package (root)  Pacman, Globus 2.0b, Condor, Pippy...  ATLAS Package (non-root)  Magda, Grappa, JDK, MySQL, Perl…  User Package  GRAT, Grappa  Experience  Repeated process 3 times during summer 2002  Worked well!  Deployed successfully at all testbed sites

10. November 14, 2002 Kaushik De PCAP Review 10 my_environment.pacman Pacman  Pacman - package manager developed at Boston University by Saul Youssef  Pacman lets you define how a mixed tarball/rpm/gpt/native environment is  Fetched  Installed  Setup  Updated  Experts can figure this out once and export to the rest of the world via trusted caches.  % pacman –get my_environment  Package examples: VDT, atlas, atlas_user, worldgrid, sciencegrid, dc1package  Pacmanization of CMS underway

11. November 14, 2002 Kaushik De PCAP Review 11 Installation Web Page

12. November 14, 2002 Kaushik De PCAP Review 12 Installation Status  Installation information also available through Pippy using Globus MDS

13. November 14, 2002 Kaushik De PCAP Review 13 User Toolkits

14. November 14, 2002 Kaushik De PCAP Review 14 Testbed Production  Goals:  Demonstrate distributed ATLAS data production, access and analysis using grid middleware and tools developed by the testbed group  Plans:  Atlfast production to test middleware and tools, and produce physics data for summer students, based on athena-atlfast fast MC  2 weeks to regenerate data, repeatable  deploy new tools and middleware quickly  move away from farm paradigm to grid model  very aggressive schedule - people limited!  DC1 production to test fabric capabilities, produce and access data, meet ATLAS deliverables, using atlsim, atrig and athena  not repeatable - but real test of grid software  increase U.S. participation - involve grid testbed  CPU intensive - ~14 hours per job/file

15. November 14, 2002 Kaushik De PCAP Review 15 Atlfast Production  Goal: demonstrate grid capabilities  July 1-15: Phase 0, 10^7 events  Globus 2.0 beta  GRAT and Grappa - using athena-atlfast 3.0.1, common disk model  Magda - data management, 1000 files  5 physics processes - http cached  Use Pythia, standard ATLAS physics group jobOptions  000 - atlfast default  001 - minbias  002 - dijets 3 GeV < P t < 100 GeV  003 - all jets  004 - W + jets  005 - Z (ll) + jets  006-010 processes repeat

16. November 14, 2002 Kaushik De PCAP Review 16 GRAT  GRid Applications Toolkit  Based on Globus, Magda & MySQL  Shell & Python scripts  Modular design  parts can be run independently  Rapid development platform  Test grid middleware, test grid performance  Useful as diagnostic tool  Includes > 50 command-line tools for  Athena-atlfast job submission (binary or afs)  Athena-atlfast production  dc1 production and data management  Modules can be easily enhanced or replaced by Condor-G, EDG resource broker, Chimera, replica catalogue… (mostly in progress)

17. November 14, 2002 Kaushik De PCAP Review 17 Database use in GRAT  MySQL databases play a central role in GRAT production scripts  Production database (Magda)  dc1jobstatus and dc1jobinfo tables  track job status, updated periodically by scripts  Data management (Magda)  input/output file transfers  file registration/catalogue  Virtual Data Catalogue (ANL)  job definition  job parameters, random numbers  Metadata catalogue (AMI)  post-production summary information  data provenance

18. November 14, 2002 Kaushik De PCAP Review 18 Atlfast Tools  Production toolkit examples:  run-atlfast-production  at [sites/all, # events, physics/all]  production-filename  generate base+date+events+physics+site  production-getvars  query gatekeeper for env variables  production-queues  return list of queues (static)  production-listqueues  MDS queried list of queues  production-primestore  primary magda cache location  production-replicastore  return replica location  checknodes  return available nodes (globus)  stats/stats2  magda file statistics

19. November 14, 2002 Kaushik De PCAP Review 19 Athena-atlfast Production Architecture Storage selection module Replica storage selection module Job scheduler module Software stage module Move files/cleanup module Execute Athena Job Query env module Filename module JobOptions module Physics Processes from Web GRAT User Node Gatekeeper Queue Node ATLAS_SCRATCH

20. November 14, 2002 Kaushik De PCAP Review 20 Production Status

21. November 14, 2002 Kaushik De PCAP Review 21 Lessons Learned  Globus, Magda and Pacman make grid production easy!  On the grid - submit anywhere, run anywhere, store data anywhere - really works!  Error reporting, recovery & cleanup very important - will always lose/hang some jobs  Found many unexpected limitations, hangs, software problems - next time, need larger team to quantify these problems and provide feedback to Globus, Condor, and other middleware teams  Large pool of hardware resources available on testbed: BNL Tier 1, LBNL (pdsf), IU & BU prototype Tier 2 sites, UTA (new $1.35M NSF- MRI), OU & UNM CS supercomputing clusters...  Testbed production effort suffering from severe shortage of human resources. Need people to debug middleware problems and provide feedback to middleware developers

22. November 14, 2002 Kaushik De PCAP Review 22 GRAT in DC1 Production  Important ATLAS deliverable  Main features:  Automatic job submission under full user control  One, many or infinite sequence of jobs  One or many sites (for scheduling of production)  Independent data management scripts to check consistency of production semi-automatically  query production database  check Globus for job completion status  check Magda for output files  recover from many possible production failures  New (and old) Magda features used:  moving and registering output files stored at BNL HPSS and at replica locations on the grid  tracking production status  Minimal requirement on grid production site  Globus & Magda installed on gatekeeper  shared $atlas_scratch disk on batch nodes

23. November 14, 2002 Kaushik De PCAP Review 23 DC1 samples 5 5000 5000 0.5 0.5 10 7 reconst- ruction 20 20 20000 20000 2 10 7 simulat-ion Total size TB GB Event size MB Number of events 10261000026000 (1) 6.5 (2) 17.5 1.5 x 10 6 10 34 4740007000 (1) 2.6 (2) 4.7 1.5 x 10 6 2 x 10 33 Total size TB Total size GB Event size MB # of events L

24. November 14, 2002 Kaushik De PCAP Review 24 Atlsim, atlfast, athena-based reconstruction 5x10 7 events October 2003 thru March 2004 DC2 Atlsim + Athena > 10 7 events November 2002 - ? DC1-p2 Atlsim (Geant) 10 7 events July 2002 thru August 2002 DC1-p1 Atlfast (Athena) 10 6 events April 2002 DC0TechnologyDataPeriod Challen ge DC1 Schedule

25. November 14, 2002 Kaushik De PCAP Review 25 GRAT DC1 Script Details  Main tools:  dc1-forever: simple opportunistic scheduler  continuously submits jobs to all sites  dc1-submit: submit multiple production jobs  submit one, many or range of partitions at one site  dc1-data-verify: check production consistency  usually run asynchronously to scheduler  dc1-list-*: generate status and statistics  many scripts to provide information  dc1-data-*: tools to manage data  Limitations found in dc1 phase 1:  Single scheduler < 100 jobs  Single gatekeeper < 50 jobs  major shutdowns required user intervention  1 TeV single particles generated - no input files required (cpu intensive: easier on the grid)

26. November 14, 2002 Kaushik De PCAP Review 26 GRAT Computing Infrastructure Gatekeeper GridFTP MDS Gatekeeper GridFTP MDS LocalRemote Batch Common Storage BQS Service to service communication

27. November 14, 2002 Kaushik De PCAP Review 27 GRAT Execution Model 1. Resource Discovery 2. Partition Selection 3. Job Creation 4. Pre-stage 5. Batch Submission 6. Job Parameterization 7. Simulation 8. Post-stage 9. Cataloging 10. Monitoring DC1 Prod. (UTA) Remote Gatekeeper Replica (local) MAGDA (BNL) Param (CERN) Batch Execution scratch 1,4,5,10 2 3 4 5 6 7 89

28. November 14, 2002 Kaushik De PCAP Review 28 GRAT Job Scheduling Create job script module Replica storage select module Site select module Stage software on atlas_scratch Move files/cleanup module Execute Atlsim Job Query environment Partition select module GRAT Scheduler Gatekeeper Queue Node ATLAS_SCRATCH Virtual Data Catalogue Register Production Magda Database

29. November 14, 2002 Kaushik De PCAP Review 29 Production Status Checking [kaushik@atlas000 usertools-0.7]$./dc1-list-jobs Checking the status of 123 globus jobs.............................................................................. Active Pending Jobs at LBNL: 39 0 Jobs at OU : 8 0 Jobs at UTA : 24 6 [kaushik@atlas000 usertools-0.7]$ condor_q -- Submitter: atlas000.uta.edu : ID OWNER ITTED RUN_TIME ST PRI SIZE CMD 2107.0 kaushik 9/8 12:41 1+02:17:43 R 0 0.0 dc1-run 2108.0 kaushik 9/8 12:43 1+02:13:29 R 0 0.0 dc1-run 2109.0 kaushik 9/8 12:45 1+02:13:27 R 0 0.0 dc1-run 2110.0 kaushik 9/8 13:12 1+01:43:58 R 0 0.0 dc1-run 2114.0 kaushik 9/8 13:21 1+01:37:06 R 0 0.0 dc1-run 2115.0 kaushik 9/8 13:50 1+01:08:37 R 0 0.0 dc1-run 2145.0 kaushik 9/9 14:44 0+00:00:00 I 0 0.0 dc1-run 2146.0 kaushik 9/9 14:46 0+00:00:00 I 0 0.0 dc1-run 30 jobs; 6 idle, 24 running, 0 held

30. November 14, 2002 Kaushik De PCAP Review 30 DC1 Production Statistics $./dc1-list-statistics Wed Nov 13 13:03:46 CST 2002 total number of partitions at BNL HPSS = 2002 total size of zebra files in HPSS = 48122 MB Files Size 1 TeV elec. (2107) 500 13488M 1 TeV photon (2117) 500 11564M 1 TeV photon (2127) 500 11543M 1 TeV photon (2137) 500 11464M Replica locations: files @ UTA 738 18895M files @ Oklahoma 361 8393M files @ LBNL 915 21140M Grid+BNL farm production = 2nd largest in DC1

31. November 14, 2002 Kaushik De PCAP Review 31 DC1 Statistics

32. November 14, 2002 Kaushik De PCAP Review 32 DC1 Production Experience  Anything that can go wrong, WILL go wrong  During 18 days of grid production (in August), every system died at least once  Local experts were not always be accessible (many of them on vacation)  Examples: scheduling machines died 5 times (thrice power failure, twice system hung)  Network outages - multiple times  Gatekeeper - died at every site at least 2-3 times  Three databases used - production, magda and virtual data. Each died at least once!  Scheduled maintenance - HPSS, Magda server, LBNL hardware, LBNL Raid array…  These outages should be expected on the grid - software design must be robust  We managed > 100 files/day (~80% efficiency) in spite of these problems!

33. November 14, 2002 Kaushik De PCAP Review 33 DC1 Phase 2  Pile up production:  plan to start now!  grid-based production using 200-300 nodes  SUSY simulation - 50k events  Athena reconstruction  grid testbed will participate  store ESD/AOD at BNL, Ntuples on grid testbed  need grid deployable athena package!  Analysis / user access to data  magda_getfile already provides access to ~30k catalogued dc1 files from/to many grid locations (need ATLAS VO to make this universal)  testbed dc1 production database in magda with GRAT provides simple ‘discovery’ mechanism  long term: DIAL being developed at BNL http://www.usatlas.bnl.gov/~dladams/dial/

34. November 14, 2002 Kaushik De PCAP Review 34 Grid Monitoring  Monitoring is critically important in distributed Grid computing  check system health, debug problems  discover resources using static data  job scheduling and resource allocation decisions using dynamic data from MDS and other monitors  Testbed monitoring priorities  Discover site configuration  Discover software installation  Application monitoring  Grid status/operations monitoring  Visualization  Also need  Well defined data for job scheduling

35. November 14, 2002 Kaushik De PCAP Review 35 Monitoring - Back End  Publishing MDS information  Glue schema - BNL & UTA  Pippy - Pacman information service provider  BNL ACAS schema  Hierarchical GIIS server  Non-MDS back ends  iPerf, Netlogger, Prophesy, Ganglia  Archiving  MySQL  GridView, BNL ACAS  RRD  Network  Work needed  What to store?  Replication of archived information  Good progress on back end!

36. November 14, 2002 Kaushik De PCAP Review 36 PIPPY  Publishes information into MDS about software packages installed via Pacman.  Uses LDAP schema elements that describe a package.  Interfaces with GRIS backend of MDS service.  pippy creates two type of entries:  Software Collection  Software Package  Collection allows aggregation of packages.  Package describes contents of a pacman file.  Will be included in next release of VDT

37. November 14, 2002 Kaushik De PCAP Review 37 Architecture MDS (slapd) GRISGIIS pip.py Pacman.db InvocationLDIF.conf files

38. November 14, 2002 Kaushik De PCAP Review 38 Monitoring - Front End  MDS based  GridView, Gridsearcher, Gstat  Converting TeraGrid and other toolkits  Non-MDS  Cricket, Ganglia  Example:  Gstat - text version

39. November 14, 2002 Kaushik De PCAP Review 39 GridView - Old Text Version   Simple visualization tool using Globus Toolkit  First native Globus application for ATLAS grid (March 2001)  Collects information using Globus tools. Archival information is stored in MySQL server on a different machine. Data published through web server on a third machine.   http://heppc1.uta.edu/atlas/grid-status/index.html http://heppc1.uta.edu/atlas/grid-status/index.html

40. November 14, 2002 Kaushik De PCAP Review 40 GridView (NG)

41. November 14, 2002 Kaushik De PCAP Review 41 Fabric Testing

42. November 14, 2002 Kaushik De PCAP Review 42 MDS Browser

43. November 14, 2002 Kaushik De PCAP Review 43 Integration!!  Coordination with other grid efforts and software developers - very difficult task!  Project centric:  GriPhyN/iVDGL - Rob Gardner  PPDG/LCG - Torre Wenaus  EDG - Ed May, Jerry Gieraltowski  ATLAS/LHCb - Rich Baker  ATLAS/CMS - Kaushik De  ATLAS/D0 - Jae Yu  Fabric/Middleware centric:  Afs Software installations - Alex Undrus, Shane Canon, Iwona Sakrejda  Networking - Shawn McKee, Rob Gardner  Virtual and Real Data Management - Wendsheng Deng, Sasha Vaniachin, Pavel Nevski, David Malon, Rob Gardner, Dan Engh, Mike Wilde, Yong Zhao, Shava Smallen  Security/Site AA/VO - Rich Baker, Dantong Yu

44. November 14, 2002 Kaushik De PCAP Review 44 Summary  Testbed ideal for rapid testing of middleware, prototyping and testing of production software, and MC production  Extensive recent experience with grid deployment for DC & SC2002  U.S. Testbed and NorduGrid were the only sites to use grid for ATLAS DC1 Phase 1  Testbed provides large pool of hardware resources through the grid  Severely short of human resources - large list of tasks spanning many WBS items  Need testbed effort to provide feedback to middleware and application developers  Important DC milestones coming up!