1. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery1 University of Florida http://www.phys.ufl.edu/~avery/ avery@phys.ufl.edu U.S. Physics Data Grid Projects International Workshop on HEP Data Grids Kyungpook National University, Daegu, Korea Nov. 8-9, 2002

2. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery2 “Trillium”: US Physics Data Grid Projects  Particle Physics Data Grid (PPDG)  Data Grid for HENP experiments  ATLAS, CMS, D0, BaBar, STAR, JLAB  GriPhyN  Petascale Virtual-Data Grids  ATLAS, CMS, LIGO, SDSS  iVDGL  Global Grid lab  ATLAS, CMS, LIGO, SDSS, NVO  Data intensive expts.  Collaborations of physicists & computer scientists  Infrastructure development & deployment  Globus + VDT based =

3. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery3 Why Trillium?  Many common aspects  Large overlap in project leadership  Large overlap in participants  Large overlap in experiments, particularly LHC  Common projects (monitoring, etc.)  Common packaging  Common use of VDT, other GriPhyN software  Funding agencies like collaboration  Good working relationship on grids between NSF and DOE  Good complementarity: DOE (labs), NSF (universities)  Collaboration of computer science/physics/astronomy encouraged  Organization from the “bottom up”  With encouragement from funding agencies

4. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery4 1800 Physicists 150 Institutes 32 Countries Driven by LHC Computing Challenges  Complexity:Millions of detector channels, complex events  Scale:PetaOps (CPU), Petabytes (Data)  Distribution:Global distribution of people & resources

5. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery5 Experiment (e.g., CMS) Global LHC Data Grid Online System CERN Computer Center > 20 TIPS USA Korea Russia UK Institute 100-200 MBytes/s 2.5 Gbits/s 0.1 - 1 Gbits/s 2.5 Gbits/s ~0.6 Gbits/s Tier 0 Tier 1 Tier 3 Tier 4 Tier0/(  Tier1)/(  Tier2) ~ 1:1:1 Tier 2 Physics cache PCs, other portals Institute Tier2 Center

6. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery6 LHC Tier2 Center (2001) Router >1 RAID WAN FEth/GEth Switch “Flat” switching topology Data Server 20-60 nodes Dual 0.8-1 GHz, P3 1 TByte RAID

7. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery7 LHC Tier2 Center (2002-2003) Router GEth/FEth Switch GEth Switch Data Server >1 RAID WAN “Hierarchical” switching topology Switch GEth/FEth 40-100 nodes Dual 2.5 GHz, P4 2-4 TBytes RAID

8. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery8  Buy late, but not too late: phased implementation  R&D Phase 2001-2004  Implementation Phase2004-2007  R&D to develop capabilities and computing model itself  Prototyping at increasing scales of capability & complexity 1.4 years 1.2 years 1.1 years 2.1 years LHC Hardware Cost Estimates

9. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery9 Particle Physics Data Grid “In coordination with complementary projects in the US and Europe, PPDG aims to meet the urgent needs for advanced Grid-enabled technology and to strengthen the collaborative foundations of experimental particle and nuclear physics.”

10. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery10 PPDG Goals  Serve high energy & nuclear physics (HENP) experiments  Funded 2001 – 2004 @ US$9.5M (DOE)  Develop advanced Grid technologies  Use Globus to develop higher level tools  Focus on end to end integration  Maintain practical orientation  Networks, instrumentation, monitoring  DB file/object replication, caching, catalogs, end-to-end movement  Serve urgent needs of experiments  Unique challenges, diverse test environments  But make tools general enough for wide community!  Collaboration with GriPhyN, iVDGL, EDG, LCG  Recent work on ESNet Certificate Authority

11. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery11 PPDG Participants and Work Program  Physicist + CS involvement  D0, BaBar, STAR, CMS, ATLAS  SLAC, LBNL, Jlab, FNAL, BNL, Caltech, Wisconsin, Chicago, USC  Computer Science Program of Work  CS1: Job description language  CS2: Schedule, manage data processing, data placement activities  CS3: Monitoring and status reporting (with GriPhyN)  CS4: Storage resource management  CS5: Reliable replication services  CS6: File transfer services  CS7: Collect/document experiment practices  generalize  …  CS11: Grid-enabled data analysis

12. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery12 GriPhyN = App. Science + CS + Grids  Participants  US-CMSHigh Energy Physics  US-ATLASHigh Energy Physics  LIGO/LSCGravity wave research  SDSSSloan Digital Sky Survey  Strong partnership with computer scientists  Design and implement production-scale grids  Develop common infrastructure, tools and services (Globus based)  Integration into the 4 experiments  Broad application to other sciences via “Virtual Data Toolkit”  Strong outreach program  Funded by NSF for 2000 – 2005  R&D for grid architecture (funded at $11.9M +$1.6M)  Integrate Grid infrastructure into experiments through VDT

13. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery13 GriPhyN: PetaScale Virtual-Data Grids Virtual Data Tools Request Planning & Scheduling Tools Request Execution & Management Tools Transforms Distributed resources (code, storage, CPUs, networks) è Resource è Management è Services Resource Management Services è Security and è Policy è Services Security and Policy Services è Other Grid è Services Other Grid Services Interactive User Tools Production Team Individual Investigator Workgroups Raw data source ~1 Petaflop ~100 Petabytes

14. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery14 GriPhyN Research Agenda  Based on Virtual Data technologies (fig.)  Derived data, calculable via algorithm  Instantiated 0, 1, or many times (e.g., caches)  “Fetch value” vs “execute algorithm”  Very complex (versions, consistency, cost calculation, etc)  LIGO example  “Get gravitational strain for 2 minutes around each of 200 gamma- ray bursts over the last year”  For each requested data value, need to  Locate item location and algorithm  Determine costs of fetching vs calculating  Plan data movements & computations required to obtain results  Schedule the plan  Execute the plan

15. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery15 Virtual Data Concept  Data request may  Compute locally  Compute remotely  Access local data  Access remote data  Scheduling based on  Local policies  Global policies  Cost Major facilities, archives Regional facilities, caches Local facilities, caches Fetch item

16. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery16 iVDGL: A Global Grid Laboratory  International Virtual-Data Grid Laboratory  A global Grid laboratory (US, EU, Asia, South America, …)  A place to conduct Data Grid tests “at scale”  A mechanism to create common Grid infrastructure  A laboratory for other disciplines to perform Data Grid tests  A focus of outreach efforts to small institutions  U.S. part funded by NSF (2001 – 2006)  $14.1M (NSF) + $2M (matching)  International partners bring own funds “We propose to create, operate and evaluate, over a sustained period of time, an international research laboratory for data-intensive science.” From NSF proposal, 2001

17. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery17 iVDGL Participants  Initial experiments (funded by NSF proposal)  CMS, ATLAS, LIGO, SDSS, NVO  Possible other experiments and disciplines  HENP:BTEV, D0, CMS HI, ALICE, …  Non-HEP:Biology, …  Complementary EU project: DataTAG  DataTAG and US pay for 2.5 Gb/s transatlantic network  Additional support from UK e-Science programme  Up to 6 Fellows per year  None hired yet 

18. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery18 iVDGL Components  Computing resources  Tier1 laboratory sites (funded elsewhere)  Tier2 university sites  software integration  Tier3 university sites  outreach effort  Networks  USA (Internet2, ESNet), Europe (Géant, …)  Transatlantic (DataTAG), Transpacific, AMPATH, …  Grid Operations Center (GOC)  Indiana (2 people)  Joint work with TeraGrid on GOC development  Computer Science support teams  Support, test, upgrade GriPhyN Virtual Data Toolkit  Coordination, management

19. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery19 iVDGL Management and Coordination Project Coordination Group US External Advisory Committee GLUE Interoperability Team Collaborating Grid Projects TeraGridEDGAsiaDataTAG BTEV LCG? BioALICEGeo? D0PDCCMS HI ? US Project Directors Outreach Team Core Software Team Facilities Team Operations Team Applications Team International Piece US Project Steering Group U.S. Piece

20. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery20 iVDGL Work Teams  Facilities Team  Hardware (Tier1, Tier2, Tier3)  Core Software Team  Grid middleware, toolkits  Laboratory Operations Team  Coordination, software support, performance monitoring  Applications Team  High energy physics, gravity waves, virtual astronomy  Nuclear physics, bioinformatics, …  Education and Outreach Team  Web tools, curriculum development, involvement of students  Integrated with GriPhyN, connections to other projects  Want to develop further international connections

21. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery21 US-iVDGL Data Grid (Sep. 2001) UF Wisconsin Fermilab BNL Indiana Boston U SKC Brownsville Hampton PSU J. Hopkins Caltech Tier1 Tier2 Tier3 Argonne UCSD/SDSC

22. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery22 US-iVDGL Data Grid (Dec. 2002) UF Wisconsin Fermilab BNL Indiana Boston U SKC Brownsville Hampton PSU J. Hopkins Caltech Tier1 Tier2 Tier3 FIU FSU Arlington Michigan LBL Oklahoma Argonne Vanderbilt UCSD/SDSC NCSA

23. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery23 Possible iVDGL Participant: TeraGrid 26 24 8 4 HPSS 5 UniTree External Networks Site Resources NCSA/PACI 8 TF 240 TB SDSC 4.1 TF 225 TB CaltechArgonne 40 Gb/s 13 TeraFlops

24. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery24 International Participation  Existing partners  European Data Grid (EDG)  DataTAG  Potential partners  KoreaT1  ChinaT1?  JapanT1?  BrazilT1  RussiaT1  ChileT2  PakistanT2  Romania?

25. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery25 Current Trillium Work  Packaging technologies: PACMAN  Used for VDT releases  very successful & powerful  Evaluated for Globus, EDG  GriPhyN Virtual Data Toolkit 1.1.3 released  Vastly simplifies installation of grid tools  New changes will further simplify configuration complexity  Monitoring (joint efforts)  Globus MDS 2.2 (GLUE schema)  Caltech MonaLisa  Condor HawkEye  Florida Gossip (low level component)  Chimera Virtual Data System (more later)  Testbeds, demo projects (more later)

26. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery26 Virtual Data: Derivation and Provenance  Most scientific data are not simple “measurements”  They are computationally corrected/reconstructed  They can be produced by numerical simulation  Science & eng. projects are more CPU and data intensive  Programs are significant community resources (transformations)  So are the executions of those programs (derivations)  Management of dataset transformations important!  Derivation: Instantiation of a potential data product  Provenance: Exact history of any existing data product Programs are valuable, like data. They should be community resources

27. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery27 Transformation Derivation Data product-of execution-of consumed-by/ generated-by “I’ve detected a mirror calibration error and want to know which derived data products need to be recomputed.” “I’ve found some interesting data, but I need to know exactly what corrections were applied before I can trust it.” “I want to search a database for dwarf galaxies. If a program that performs this analysis exists, I won’t have to write one from scratch.” “I want to apply a shape analysis to 10M galaxies. If the results already exist, I’ll save weeks of computation.” Motivations (1)

28. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery28 Motivations (2)  Data track-ability and result audit-ability  Universally sought by GriPhyN applications  Facilitates tool and data sharing and collaboration  Data can be sent along with its recipe  Repair and correction of data  Rebuild data products—c.f., “make”  Workflow management  A new, structured paradigm for organizing, locating, specifying, and requesting data products  Performance optimizations  Ability to re-create data rather than move it

29. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery29 “Chimera” Virtual Data System  Virtual Data API  A Java class hierarchy to represent transformations & derivations  Virtual Data Language  Textual for people & illustrative examples  XML for machine-to-machine interfaces  Virtual Data Database  Makes the objects of a virtual data definition persistent  Virtual Data Service (future)  Provides a service interface (e.g., OGSA) to persistent objects

30. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery30 Virtual Data Catalog Object Model

31. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery31  Virtual Data Language (VDL)  Describes virtual data products  Virtual Data Catalog (VDC)  Used to store VDL  Abstract Job Flow Planner  Creates a logical DAG (dependency graph)  Concrete Job Flow Planner  Interfaces with a Replica Catalog  Provides a physical DAG submission file to Condor-G  Generic and flexible  As a toolkit and/or a framework  In a Grid environment or locally  Currently in beta version Logical Physical Abstract Planner VDC Replica Catalog Concrete Planner DAX DAGMan DAG VDL XML Chimera as a Virtual Data System XML

32. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery32 Size distribution of galaxy clusters? Galaxy cluster size distribution Chimera Virtual Data System + GriPhyN Virtual Data Toolkit + iVDGL Data Grid (many CPUs) Chimera Application: SDSS Analysis

33. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery33 US-CMS Testbed

34. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery34 Other CMS Institutes Encouraged to Join Expressions of interest Princeton Brazil South Korea Minnesota Iowa Possibly others

35. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery35 Grid Middleware Used in Testbed  Virtual Data Toolkit 1.1.3  VDT Client: Globus Toolkit 2.0 Condor-G 6.4.3  VDT Server: Globus Toolkit 2.0 mkgridmap Condor 6.4.3 ftsh GDMP 3.0.7  Virtual Organization (VO) Management  LDAP Server deployed at Fermilab  GroupMAN (adapted from EDG) used to manage the VO  Use D.O.E. Science Grid certificates  Accept EDG and Globus certificates

36. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery36 Commissioning the CMS Grid Testbed  A complete prototype (fig.)  CMS Production Scripts  Globus  Condor-G  GridFTP  Commissioning: Require production quality results!  Run until the Testbed "breaks"  Fix Testbed with middleware patches  Repeat procedure until the entire Production Run finishes!  Discovered/fixed many Globus and Condor-G problems  Huge success from this point of view alone  … but very painful

37. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery37 CMS Grid Testbed Production Remote Site 2 Master Site Remote Site 1 IMPALA mop_submitter DAGMan Condor-G GridFTP Batch Queue GridFTP Batch Queue GridFTP Remote Site N Batch Queue GridFTP

38. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery38 LinkerScriptGenerator Configurator Requirements Self Description MasterScript"DAGMaker"VDL MOP Chimera MCRunJob Production Success on CMS Testbed  Results  150k events generated, ~200 GB produced  1.5 weeks continuous running across all 5 testbed sites  1M event run just started on larger testbed (~30% complete!)

39. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery39 Grid Coordination Efforts  Global Grid Forum (www.gridforum.org)  International forum for general Grid efforts  Many working groups, standards definitions  Next one in Japan, early 2003  HICB (High energy physics)  Joint development & deployment of Data Grid middleware  GriPhyN, PPDG, iVDGL, EU-DataGrid, LCG, DataTAG, Crossgrid  GLUE effort (joint iVDGL – DataTAG working group)  LCG (LHC Computing Grid Project)  Strong “forcing function”  Large demo projects  IST2002Copenhagen  Supercomputing 2002Baltimore  New proposal (joint NSF + Framework 6)?

40. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery40 WorldGrid Demo  Joint Trillium-EDG-DataTAG demo  Resources from both sides in Intercontinental Grid Testbed  Use several visualization tools (Nagios, MapCenter, Ganglia)  Use several monitoring tools (Ganglia, MDS, NetSaint, …)  Applications  CMS:CMKIN, CMSIM  ATLAS:ATLSIM  Submit jobs from US or EU  Jobs can run on any cluster  Shown at IST2002 (Copenhagen)  To be shown at SC2002 (Baltimore)  Brochures now available describing Trillium and demos  I have 10 with me now (2000 just printed)

41. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery41 WorldGrid

42. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery42 Summary  Very good progress on many fronts  Packaging  Testbeds  Major demonstration projects  Current Data Grid projects are providing good experience  Looking to collaborate with more international partners  Testbeds  Monitoring  Deploying VDT more widely  Working towards new proposal  Emphasis on Grid-enabled analysis  Extending Chimera virtual data system to analysis

43. Korean HEP Grid Workshop (Nov. 8, 2002)Paul Avery43 Grid References  Grid Book  www.mkp.com/grids  Globus  www.globus.org  Global Grid Forum  www.gridforum.org  TeraGrid  www.teragrid.org  EU DataGrid  www.eu-datagrid.org  PPDG  www.ppdg.net  GriPhyN  www.griphyn.org  iVDGL  www.ivdgl.org