1. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery1 University of Florida http://www.phys.ufl.edu/~avery/ avery@phys.ufl.edu Physics Colloquium University of Texas at Arlington Jan. 24, 2002 Global Data Grids for 21 st Century Science

2. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery2 What is a Grid? è Grid: Geographically distributed computing resources configured for coordinated use è Physical resources & networks provide raw capability è “Middleware” software ties it together

3. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery3 Applications for Grids è Climate modeling  Climate scientists visualize, annotate, & analyze Terabytes of simulation data è Biology  A biochemist exploits 10,000 computers to screen 100,000 compounds in an hour è High energy physics  3,000 physicists worldwide pool Petaflops of CPU resources to analyze Petabytes of data è Engineering  Civil engineers collaborate to design, execute, & analyze shake table experiments  A multidisciplinary analysis in aerospace couples code and data in four companies From Ian Foster

4. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery4 Applications for Grids (cont.) è Application Service Providers  A home user invokes architectural design functions at an application service provider  An application service provider purchases cycles from compute cycle providers è Commercial  Scientists at a multinational soap company design a new product è Communities  An emergency response team couples real time data, weather model, population data  A community group pools members’ PCs to analyze alternative designs for a local road è Health  Hospitals and international agencies collaborate on stemming a major disease outbreak From Ian Foster

5. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery5 Proto-Grid: SETI@home è Community: SETI researchers + enthusiasts è Arecibo radio data sent to users (250KB data chunks) è Over 2M PCs used

6. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery6 è Community  1000s of home computer users  Philanthropic computing vendor (Entropia)  Research group (Scripps) è Common goal  Advance AIDS research More Advanced Proto-Grid: Evaluation of AIDS Drugs

7. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery7 Early Information Infrastructure è Network-centric  Simple, fixed end systems  Few embedded capabilities  Few services  No user-level quality of service O(10 8 ) nodes Network

8. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery8 Emerging Information Infrastructure è Application-centric  Heterogeneous, mobile end-systems  Many embedded capabilities  Rich services  User-level quality of service QoS Resource Discovery O(10 10 ) nodes Qualitatively different, not just “faster and more reliable” Processing Grid Caching

9. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery9 Why Grids? è Resources for complex problems are distributed  Advanced scientific instruments (accelerators, telescopes, …)  Storage and computing  Groups of people è Communities require access to common services  Scientific collaborations (physics, astronomy, biology, eng. …)  Government agencies  Health care organizations, large corporations, … è Goal is to build “Virtual Organizations”  Make all community resources available to any VO member  Leverage strengths at different institutions  Add people & resources dynamically

10. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery10 Grid Challenges è Overall goal  Coordinated sharing of resources è Technical problems to overcome  Authentication, authorization, policy, auditing  Resource discovery, access, allocation, control  Failure detection & recovery  Resource brokering è Additional issue: lack of central control & knowledge  Preservation of local site autonomy  Policy discovery and negotiation important

11. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery11 Layered Grid Architecture (Analogy to Internet Architecture) Application Fabric Controlling things locally: Accessing, controlling resources Connectivity Talking to things: communications, security Resource Sharing single resources: negotiating access, controlling use Collective Managing multiple resources: ubiquitous infrastructure services User Specialized services: App. specific distributed services Internet Transport Application Link Internet Protocol Architecture From Ian Foster

12. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery12 Globus Project and Toolkit è Globus Project™ (Argonne + USC/ISI)  O(40) researchers & developers  Identify and define core protocols and services è Globus Toolkit™  A major product of the Globus Project  Reference implementation of core protocols & services  Growing open source developer community è Globus Toolkit used by all Data Grid projects today  US:GriPhyN, PPDG, TeraGrid, iVDGL  EU:EU-DataGrid and national projects

13. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery13 Globus General Approach è Define Grid protocols & APIs  Protocol-mediated access to remote resources  Integrate and extend existing standards è Develop reference implementation  Open source Globus Toolkit  Client & server SDKs, services, tools, etc. è Grid-enable wide variety of tools  Globus Toolkit  FTP, SSH, Condor, SRB, MPI, … è Learn about real world problems  Deployment  Testing  Applications Diverse global services Core services Diverse OS services Applications

14. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery14 Globus Toolkit Protocols è Security (connectivity layer)  Grid Security Infrastructure (GSI) è Resource management (resource layer)  Grid Resource Allocation Management (GRAM) è Information services (resource layer)  Grid Resource Information Protocol (GRIP) è Data transfer (resource layer)  Grid File Transfer Protocol (GridFTP)

15. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery15 Data Grids

16. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery16 Data Intensive Science: 2000-2015 è Scientific discovery increasingly driven by IT  Computationally intensive analyses  Massive data collections  Data distributed across networks of varying capability  Geographically distributed collaboration è Dominant factor: data growth (1 Petabyte = 1000 TB)  2000~0.5 Petabyte  2005~10 Petabytes  2010~100 Petabytes  2015~1000 Petabytes? How to collect, manage, access and interpret this quantity of data? Drives demand for “Data Grids” to handle additional dimension of data access & movement

17. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery17 Global Data Grid Challenge “Global scientific communities will perform computationally demanding analyses of distributed datasets that will grow by at least 3 orders of magnitude over the next decade, from the 100 Terabyte to the 100 Petabyte scale.”

18. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery18 Data Intensive Physical Sciences è High energy & nuclear physics è Gravity wave searches  LIGO, GEO, VIRGO è Astronomy: Digital sky surveys  Now:Sloan Sky Survey, 2MASS  Future:VISTA, other Gigapixel arrays  “Virtual” Observatories (Global Virtual Observatory) è Time-dependent 3-D systems (simulation & data)  Earth Observation  Climate modeling  Geophysics, earthquake modeling  Fluids, aerodynamic design  Pollutant dispersal scenarios

19. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery19 Data Intensive Biology and Medicine è Medical data  X-Ray, mammography data, etc. (many petabytes)  Digitizing patient records (ditto) è X-ray crystallography  Bright X-Ray sources, e.g. Argonne Advanced Photon Source è Molecular genomics and related disciplines  Human Genome, other genome databases  Proteomics (protein structure, activities, …)  Protein interactions, drug delivery è Brain scans (3-D, time dependent) è Virtual Population Laboratory (proposed)  Database of populations, geography, transportation corridors  Simulate likely spread of disease outbreaks Craig Venter keynote @SC2001

20. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery20 Data and Corporations è Corporations and Grids  National, international, global  Business units, research teams  Sales data  Transparent access to distributed databases è Corporate issues  Short term and long term partnerships  Overlapping networks  Manage, control access to data and resources  Security

21. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery21 Example: High Energy Physics “Compact” Muon Solenoid at the LHC (CERN) Smithsonian standard man

22. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery22 LHC Computing Challenges è “Events” resulting from beam-beam collisions:  Signal event is obscured by 20 overlapping uninteresting collisions in same crossing  CPU time does not scale from previous generations 20002007

23. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery23 10 9 events/sec, selectivity: 1 in 10 13 LHC: Higgs Decay into 4 muons

24. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery24 1800 Physicists 150 Institutes 32 Countries LHC Computing Challenges è Complexity of LHC interaction environment & resulting data è Scale: Petabytes of data per year (100 PB by ~2010-12) è GLobal distribution of people and resources

25. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery25 Tier0 CERN Tier1 National Lab Tier2 Regional Center (University, etc.) Tier3 University workgroup Tier4 Workstation Global LHC Data Grid Tier 1 T2 3 3 3 3 3 3 3 3 3 3 3 Tier 0 (CERN) 4 4 4 4 3 3 Key ideas: è Hierarchical structure è Tier2 centers

26. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery26 Global LHC Data Grid Tier2 Center Online System CERN Computer Center > 20 TIPS USA Center France Center Italy Center UK Center Institute Institute ~0.25TIPS Workstations, other portals ~100 MBytes/sec 2.5 Gbits/sec 100 - 1000 Mbits/sec Bunch crossing per 25 nsecs. 100 triggers per second Event is ~1 MByte in size Physicists work on analysis “channels”. Each institute has ~10 physicists working on one or more channels Physics data cache ~PBytes/sec 2.5 Gbits/sec Tier2 Center ~622 Mbits/sec Tier 0 +1 Tier 1 Tier 3 Tier 4 Tier2 Center Tier 2 Experiment CERN/Outside Resource Ratio ~1:2 Tier0/(  Tier1)/(  Tier2) ~1:1:1

27. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery27 Example: Global Virtual Observatory Source Catalogs Image Data Specialized Data: Spectroscopy, Time Series, Polarization Information Archives: Derived & legacy data: NED,Simbad,ADS, etc Discovery Tools: Visualization, Statistics Standards Multi-wavelength astronomy, Multiple surveys

28. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery28 GVO Data Challenge è Digital representation of the sky  All-sky + deep fields  Integrated catalog and image databases  Spectra of selected samples è Size of the archived data  40,000 square degrees  Resolution 50 trillion pixels  One band (2 bytes/pixel)100 Terabytes  Multi-wavelength:500-1000 Terabytes  Time dimension:Many Petabytes è Large, globally distributed database engines  Integrated catalog and image databases  Multi-Petabyte data size  Gbyte/s aggregate I/O speed per site

29. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery29 Sloan Digital Sky Survey Data Grid

30. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery30 LIGO (Gravity Wave) Data Grid Hanford Observatory Livingston Observatory Caltech MIT INet2 Abilene Tier1 LSC Tier2 OC3 OC48 OC3 OC12 OC48

31. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery31 Data Grid Projects

32. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery32 Large Data Grid Projects è Funded projects  GriPhyNUSANSF$11.9M + $1.6M2000-2005  EU DataGridEUEC€10M2001-2004  PPDGUSADOE$9.5M2001-2004  TeraGridUSANSF$53M2001-?  iVDGLUSANSF$13.7M + $2M2001-2006  DataTAGEUEC€4M2002-2004 è Proposed projects  GridPPUKPPARC>$15M?2001-2004 è Many national projects  Initiatives in US, UK, Italy, France, NL, Germany, Japan, …  EU networking initiatives (Géant, SURFNet)

33. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery33 Future OO-collection export Cache, state tracking Prediction PPDG Middleware Components Object- and File-based Application Services (Request Interpreter) Cache Manager File Access Service (Request Planner) Matchmaking Service Cost Estimation File Fetching Service File Replication Index End-to-End Network Services Mass Storage Manager Resource Management File Mover Site Boundary Security Domain

34. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery34 EU DataGrid Project

35. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery35 GriPhyN: PetaScale Virtual-Data Grids Virtual Data Tools Request Planning & Scheduling Tools Request Execution & Management Tools Transforms Distributed resources (code, storage, CPUs, networks) è Resource è Management è Services Resource Management Services è Security and è Policy è Services Security and Policy Services è Other Grid è Services Other Grid Services Interactive User Tools Production Team Individual Investigator Workgroups Raw data source ~1 Petaflop ~100 Petabytes

36. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery36 GriPhyN Research Agenda è Virtual Data technologies (fig.)  Derived data, calculable via algorithm  Instantiated 0, 1, or many times (e.g., caches)  “Fetch value” vs “execute algorithm”  Very complex (versions, consistency, cost calculation, etc) è LIGO example  “Get gravitational strain for 2 minutes around each of 200 gamma- ray bursts over the last year” è For each requested data value, need to  Locate item location and algorithm  Determine costs of fetching vs calculating  Plan data movements & computations required to obtain results  Execute the plan

37. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery37 Virtual Data in Action è Data request may  Compute locally  Compute remotely  Access local data  Access remote data è Scheduling based on  Local policies  Global policies  Cost Major facilities, archives Regional facilities, caches Local facilities, caches Fetch item

38. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery38 GriPhyN/PPDG Data Grid Architecture Application Planner Executor Catalog Services Info Services Policy/Security Monitoring Repl. Mgmt. Reliable Transfer Service Compute ResourceStorage Resource DAG DAGMAN, Kangaroo GRAMGridFTP; GRAM; SRM GSI, CAS MDS MCAT; GriPhyN catalogs GDMP MDS Globus = initial solution is operational

39. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery39 Transparency wrt materialization Id Trans FParamName … i1 F X F.X … i2 F Y F.Y … i10 G Y PG(P).Y … TransProgCost … F URL:f 10 … G URL:g 20 … Program storage Trans. name URLs for program location Derived Data Catalog Transformation Catalog Update upon materialization App specificattr. id … …i2,i10 … … Derived Metadata Catalog id Id TransParam Name … i1 F X F.X … i2 F Y F.Y … i10 G Y PG(P).Y … Trans ProgCost … F URL:f 10 … G URL:g 20 … Program storage Trans. name URLs for program location App-specific-attr id … …i2,i10 … … id Physical file storage URLs for physical file location NameLObjN… F.XlogO3 … … LCNPFNs… logC1 URL1 logC2 URL2 URL3 logC3 URL4 logC4 URL5 URL6 Metadata Catalog Replica Catalog Logical Container Name GCMS Object Name Transparency wrt location Name LObjN … … X logO1 … … Y logO2 … … F.X logO3 … … G(1).Y logO4 … … LCNPFNs… logC1 URL1 logC2 URL2 URL3 logC3 URL4 logC4 URL5 URL6 Replica Catalog GCMS Object Name Catalog Architecture Metadata Catalog

40. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery40 April 2001  Caltech  NCSA  Wisconsin NCSA Linux cluster 5) Secondary reports complete to master Master Condor job running at Caltech 7) GridFTP fetches data from UniTree NCSA UniTree - GridFTP- enabled FTP server 4) 100 data files transferred via GridFTP, ~ 1 GB each Secondary Condor job on UW pool 3) 100 Monte Carlo jobs on Wisconsin Condor pool 2) Launch secondary job on Wisconsin pool; input files via Globus GASS Caltech workstation 6) Master starts reconstruction jobs via Globus jobmanager on cluster 8) Processed objectivity database stored to UniTree 9) Reconstruction job reports complete to master Early GriPhyN Challenge Problem: CMS Data Reconstruction

41. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery41 Pre / Simulation Jobs / Post (UW Condor) ooHits at NCSA ooDigis at NCSA Delay due to script error Trace of a Condor-G Physics Run

42. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery42 iVDGL: A World Grid Laboratory è International Virtual-Data Grid Laboratory  A global Grid laboratory (US, EU, Asia, …)  A place to conduct Data Grid tests “at scale”  A mechanism to create common Grid infrastructure  A facility to perform production exercises for LHC experiments  A laboratory for other disciplines to perform Data Grid tests è US part funded by NSF: Sep. 25, 2001  $13.65M + $2M “We propose to create, operate and evaluate, over a sustained period of time, an international research laboratory for data-intensive science.” From NSF proposal, 2001

43. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery43 iVDGL Summary Information è Principal components  Tier1 sites (laboratories)  Tier2 sites (universities)  Selected Tier3 sites (universities)  Fast networks: US, Europe, transatlantic, transpacific  Grid Operations Center (GOC)  Computer Science support teams (6 UK Fellows)  Coordination, management è Proposed international participants  Initially US, EU, Japan, Australia  Other world regions later  Discussions w/ Russia, China, Pakistan, India, Brazil è Complementary EU project: DataTAG  Transatlantic network from CERN to STAR-TAP (+ people)  Initially 2.5 Gb/s

44. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery44  U FloridaCMS  CaltechCMS, LIGO  UC San DiegoCMS, CS  Indiana UATLAS, iGOC  Boston UATLAS  U Wisconsin, MilwaukeeLIGO  Penn StateLIGO  Johns HopkinsSDSS, NVO  U ChicagoCS  U Southern CaliforniaCS  U Wisconsin, MadisonCS  Salish KootenaiOutreach, LIGO  Hampton UOutreach, ATLAS  U Texas, BrownsvilleOutreach, LIGO  FermilabCMS, SDSS, NVO  BrookhavenATLAS  Argonne LabATLAS, CS US iVDGL Proposal Participants T2 / Software CS support T3 / Outreach T1 / Labs (not funded)

45. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery45 Initial US-iVDGL Data Grid Tier1 (FNAL) Proto-Tier2 Tier3 university Caltech/UCSD Florida Wisconsin Fermilab BNL Indiana BU Michigan Other sites to be added in 2002 SKC Brownsville Hampton PSU

46. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery46 iVDGL Map (2002-2003) Tier0/1 facility Tier2 facility 10 Gbps link 2.5 Gbps link 622 Mbps link Other link Tier3 facility DataTAG Surfnet

47. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery47 “Infrastructure” Data Grid Projects è GriPhyN (US, NSF)  Petascale Virtual-Data Grids  http://www.griphyn.org/ è Particle Physics Data Grid (US, DOE)  Data Grid applications for HENP  http://www.ppdg.net/ è European Data Grid (EC, EU)  Data Grid technologies, EU deployment  http://www.eu-datagrid.org/ è TeraGrid Project (US, NSF)  Dist. supercomp. resources (13 TFlops)  http://www.teragrid.org/ è iVDGL + DataTAG (NSF, EC, others)  Global Grid lab & transatlantic network  Collaborations of application scientists & computer scientists  Focus on infrastructure development & deployment  Broad application

48. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery48 Data Grid Project Timeline GriPhyN approved, $11.9M+$1.6M EU DataGrid approved, $9.3M 1 st Grid coordination meeting PPDG approved, $9.5M 2 nd Grid coordination meeting iVDGL approved, $13.65M+$2M TeraGrid approved ($53M) Q1 02 Q4 00 Q1 01 Q2 01 Q3 01 Q4 01 3 rd Grid coordination meeting 4 th Grid coordination meeting DataTAG approved (€4M) LHC Grid Computing Project

49. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery49 Need for Common Grid Infrastructure è Grid computing sometimes compared to electric grid  You plug in to get a resource (CPU, storage, …)  You don’t care where the resource is located Want to avoid this situation in Grid computing è This analogy is more appropriate than originally intended  It expresses a USA viewpoint  uniform power grid  What happens when you travel around the world? Different frequencies60 Hz, 50 Hz Different voltages120 V, 220 V Different sockets!USA, 2 pin, France, UK, etc.

50. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery50 Role of Grid Infrastructure è Provide essential common Grid services  Cannot afford to develop separate infrastructures (Manpower, timing, immediate needs, etc.) è Meet needs of high-end scientific & engin’g collaborations  HENP, astrophysics, GVO, earthquake, climate, space, biology, …  Already international and even global in scope  Drive future requirements è Be broadly applicable outside science  Government agencies: National, regional (EU), UN  Non-governmental organizations (NGOs)  Corporations, business networks (e.g., suppliers, R&D)  Other “virtual organizations” (see Anatomy of the Grid) è Be scalable to the Global level

51. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery51 Grid Coordination Efforts è Global Grid Forum (GGF)  www.gridforum.org  International forum for general Grid efforts  Many working groups, standards definitions  Next one in Toronto, Feb. 17-20 è HICB (High energy physics)  Represents HEP collaborations, primarily LHC experiments  Joint development & deployment of Data Grid middleware  GriPhyN, PPDG, TeraGrid, iVDGL, EU-DataGrid, LCG, DataTAG, Crossgrid  Common testbed, open source software model  Several meeting so far è New infrastructure Data Grid projects?  Fold into existing Grid landscape (primarily US + EU)

52. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery52 Summary è Data Grids will qualitatively and quantitatively change the nature of collaborations and approaches to computing è The iVDGL will provide vast experience for new collaborations è Many challenges during the coming transition  New grid projects will provide rich experience and lessons  Difficult to predict situation even 3-5 years ahead

53. UT Arlington Colloquium (Jan. 24, 2002)Paul Avery53 Grid References è Grid Book  www.mkp.com/grids è Globus  www.globus.org è Global Grid Forum  www.gridforum.org è TeraGrid  www.teragrid.org è EU DataGrid  www.eu-datagrid.org è PPDG  www.ppdg.net è GriPhyN  www.griphyn.org è iVDGL  www.ivdgl.org