1. High Energy Physics: Networks & Grids Systems for Global Science High Energy Physics: Networks & Grids Systems for Global Science Harvey B. Newman Harvey B. Newman California Institute of Technology AMPATH Workshop, FIU January 31, 2003 California Institute of Technology AMPATH Workshop, FIU January 31, 2003

2. Next Generation Networks for Experiments: Goals and Needs u Providing rapid access to event samples, subsets and analyzed physics results from massive data stores è From Petabytes by 2002, ~100 Petabytes by 2007, to ~1 Exabyte by ~2012. u Providing analyzed results with rapid turnaround, by coordinating and managing the large but LIMITED computing, data handling and NETWORK resources effectively u Enabling rapid access to the data and the collaboration è Across an ensemble of networks of varying capability u Advanced integrated applications, such as Data Grids, rely on seamless operation of our LANs and WANs è With reliable, monitored, quantifiable high performance Large data samples explored and analyzed by thousands of globally dispersed scientists, in hundreds of teams

3. 10 9 events/sec, selectivity: 1 in 10 13 (1 person in 1000 world populations) LHC: Higgs Decay into 4 muons (Tracker only); 1000X LEP Data Rate

4. Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*] u [*] BW Requirements Increasing Faster Than Moore’s Law See http://gate.hep.anl.gov/lprice/TAN

5. HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps Continuing the Trend: ~1000 Times Bandwidth Growth Per Decade; We are Rapidly Learning to Use and Share Multi-Gbps Networks

6. NL SURFnet GENEVA UK SuperJANET4 ABILENE ESNET CALREN 2 It GARR-B GEANT NewYork Fr INRIA STAR-TAP STARLIGHT DataTAG Project u EU-Solicited Project. CERN, PPARC (UK), Amsterdam (NL), and INFN (IT); and US (DOE/NSF: UIC, NWU and Caltech) partners u Main Aims:  Ensure maximum interoperability between US and EU Grid Projects  Transatlantic Testbed for advanced network research u 2.5 Gbps Wavelength Triangle from 7/02; to 10 Gbps Triangle by Early 2003 Wave Triangle 2.5 to 10G 10G Atrium VTHD

7. * Also see http://www-iepm.slac.stanford.edu/monitoring/bulk/; and the Internet2 E2E Initiative: http://www.internet2.edu/e2e Progress: Max. Sustained TCP Thruput on Transatlantic and US Links  8-9/01 105 Mbps 30 Streams: SLAC-IN2P3; 102 Mbps 1 Stream CIT-CERN  11/5/01 125 Mbps in One Stream (modified kernel): CIT-CERN  1/09/02 190 Mbps for One stream shared on 2 155 Mbps links  3/11/02 120 Mbps Disk-to-Disk with One Stream on 155 Mbps link (Chicago-CERN)  5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams  6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel)  9/02 850, 1350, 1900 Mbps Chicago-CERN 1,2,3 GbE Streams, OC48 Link  11-12/02 FAST: 940 Mbps in 1 Stream SNV-CERN; 9.4 Gbps in 10 Flows SNV-Chicago

8. FAST (Caltech): A Scalable, “Fair” Protocol for Next-Generation Networks: from 0.1 To 100 Gbps URL: netlab.caltech.edu/FAST SC2002 11/02 Experiment SunnyvaleBaltimore Chicago Geneva 3000km 1000km 7000km C. Jin, D. Wei, S. Low FAST Team & Partners Internet: distributed feedback system R f (s) R b ’ (s) p TCP AQM Theory 22.8.02 IPv6 9.4.02 1 flow 29.3.00 multiple Baltimore-Geneva Baltimore-Sunnyvale SC2002 1 flow SC2002 2 flows SC2002 10 flows I2 LSR Sunnyvale-Geneva  Standard Packet Size  940 Mbps single flow/GE card 9.4 petabit-m/sec 1.9 times LSR  9.4 Gbps with 10 flows 37.0 petabit-m/sec 6.9 times LSR  22 TB in 6 hours; in 10 flows Implementation  Sender-side (only) mods  Delay (RTT) based  Stabilized Vegas Highlights of FAST TCP Next: 10GbE; 1 GB/sec disk to disk

9. netlab.caltech.edu FAST TCP: Aggregate Throughput 1 flow 2 flows 7 flows 9 flows 10 flows Average utilization 95% 92% 90% 88% FAST  Standard MTU  Utilization averaged over > 1hr

10. HENP Lambda Grids: Fibers for Physics u Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data Stores u Survivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007) requires that each transaction be completed in a relatively short time. u Example: Take 800 secs to complete the transaction. Then Transaction Size (TB) Net Throughput (Gbps) Transaction Size (TB) Net Throughput (Gbps) 1 10 1 10 10 100 10 100 100 1000 (Capacity of Fiber Today) 100 1000 (Capacity of Fiber Today) u Summary: Providing Switching of 10 Gbps wavelengths within ~3-5 years; and Terabit Switching within 5-8 years would enable “Petascale Grids with Terabyte transactions”, as required to fully realize the discovery potential of major HENP programs, as well as other data-intensive fields.

11. Grids NOW  Efficient sharing of distributed heterogeneous compute and storage resources  Virtual Organizations and Institutional resource sharing  Dynamic reallocation of resources to target specific problems  Collaboration-wide data access and analysis environments  Grid solutions NEED to be scalable & robust  Must handle many petabytes per year  Tens of thousands of CPUs  Tens of thousands of jobs  Grid solutions presented here are supported in part by the GriPhyN, iVDGL, PPDG, EDG, and DataTag  We are learning a lot from these current efforts  For Example 1M Events Processed using VDT Oct.-Dec. 2002 Data Intensive Grids Now: Large Scale Production

12. Beyond Peoduction: Web Services for Ubiquitous Data Access and Analysis by a Worldwide Scientific Community  Web Services: easy, flexible, platform-independent access to data (Object Collections in Databases)  Well-adapted to use by individual physicists, teachers & students  SkyQuery Example JHU/FNAL/Caltech with Web Service based access to astronomy surveys  Can be individually or simultaneously queried via Web interface  Simplicity of interface hides considerable server power (from stored procedures etc.)  This is a “Traditional” Web Service, with no user authentication required

13. COJAC: CMS ORCA Java Analysis Component: Java3D Objectivity JNI Web Services Demonstrated Caltech-Rio de Janeiro and Chile in 2002

14. u Grid projects have been a step forward for HEP and LHC: a path to meet the “LHC Computing” challenges  “Virtual Data” Concept: applied to large-scale automated data processing among worldwide-distributed regional centers u The original Computational and Data Grid concepts are largely stateless, open systems: known to be scalable  Analogous to the Web u The classical Grid architecture has a number of implicit assumptions  The ability to locate and schedule suitable resources, within a tolerably short time (i.e. resource richness)  Short transactions; Relatively simple failure modes u HEP Grids are data-intensive and resource constrained  Long transactions; some long queues  Schedule conflicts; policy decisions; task redirection  A Lot of global system state to be monitored+tracked LHC Distributed CM: HENP Data Grids Versus Classical Grids

15. Current Grid Challenges: Secure Workflow Management and Optimization uMaintaining a Global View of Resources and System State èCoherent end-to-end System Monitoring èAdaptive Learning: new algorithms and strategies for execution optimization (increasingly automated) uWorkflow: Strategic Balance of Policy Versus Moment-to-moment Capability to Complete Tasks èBalance High Levels of Usage of Limited Resources Against Better Turnaround Times for Priority Jobs èGoal-Oriented Algorithms; Steering Requests According to (Yet to be Developed) Metrics uHandling User-Grid Interactions: Guidelines; Agents uBuilding Higher Level Services, and an Integrated Scalable User Environment for the Above

16. Distributed System Services Architecture (DSSA): CIT/Romania/Pakistan u Agents: Autonomous, Auto- discovering, self-organizing, collaborative u “Station Servers” (static) host mobile “Dynamic Services” u Servers interconnect dynamically; form a robust fabric in which mobile agents travel, with a payload of (analysis) tasks u Adaptable to Web services: OGSA; and many platforms u Adaptable to Ubiquitous, mobile working environments StationServer StationServer StationServer LookupService LookupService Proxy Exchange Registration Service Listener Lookup Discovery Service Remote Notification Managing Global Systems of Increasing Scope and Complexity, In the Service of Science and Society, Requires A New Generation of Scalable, Autonomous, Artificially Intelligent Software Systems

17. By I. Legrand (Caltech)  Deployed on US CMS Grid  Agent-based Dynamic information / resource discovery mechanism  Talks w/Other Mon. Systems  Implemented in  Java/Jini; SNMP  WDSL / SOAP with UDDI  Part of a Global “ Grid Control Room ” Service MonaLisa: A Globally Scalable Grid Monitoring System

18. MONARC SONN: 3 Regional Centres “Learning” to Export Jobs (Day 9) NUST 20 CPUs CERN 30 CPUs CALTECH 25 CPUs 1MB/s ; 150 ms RTT 1.2 MB/s 150 ms RTT 0.8 MB/s 200 ms RTT Optimized: Day = 9 = 0.73 = 0.66 = 0.83

19.  Develop and build Dynamic Workspaces  Build Private Grids to support scientific analysis communities  Using Agent Based Peer-to-peer Web Services  Construct Autonomous Communities Operating Within Global Collaborations  Empower small groups of scientists (Teachers and Students) to profit from and contribute to int’l big science  Drive the democratization of science via the deployment of new technologies NSF ITR: Globally Enabled Analysis Communities

20. Private Grids and P2P Sub- Communities in Global CMS

21. 14600 Host Devices; 7800 Registered Users in 64 Countries 45 Network Servers Annual Growth 2 to 3X

22. An Inter-Regional Center for Research, Education and Outreach, and CMS CyberInfrastucture Foster FIU and Brazil (UERJ) Strategic Expansion Into CMS Physics Through Grid-Based “Computing” uDevelopment and Operation for Science of International Networks, Grids and Collaborative Systems èFocus on Research at the High Energy frontier uDeveloping a Scalable Grid-Enabled Analysis Environment èBroadly Applicable to Science and Education èMade Accessible Through the Use of Agent-Based (AI) Autonomous Systems; and Web Services uServing Under-Represented Communities èAt FIU and in South America èTraining and Participation in the Development of State of the Art Technologies èDeveloping the Teachers and Trainers uRelevance to Science, Education and Society at Large èDevelop the Future Science and Info. S&E Workforce èClosing the Digital Divide