High Energy Physics: Networks & Grids Systems for Global Science High Energy Physics: Networks & Grids Systems for Global Science Harvey B. Newman Harvey B. Newman California Institute of Technology AMPATH Workshop, FIU January 31, 2003 California Institute of Technology AMPATH Workshop, FIU January 31, 2003

Next Generation Networks for Experiments: Goals and Needs u Providing rapid access to event samples, subsets and analyzed physics results from massive data stores è From Petabytes by 2002, ~100 Petabytes by 2007, to ~1 Exabyte by ~2012. u Providing analyzed results with rapid turnaround, by coordinating and managing the large but LIMITED computing, data handling and NETWORK resources effectively u Enabling rapid access to the data and the collaboration è Across an ensemble of networks of varying capability u Advanced integrated applications, such as Data Grids, rely on seamless operation of our LANs and WANs è With reliable, monitored, quantifiable high performance Large data samples explored and analyzed by thousands of globally dispersed scientists, in hundreds of teams

10 9 events/sec, selectivity: 1 in (1 person in 1000 world populations) LHC: Higgs Decay into 4 muons (Tracker only); 1000X LEP Data Rate

Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*] u [*] BW Requirements Increasing Faster Than Moore’s Law See

HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps Continuing the Trend: ~1000 Times Bandwidth Growth Per Decade; We are Rapidly Learning to Use and Share Multi-Gbps Networks

NL SURFnet GENEVA UK SuperJANET4 ABILENE ESNET CALREN 2 It GARR-B GEANT NewYork Fr INRIA STAR-TAP STARLIGHT DataTAG Project u EU-Solicited Project. CERN, PPARC (UK), Amsterdam (NL), and INFN (IT); and US (DOE/NSF: UIC, NWU and Caltech) partners u Main Aims:  Ensure maximum interoperability between US and EU Grid Projects  Transatlantic Testbed for advanced network research u 2.5 Gbps Wavelength Triangle from 7/02; to 10 Gbps Triangle by Early 2003 Wave Triangle 2.5 to 10G 10G Atrium VTHD

* Also see and the Internet2 E2E Initiative: Progress: Max. Sustained TCP Thruput on Transatlantic and US Links  8-9/ Mbps 30 Streams: SLAC-IN2P3; 102 Mbps 1 Stream CIT-CERN  11/5/ Mbps in One Stream (modified kernel): CIT-CERN  1/09/ Mbps for One stream shared on Mbps links  3/11/ Mbps Disk-to-Disk with One Stream on 155 Mbps link (Chicago-CERN)  5/20/ Mbps SLAC-Manchester on OC12 with ~100 Streams  6/1/ Mbps Chicago-CERN One Stream on OC12 (mod. Kernel)  9/02 850, 1350, 1900 Mbps Chicago-CERN 1,2,3 GbE Streams, OC48 Link  11-12/02 FAST: 940 Mbps in 1 Stream SNV-CERN; 9.4 Gbps in 10 Flows SNV-Chicago

FAST (Caltech): A Scalable, “Fair” Protocol for Next-Generation Networks: from 0.1 To 100 Gbps URL: netlab.caltech.edu/FAST SC /02 Experiment SunnyvaleBaltimore Chicago Geneva 3000km 1000km 7000km C. Jin, D. Wei, S. Low FAST Team & Partners Internet: distributed feedback system R f (s) R b ’ (s) p TCP AQM Theory IPv flow multiple Baltimore-Geneva Baltimore-Sunnyvale SC flow SC flows SC flows I2 LSR Sunnyvale-Geneva  Standard Packet Size  940 Mbps single flow/GE card 9.4 petabit-m/sec 1.9 times LSR  9.4 Gbps with 10 flows 37.0 petabit-m/sec 6.9 times LSR  22 TB in 6 hours; in 10 flows Implementation  Sender-side (only) mods  Delay (RTT) based  Stabilized Vegas Highlights of FAST TCP Next: 10GbE; 1 GB/sec disk to disk

netlab.caltech.edu FAST TCP: Aggregate Throughput 1 flow 2 flows 7 flows 9 flows 10 flows Average utilization 95% 92% 90% 88% FAST  Standard MTU  Utilization averaged over > 1hr

HENP Lambda Grids: Fibers for Physics u Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data Stores u Survivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007) requires that each transaction be completed in a relatively short time. u Example: Take 800 secs to complete the transaction. Then Transaction Size (TB) Net Throughput (Gbps) Transaction Size (TB) Net Throughput (Gbps) (Capacity of Fiber Today) (Capacity of Fiber Today) u Summary: Providing Switching of 10 Gbps wavelengths within ~3-5 years; and Terabit Switching within 5-8 years would enable “Petascale Grids with Terabyte transactions”, as required to fully realize the discovery potential of major HENP programs, as well as other data-intensive fields.

Grids NOW  Efficient sharing of distributed heterogeneous compute and storage resources  Virtual Organizations and Institutional resource sharing  Dynamic reallocation of resources to target specific problems  Collaboration-wide data access and analysis environments  Grid solutions NEED to be scalable & robust  Must handle many petabytes per year  Tens of thousands of CPUs  Tens of thousands of jobs  Grid solutions presented here are supported in part by the GriPhyN, iVDGL, PPDG, EDG, and DataTag  We are learning a lot from these current efforts  For Example 1M Events Processed using VDT Oct.-Dec Data Intensive Grids Now: Large Scale Production

Beyond Peoduction: Web Services for Ubiquitous Data Access and Analysis by a Worldwide Scientific Community  Web Services: easy, flexible, platform-independent access to data (Object Collections in Databases)  Well-adapted to use by individual physicists, teachers & students  SkyQuery Example JHU/FNAL/Caltech with Web Service based access to astronomy surveys  Can be individually or simultaneously queried via Web interface  Simplicity of interface hides considerable server power (from stored procedures etc.)  This is a “Traditional” Web Service, with no user authentication required

COJAC: CMS ORCA Java Analysis Component: Java3D Objectivity JNI Web Services Demonstrated Caltech-Rio de Janeiro and Chile in 2002

u Grid projects have been a step forward for HEP and LHC: a path to meet the “LHC Computing” challenges  “Virtual Data” Concept: applied to large-scale automated data processing among worldwide-distributed regional centers u The original Computational and Data Grid concepts are largely stateless, open systems: known to be scalable  Analogous to the Web u The classical Grid architecture has a number of implicit assumptions  The ability to locate and schedule suitable resources, within a tolerably short time (i.e. resource richness)  Short transactions; Relatively simple failure modes u HEP Grids are data-intensive and resource constrained  Long transactions; some long queues  Schedule conflicts; policy decisions; task redirection  A Lot of global system state to be monitored+tracked LHC Distributed CM: HENP Data Grids Versus Classical Grids

Current Grid Challenges: Secure Workflow Management and Optimization uMaintaining a Global View of Resources and System State èCoherent end-to-end System Monitoring èAdaptive Learning: new algorithms and strategies for execution optimization (increasingly automated) uWorkflow: Strategic Balance of Policy Versus Moment-to-moment Capability to Complete Tasks èBalance High Levels of Usage of Limited Resources Against Better Turnaround Times for Priority Jobs èGoal-Oriented Algorithms; Steering Requests According to (Yet to be Developed) Metrics uHandling User-Grid Interactions: Guidelines; Agents uBuilding Higher Level Services, and an Integrated Scalable User Environment for the Above

Distributed System Services Architecture (DSSA): CIT/Romania/Pakistan u Agents: Autonomous, Auto- discovering, self-organizing, collaborative u “Station Servers” (static) host mobile “Dynamic Services” u Servers interconnect dynamically; form a robust fabric in which mobile agents travel, with a payload of (analysis) tasks u Adaptable to Web services: OGSA; and many platforms u Adaptable to Ubiquitous, mobile working environments StationServer StationServer StationServer LookupService LookupService Proxy Exchange Registration Service Listener Lookup Discovery Service Remote Notification Managing Global Systems of Increasing Scope and Complexity, In the Service of Science and Society, Requires A New Generation of Scalable, Autonomous, Artificially Intelligent Software Systems

By I. Legrand (Caltech)  Deployed on US CMS Grid  Agent-based Dynamic information / resource discovery mechanism  Talks w/Other Mon. Systems  Implemented in  Java/Jini; SNMP  WDSL / SOAP with UDDI  Part of a Global “ Grid Control Room ” Service MonaLisa: A Globally Scalable Grid Monitoring System

MONARC SONN: 3 Regional Centres “Learning” to Export Jobs (Day 9) NUST 20 CPUs CERN 30 CPUs CALTECH 25 CPUs 1MB/s ; 150 ms RTT 1.2 MB/s 150 ms RTT 0.8 MB/s 200 ms RTT Optimized: Day = 9 = 0.73 = 0.66 = 0.83

 Develop and build Dynamic Workspaces  Build Private Grids to support scientific analysis communities  Using Agent Based Peer-to-peer Web Services  Construct Autonomous Communities Operating Within Global Collaborations  Empower small groups of scientists (Teachers and Students) to profit from and contribute to int’l big science  Drive the democratization of science via the deployment of new technologies NSF ITR: Globally Enabled Analysis Communities

Private Grids and P2P Sub- Communities in Global CMS

14600 Host Devices; 7800 Registered Users in 64 Countries 45 Network Servers Annual Growth 2 to 3X

An Inter-Regional Center for Research, Education and Outreach, and CMS CyberInfrastucture Foster FIU and Brazil (UERJ) Strategic Expansion Into CMS Physics Through Grid-Based “Computing” uDevelopment and Operation for Science of International Networks, Grids and Collaborative Systems èFocus on Research at the High Energy frontier uDeveloping a Scalable Grid-Enabled Analysis Environment èBroadly Applicable to Science and Education èMade Accessible Through the Use of Agent-Based (AI) Autonomous Systems; and Web Services uServing Under-Represented Communities èAt FIU and in South America èTraining and Participation in the Development of State of the Art Technologies èDeveloping the Teachers and Trainers uRelevance to Science, Education and Society at Large èDevelop the Future Science and Info. S&E Workforce èClosing the Digital Divide