1. Rick Cavanaugh, University of Florida CHEP06 Mumbai, 13 February, 2006 An Ultrascale Information Facility for Data Intensive Research

2. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India2 The UltraLight Collaboration California Institute of Technology University of Michigan University of Florida Florida International University Internet2 Fermilab Brookhaven SLAC University of California, San Diego Massachusetts Institute of Technology Boston University University of California, Riverside UCAID

3. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India3 The Project UltraLight is –A four year $2M NSF ITR funded by MPS –Application driven Network R&D Two Primary, Synergistic Activities –Network “Backbone”: Perform network R&D / engineering –Applications “Driver”: System Services R&D / engineering Ultimate goal : Enable physics analysis and discoveries which could not otherwise be achieved

4. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India4 The Motivation Ability to rapidly transport large datasets will strongly impact computing models –Datasets (used for analysis) no longer need be pinned for long periods –SE’s more willing to grant greater temporary storage –Opportunistic use of volatile (non-VO controlled) resources enhanced –Particularly important in resource over- subscribed environments

5. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India5 Expose the Network as an Actively Managed Resource Based on a “Hybrid” packet- and circuit-switched optical network infrastructure –Ultrascale Protocols (e.g. FAST) and Dynamic Optical Paths Monitor, Manage and Optimize resources in real-time –Using a set of Agent-Based Intelligent Global Services Leverages already-existing, developing software infrastructure in round-the-clock operation: –MonALISA, GAE/Clarens, OSG Exceptional Support from –Industry: Cisco & Calient –Research community: NLR, CENIC, Internet2/Abilene, ESnet A New Class of Integrated Information Systems See talk from S. McKee

6. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India6 UltraLight Activities TEAMS: Physicists, Computer Scientists, Network Engineers High Energy Physics Application Services –Integrate and Develop physics applications into the UltraLight Fabric: Production Codes, Grid-enabled analysis, User Interfaces to Fabric Global System Services –Critical “Upperware” software components in the UltraLight Fabric: Monitoring, Scheduling, Agent-based Services, etc. Network Engineering –Routing, Switching, Dynamic Path Construction Ops., Management Testbed Deployment and Operations –Including Optical Network, Compute Cluster, Storage, Kernel and UltraLight System Software Configs. Education and Outreach

7. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India7 Project Structure Steering Committee Overall Project Guidance Management Team Day-to-day Internal and External Coordination Technical Groups Day-to-day Activities and Operations Network Applications Education & Outreach User Community External Projects ATLAS, CMS, DISUN, LCG, OSG, TeraPaths, CHEPREO, AMPATH, KyraTera, … External Peering Partners NLR, ESNet, USNet, LHCNet, HOPI, TeraGrid, Pacific Wave, WIDE, AARNet, Brazil/HEPGrid CA*net4, GLORIAD, IEEAF JGN2, KEK, Korea, NetherLIght, TIFR, UKLight/ESLEA, …

8. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India8 Main Science Driver: The LHC 10 -0 10 3 6 sec 25ns3µs hour yearms Reconstruction&ANALYSIS TIER0/1/2 Centers ON-line OFF-line sec GigaTera Petabit H  2Z  4l

9. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India9 10 -0 10 3 6 sec 25ns3µs hour yearms Reconstruction&ANALYSIS TIER0/1/2 Centers ON-line OFF-line sec GigaTera Petabit New Physics Searches  multi-Terabyte scale Datasets! H  2Z  4l Main Science Driver: The LHC Requests from Multiple users for Multiple types of… …Multiple times! Individual TB transactions should finish in minutes to hours, rather than hours to days

10. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India10 Science Areas 2005 End2End Throughput 5 years End2End Throughput 5-10 Years End2End Throughput Remarks High Energy Physics 0.5 Gb/s100 Gb/s 1000 Gb/s High bulk throughput Climate (Data & Computation) 0.5 Gb/s160-200 Gb/s N x 1000 Gb/s High bulk throughput SNS NanoScience Not yet started 1 Gb/s1000 Gb/s + QoS for Control Channel Remote control and time critical throughput Fusion Energy0.066 Gb/s (500 MB/s burst) 0.198 Gb/s (500MB/ 20 sec. burst) N x 1000 Gb/s Time critical throughput Astrophysics0.013 Gb/s (1 TByte/week) N*N multicast 1000 Gb/s Computat’l steering and collaborations Genomics Data & Computation 0.091 Gb/s (1 TBy/day) 100s of users1000 Gb/s + QoS for Control Channel High throughput and steering Evolving Science Requirements for Networks (DOE High Perf. Network Workshop) Slide taken from H. Newman

11. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India11 Ever-increasing Network Flows Amsterdam Internet Exchange Point ES Net Total Traffic Jan 2006 120+ Gbits/sec Now: Should be at Petabyte/month These two examples are representative of the trend in research and education networks worldwide ATLAS/CMS data flows are “in the ballpark” in comparison

12. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India12 Project Scope and Context Application Frameworks Augmented to Interact Effectively with the Global Services (GS) –GS Interact in Turn with the Storage Access & Local Execution Service Layers Apps. Provide Hints to High-Level Services About Requirements –Interfaced also to managed Net and Storage services –Allows effective caching, pre- fetching; opportunities for global and local optimization of thru-put Make the Network an Integrated Managed Resource a la CPU & Storage P h E D E x

13. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India13 GAE and Ultralight Make UltraLight available to Physics applications and their environments Unpredictable multi user analysis Overall demand typically fills the capacity of the resources Real time monitor systems for networks, storage, computing resources,… : E2E monitoring Network Resources Network Planning Request Planning Monitor Application Interfaces Support data transfers ranging from predictable movement of large scale (simulated and real) data, to highly dynamic analysis tasks initiated by rapidly changing teams of scientists

14. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India14 Network Resource Testbed See talk from S. McKee

15. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India15 Global Network Planning Services VINCI : –Virtual Intelligent Networks for Computing Infrastructures –Based on existing MonALISA framework LISA : –Localhost Information Service Agent –Monitors end-systems User servers See talk from I. Legrand

16. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India16 Prototype App. Layer: E2E ATLAS/CMS Software stacks are complex and still developing –Integration work is challenging & constantly evolving Generic Service Oriented Architecture crucial for integration –Catalogs to select datasets, –Resource & Application Discovery –Schedulers guide jobs to resources –Policies enable “fair” access to resources –Robust (large size) data (set) transfer 8 Client Application Discovery Planner/ Scheduler Monitor Information Policy Steering Catalogs Job Submission Storage Management Execution 1 2 3 4 5 5 6 7 Dataset service 9 Data Transfer Ultralight Focus : data transfer, planning scheduling, (sophisticated) policy management on VO level, integration

17. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India17 Supercomputing 2005 Internet Land Speed Record 151 Gbps peak rate 100+ Gbps sustained throughput for hours 475 Terabytes of physics data transported in less than 24 hours Sustained rate of 100+ Gbps translates to greater than 1 Petabyte per day 02412618 t [hours] 060301545 t [min] Cumulative [TB] Rate [Gbs]

18. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India18 Project Milestones High-level Milestones –Link critical services and applications together –Multiple-services & -clients –Distributed system, some fault-tolerance –Logical grid (physical details hidden) –Strategic Steering of Work/Dataflows –Self-organizing, robust distributed E2E system User Adoption –Identify small community of users (some within UL) –Integrate UL services seamlessly with LHC Environ. –Deliver LHC Physics Analyses within LHC Timeline

19. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India19 UltraLight Plans UltraLight is a 4 year program delivering a new, high- performance, network-integrated infrastructure: Phase I (12 months) 2004-2005: focused on deploying initial network infrastructure & bringing up first services Phase II (18 months) 2005-2006: concentrates on implementing all the needed services & extending the infrastructure to additional sites Phase III (18 months) 2007-2008: will focus on a transition to production in support of LHC Physics

20. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India20 Beyond UltraLight: PLaNetS – Physics Lambda Network System

21. 13.02.2006R. Cavanaugh, CHEP06, Mumbai, India21 Summary For many years the WAN has been the bottleneck; This no longer the case in many countries –Deployment of Grid infrastructure now a reality! –Recent land-speed records  network can be truly transparent throughputs limited by end-hosts –Challenge shifting from getting adequate bandwidth to deploying adequate infrastructure to make effective use of it! UltraLight is delivering a critical missing component for future eScience: the integrated, managed network Extend and augment existing grid infrastructures (currently focused on CPU/storage) to include the network as an integral component