1. Presented by CCS Network Roadmap Steven M. Carter Network Task Lead Center for Computational Sciences

2. 2 Carter_LCF Networking_0611 CCS network roadmap  FY 2006–FY 2007  Scaled Ethernet to meet wide-area needs and current day local-area data movement  Developed wire-speed, low-latency perimeter security to fully utilize 10 G production and research WAN connections  Began IB LAN deployment  FY 2007–FY 2008  Continue building out IB LAN infrastructure to satisfy the file system needs of the Baker system  Test Lustre on IB/WAN for possible deployment Summary: Hybrid Ethernet/Infiniband (IB) network to provide both high-speed wide-area connectivity and uber-speed local-area data movement

3. 3 Carter_LCF Networking_0611 Ethernet [O(10GB/s)] Ethernet core scaled to match wide-area connectivity and archive Infiniband core scaled to match central file system and data transfer NCCS network roadmap summary Viz High-Performance Storage System (HPSS) Jaguar Lustre Baker Infiniband [O(100GB/s)] Gateway

4. 4 Carter_LCF Networking_0611 Juniper T640 Ciena CoreStream Juniper T320 Force10 E600 Cisco 6509 NSTG TeraGrid CCS DOE UltraScience/ NSF CHEETAH CCS OC-192 to TeraGrid 2 x OC-192 to USN OC-192 to CHEETAH CCS WAN overview OC-192 to Internet 2 OC-192 to Esnet Internal 10G Link Internal 1G Link Ciena CoreDirector

5. 5 Carter_LCF Networking_0611 4 4 4 20 48 128 DDR Infiniband 64 DDR 24 SDR 87 SDR 3 SDR 64 DDR 20 SDR 32 DDR 32 Ethernet 48-96 SDR Jaguar Spider60 Viz HPSS Devel E2E Spider10 CCS Network 2007

6. 6 Carter_LCF Networking_0611 50 DDR 24 SDR20 SDR 300 DDR (50 DDR/link) 64 DDR 32 DDR 87 SDR Jaguar Viz HPSS E2E DevelSpider10 Spider240Baker 50 DDR 48-96 SDR (16-32 SDR/link) CCS IB network 2008

7. 7 Carter_LCF Networking_0611 FY 2007 milestones Nov 2006 - Completed IB WAN testing Jan 2007 - Completed 10-G WAN upgrade Jan 2007 - IBoIP firewall installation Mar 2007 - Force10 P10 upgrade Apr 2007 - Completed VL/QOS testing Jun 2007 - IBoIP/SDP decision point

8. 8 Carter_LCF Networking_0611 IB over WAN testing  Placed 2 x Obsidiam Longbow devices between Voltaire 9024 and Voltaire 9288  Provisioned loopback circuits of various lengths on the DOE UltraScience Network and ran test.  RDMA Test Results: Local: 7.5Gbps (Longbow to Longbow) ORNL ORNL (0.2mile): 7.5Gbps ORNL Chicago (1400miles): 7.46Gbps ORNL Seattle (6600 miles): 7.23Gbps ORNL Sunnyvale (8600 miles): 7.2Gbps 2081 Obsidian Longbow Ciena CD-CI (SNV) 4x Infiniband SDR OC-192 SONET LustreEnd-to-End Ciena CD-CI (ORNL) Obsidian Longbow Voltaire 9024 Voltaire 9288 DOE UltraScience Network

9. 9 Carter_LCF Networking_0611 Spider (Linux Cluster) Rizzo (XT3) Voltaire 9024 Utilizing a server on spider (a commodity x86_64 Linux cluster), we were able to show the first successful infiniband implementation on the XT3. The basic RDMA test (as provided by the Open Fabrics Enterprise Distribution), allows us to see the maximum bandwidth that we could achieve (about 900MB/s unidirectionally) from the XT3's 133MHz PCI-X bus. Network Task Lead

10. 10 Carter_LCF Networking_0611 Contact Steven Carter Tech Integration Center for Computational Sciences (865) 576-2672 scarter@ornl.gov 10 Presenter_date