Presented by NCCS Network Roadmap Josh Lothian High Performance Computing Operations National Center for Computational Sciences

2 Lothian_Networking_SC07 NCCS network roadmap  2007  Scaled Ethernet to meet wide-area needs and current day local- area data movement  Developed wire-speed, low-latency perimeter security to fully utilize 10Gb production and research WAN connections  Began IB LAN deployment  2008  Continue building out IB LAN infrastructure to satisfy the file system needs of the Baker system  Test Lustre on IB/WAN for possible deployment  Increase security on 10Gb WAN connections Summary: Hybrid Ethernet/Infiniband (IB) network to provide both high-speed wide-area connectivity and uber-speed local-area data movement

3 Lothian_Networking_SC07 Ethernet [O(10Gb/s)] Ethernet core scaled to match wide-area connectivity and archive Infiniband core scaled to match central file system and data transfer NCCS network roadmap summary Gateway Viz High-Performance Storage System (HPSS) ‏ Jaguar Lustre Baker Infiniband [O(100 Gb/s)]

4 Lothian_Networking_SC07 Juniper T640 Ciena CoreStream Juniper T320 Force10 E600 Cisco 6509 NSTG TeraGrid CCS DOE UltraScience/ NSF CHEETAH CCS OC-192 to TeraGrid 2 x OC-192 to USN OC-192 to CHEETAH NCCS WAN overview OC-192 to Internet 2 OC-192 to Esnet Internal 10Gb Link Internal 1Gb Link Ciena CoreDirector

5 Lothian_Networking_SC DDR Infiniband 6-12 DDR 64 DDR 24 SDR 87 SDR 6-12 DDR SDR 32 DDR 32 Ethernet 32 DDR Spider30 Spider10 Viz HPSS Devel E2E Jaguar NCCS Network 2007 Force10 E1200 Voltaire ISR2012 Cisco 7024D Voltaire ISR9288

6 Lothian_Networking_SC07 50 DDR 24 SDR 20 DDR 300 DDR (50 DDR/link)‏ 64 DDR 32 DDR 87 SDR Jaguar Viz HPSS E2E Devel Spider240Baker 50 DDR SDR (16-32 SDR/link)‏ NCCS IB network 2008 Spider10 Spider30 20 DDR 32 DDR Voltaire ISR2012 Voltaire ISR9288 Cisco 7024D

7 Lothian_Networking_SC milestones April 2007—Completed 10 G WAN upgrade June 2007—Completed VL/QOS testing November 2007—Complete fiber IB cable testing December 2007—IBoIP/SDP decision point

8 Lothian_Networking_SC milestones January—4x QDR testing February—Deploy flow-based perimeter security February—Complete 4X QDR testing March—Enable routing of XT3 HSN to production network June—Deploy redundant links between IB switches

9 Lothian_Networking_SC07 IB over WAN testing  Placed 2 x Obsidiam Longbow devices between Voltaire 9024 and Voltaire 9288  Provisioned loopback circuits of various lengths on the DOE UltraScience Network and ran test  RDMA Test Results: Local: 7.5 Gbps (Longbow to Longbow) ORNL ↔ ORNL (0.2 mile): 7.5 Gbps ORNL ↔ Chicago (1400 miles): 7.46 Gbps ORNL ↔ Seattle (6600 miles): 7.23 Gbps ORNL ↔ Sunnyvale (8600 miles): 7.2 Gbps 2081 Obsidian Longbow Ciena CD-CI (SNV)‏ 4x Infiniband SDR OC-192 SONET LustreEnd-to-End Ciena CD-CI (ORNL)‏ Obsidian Longbow Voltaire 9024 Voltaire 9288 DOE UltraScience Network

10 Lothian_Networking_SC07 Contact Josh Lothian High Performance Computing Operations National Center for Computational Sciences (865) Lothian_Networking_SC07