1. Network Power Users The Case for Jumbo Packets with WestGrid Examples

2. Research Rutherford Apparent Networks What are Jumbo Frames & What is 9k MTU?

3. Research Rutherford Apparent Networks 9k MTU - APP- DDS Project Evolution… 9k MTU Project Core R&E Router Troubleshooting Large Xfer Measurements International Links Internet 2 Sponsored Physics Participants Manual Lightpath TRIUMF to CERN Test Few 9k Taps onto Core Main Tap SDSC 4 yrs 9k APP Project Lightpath R&E Non-Routed… UCLP Build Dedicated 9k Links ZX GBICs/quad gigE/campus MM CANARIE NETERA BCNET 3 Universities Only 9k Link on Campus Physics & Biochem Participants HEPnet - WestGrid - TRIUMF 9k Node - HPC - 9k Node Viz HPC 9k Tests NFS HPC 9k Tests 0.5 yrs 9k DDS Project Lightpath R&E Clone-Tune 9k APP… Handoff Provide 9k Switch Biochem UVic - SFU - UofA Prelim 9k SOA Framework Drug Discovery System Flow Tune 9k SOA Framework Integrate 9k SOA Framework Demo 9k SOA-DDS Based Collab 9k SOA-DDS Software Dev Examples Complete Handoff … ;-) 2 yrs

4. Research Rutherford Apparent Networks 9K MTU Project - Results 512 MTU 2048 MTU 3072 MTU 4096 MTU 5120 MTU 6144 MTU 7168 MTU 8192 MTU 9000 MTU

5. Research Rutherford Apparent Networks How do Jumbo Packets Affect Bandwidth? If TCP window size and network capacity are not rate limiting factors then (roughly): 0.7 * Max Segment Size (MTU) e2e throughput < Round Trip Time (latency) sqrt[loss] M. Mathis, et.al. Double the MSS, double the throughput Effect of slow start? Effect of irregular flow?

6. Research Rutherford Apparent Networks 9k APP – Approaching Application Performance Need metrics for defining network impact on dependent applications Best current example – MOS as indication of VoIP performance Models of network dependence required Applicable to QoS/SLA

7. Research Rutherford Apparent Networks 9k APP – User Experience User expectations of applications Examples: Interaction with 3D models Collaboration with multiple models/data/voice/video Massive data set manipulation Collaborative HPC simulations

8. Research Rutherford Apparent Networks 9k APP - General Models of Network Dependency Near Real Time (nrt) Congestion, Drop, MTU & Transit time sensitive Transactional (tr) Congestion, MTU & Drop sensitive Data Transfer (dt) Congestion, Drop, MTU & Transit time sensitive Best Effort (be) Not sensitive

9. Research Rutherford Apparent Networks 9k APP – Real-Time Applications Examples – IPTV and Voice-over-IP Requirement – support human interaction through highly subjective perceptive processes Nature - asynchronous, constant, low- rate, non-TCP, streams Dependencies Highly sensitive to bursty loss Sensitive to latency, particular in conversation context Robust to jitter, up to some limit

10. Research Rutherford Apparent Networks 9k APP – Synchronous/Transactional Examples - interactive collaborative systems; distributed file systems Requirement - maintain some form of state at two or more remote locations; Nature - intensive, bursty, synchronous traffic; varies from very small amounts of data to huge exchanges Dependencies – requires high transfer rates, irregular; highly sensitive to latencies; intolerant of slow-start

11. Research Rutherford Apparent Networks 9k APP - Data Transfer Example – FTP; data backup; emergency recovery Requirement – transfer massive amounts of data, as quickly as possible Nature - sustained one-way flows at maximum rates of transfer Dependencies – sensitive to the characteristics of the end-host transmission protocols (i.e. TCP); high capacity; high impact on other flows

12. Research Rutherford Apparent Networks 9k APP - Best Effort Examples - e-mail, Web browsing, and remote login Requirement – sufficient resource to maintain minimal state or connection Nature - largely stateless; low rates of data transfer required; not gated by human response Dependencies – no critical requirements for network responsiveness

13. Research Rutherford Apparent Networks 9k APP - Choice of Application Performance Key performance factor – jumbo packets Previous 9k MTU testing applicable to simple data transfer use case Make the case for a more demanding 9k application performance category High performance: Interactive/collaborative visualization WestGrid visualization as exemplar Distributed file systems WestGrid Gridstore as exemplar

14. Research Rutherford Apparent Networks Application Performance in Context

15. Research Rutherford Apparent Networks 9k APP Project Components Three networks: Netera, CA*net4, BCNET

16. Research Rutherford Apparent Networks 9k APP Project Sites & Nodes Three sites: UofA, SFU, UVic Four nodes TRIUMF HEPnet Vizserver Gridstore Why these? Physics expertise 9k lightpath ready WestGrid applications vizserver gridstore

17. Research Rutherford Apparent Networks 9k APP Project - Define 9K paths Two phases: Viz & Grid Phase 1 Viz TRIUMF HEPnet Vizserver Phase 2 Grid TRIUMF HEPnet Gridstore

18. Research Rutherford Apparent Networks 9k APP Project - Phase 1 - Viz Vizserver session – optional compression IRIX OS – render pipe TCP: vary MTU 68 – 9000 at server Measure performance vs. MTU Vizclient session – wraps local openGL Fully interactive local X session refresh requests in render pipe VMD: Visual Molecular Dynamics Xserver & Graphics calls to render pipe Collaborative VMD sessions

19. Research Rutherford Apparent Networks 9k APP Project - Network Build 3 Lightpaths - UCLP - SONET Each path STM-24c dropped on gigE New quad gigE blade BCNET ONS 2 new ZX GBICs SM fibre run to SFU lit SFU Campus Network ZX GBIC installed in Enterasys ER16 Phase 1 - Viz MM fibre run direct to Cisco 6509 Phase 2 - Grid MM fibre run via Enterasys N7

20. Research Rutherford Apparent Networks 9k APP Project - Short Circuit Routing 1.5k 9k

21. Research Rutherford Apparent Networks 9k APP Project – Testing Procedure - pMTU Accumulated data from 9k MTU and 9k APP Projects

22. Research Rutherford Apparent Networks As MTU increases and increasing varies between hops determine optimal pMTU before and changes during application use locate problem hops with unusual behavior Is larger effective lower layer pMTU actually better from the application perspective? Are packets actually sized appropriately, by the packetization layer, given a larger pMTU? What are some effects of larger pMTU under congestion conditions? 9k APP Project – pMTU Issues

23. Research Rutherford Apparent Networks 9k APP Project - Phase 2 - Grid Distributed file system Possible candidates CXFS, GPFS, NFSv4 Possible clients/servers SGI IRIX, IBM AIX, Linux Preliminary model NFSv4 on Linux & IBM AIX(gridstore) Primary use cases File sharing – massive data sets Physics Bioinformatics

24. Research Rutherford Apparent Networks 9k APP Project - Grid NFS Application NFS session AIX NFSv4 server Linux NFSv4 clients over TCP not UDP NFS server – wraps local HPC file system Fully interactive local file system session Fast metadata updates for directory browsing

25. Research Rutherford Apparent Networks 9k APP Project – Grid Network & Testing Network reconfigured with intermediate link Viz performance rechecked Gridstore VLAN setup Fractional quad gigE Test via NFSTest suite (opensource) TCP: vary MTU 68 – 9000 at server Probe network after MTU alteration Measure performance vs. MTU Linux NFSv4 to AIX NFSv4 Probe while test in progress

26. Research Rutherford Apparent Networks 9k APP Project - Grid - NFS - Tuning Objectives derived from WestGrid gridstore Baseline performance – simple NFS client Session TCP - NFSv4 NFStest suite Time to complete vs. MTU Individual test performance vs. MTU Blocksize and other tuning considerations NFS filesystem mount options Block size = 8192 bytes Fragmentation factors Native filesystem block size

27. Research Rutherford Apparent Networks 9k APP Project Bill Rutherford (Rutherford Research/RRX – Project Coordinator) Loki Jorgenson (Apparent Networks/SFU – Project Coordinator) Thomas Tam (CANARIE/CA*net4 – CANARIE/UCLP Coordinator) Bryan Caron (TRIUMF/UofAlberta – TRIUMF/UCLP Coordinator) Randy Sobie (HEPnet/UVic – HEPnet President/Grid Integration) Brian Corrie (WestGrid/IRMACS/SFU - IRMACS Coordinator) Rob Ballantyne (IRMACS/SFU - IRMACS Network Coordinator) Martin Siegert (WestGrid/SFU – WestGrid/GridStore Coordinator) Dave Bickle (HEPnet/UVic – HEPnet Coordinator/Grid Integration) Ken Howard (Network Services/UVic – Network Coordinator) Peter van Epp (Network Services/SFU – Network Coordinator)

28. Research Rutherford Apparent Networks 9k DDS Project – Drug Discovery System Based on 9k APP Project Combined Physics, Grid, Bioinformatics Joint development of network & software Share network expertise Help develop preliminary software SOA approach Collaborative viz Distributed file systems Instrument interfaces Grid integration Lightpath integration

29. Research Rutherford Apparent Networks 9k DDS Project – Network Overview

30. Research Rutherford Apparent Networks Future – Performance Profiles by Application APP Network Performance Profile Build up statistical APP profiles Use APP profiles to optimize context Next Generation Router Design Use APP Profiles Allocate resources Design microflow queues Identify MTU issues Dynamically configure path mechanics

31. Research Rutherford Apparent Networks End of Presentation Note: 9k APP Project Meeting in Room 1535 at 3:00

32. Research Rutherford Apparent Networks BCNET ANC 2005 Outline…

33. Research Rutherford Apparent Networks Outline Short overview of previous 9K work What is 9k? 9k XXX Project snapshots? Bandwidth value … example data? How effect bandwidth? – equations? Application performance – definition User experience … limits? Near Real-time (nrt) Transactional (tr) Bulk transfer (bt) Best-effort (be) Value of MOS… expand? to the VoIP industry Project objectives Isolate a single simple performance factor – packet size? Identify prospective applications Interactive collaborative visualization (nrt) Distributed file system (tr + bt + nrt[metadata]) Characterize application performance in context … stack mechanics … grid integration?

34. Research Rutherford Apparent Networks Outline – cont. Project components Three sites: UofA, SFU, Uvic Why these sites … only 9k available + RTT factor? Three networks: BCNET, Netera, CA*net4 WestGrid applications Brian’s Westgrid vizserver… IRMACS Martin’s Westgrid grid storage Define 9K paths Application profiling Visualization server – phase 1 Identify system Define primary use case … why vmd … why collab? Define network profile … describe nw build? Identify testing procedure … pMTU tests … issues

35. Research Rutherford Apparent Networks Outline – cont. Distributed file system – phase 2 Identify candidates – CXFS, GPFS, NFSv3-4 Identify possible clients/servers Define primary use cases … physics … DDS file sharing Define network profile Identify testing procedure (NFStest) Basic types of test … ? Futures 9k DDS … trend to separate purpose nw… key role of UCLP Performance profiles by application … ??? Ng rtr … mtu issues … app microflow queues? Credits Meeting Reminder

36. Research Rutherford Apparent Networks BCNET ANC 9k APP Project Meeting Agenda – April 26 3-4pm Review of current status Lightpaths IRMACS Gridstore HEPnet TRIUMF Viz test Network Probes & schedule Tests Demos (incl special medicine collab demo UVic – UofA) Gridstore test Network Probes & schedule Tests Demos Follow on 9k DDS preliminary UCLP integration … ? SOA ideas… ?