1. Paul Grun System Fabric Works pgrun@systemfabricworks.com 4/7/08

2. 2  Channel I/O – an embarrassment of riches  Winning in the low latency space, marking time in the commercial sector  Never bet against Ethernet  Moving our customers forward  Conclusion

3. 3 It all started with VIA VIA gave us the concept of stack bypass  This gave us the makings of a low latency interconnect.  Perfectly suited for, among other things, clusters.

4. 4 From VIA sprang InfiniBand (after a few false starts) An efficient transport Network and phy layers Virtual lanes A mgmt infrastructure Std methods for accessing these services … In short, the ability to efficiently conduct multiple traffic flows over a single wire Interesting…the basics for a unified fabric

5. 5 Equally interesting is this… Beyond a low latency unified fabric, we also gained the ability to directly connect virtual address spaces located in disjoint physical addresses - RDMA

6. 6 Tailor-made for virtualization Datacenter virtualization - applications connected to pools of virtualized resources VM App VM App Server virtualization- Platform resources are shared among the ‘containers’ within the platform VMMVMM … VM app VM app Better yet…more efficient use of resources improves the ‘green footprint’…dramatically

7. 7 High bandwidth, no packet dropping…all we would need would be a storage protocol and we’d have the perfect fabric. Enter SRP, followed a little later by iSER. Voila, IB is an ideal storage interconnect!

8. 8 Eventually, we arrived at the crux of the issue. What really matters is the way an application communicates with other applications and storage. This is the notion of Channel I/O. OFA emerged as the keeper and developer of the Channel Interface

9. 9 Application transport network Application transport network switch phy switch phy Channel I/O is about creating pipes Channel i/f It’s very much an application-centric view…what matters is how the application communicates.

10. 10 As the industry began to focus on Channel I/O, the underlying wire seemed to become progressively less important. Someone noticed that it made sense to define RDMA over TCP/IP/Ethernet…iWARP emerged and became a member of the family of Channel I/O protocols. Now we’re really cooking with gas.

11. 11 This is looking pretty good! So far, we’ve got:  The world’s lowest latency standards-based interconnect  An incredible foundation for virtualization  both server and data center virtualization  a native storage solution based on standard SCSI classes  wire independence  a unified fabric  An open Channel Interface for both Linux and Windows platforms  It works for legacy apps, and for those plucky native apps  And it’s cheap to boot! (no pun intended) I mean, is this nirvana, or what???

12. 12 With this embarrassment of riches, we can address some of the world’s most vexing computer science problems  The world’s fastest/cheapest supercomputers  The world’s lowest price/perf clusters  The world’s most flexible/agile datacenters  Highly energy efficient data centers The world is our oyster, right?

13. 13 So how come we’re not all rich?

14. 14 The box score so far looks like: Naturally most of the gold is in that last bucket IBEthernet Supercomputers  HPC, clusters HPC-like portions of the commercial space EVERYTHING Else

15. 15  It’s interesting that both supercomputing and HPC/clustering are primarily ‘single fabric’ environments…  With a few exceptions, these installations are purpose-built from the ground up, using a single fabric But what about the commercial spaces?

16. 16 Commercial datacenters, OTOH  …tend to be built on top of a combination of fabrics,  …tend to include huge application investments,  …tend to involve huge amounts of existing infrastructure  …tend to rely on high volume, commodity OTS hardware These are some pretty large rocks to roll uphill

17. 17 (Of course, the commercial space isn’t monolithic…clearly, there are environments where the calculus of channel I/O produces a positive ROI.)

18. 18 What about virtualization? Isn’t that a key driver for the commercial space? Server virtualization, which doesn’t depend on channel I/O, is doing very nicely at driving up utilization, and helping reduce the physical space requirements. Great progress is being made here, with or without channel I/O.

19. 19 How about the green datacenter? There is some promise here, but compared to the cost of migrating a massive investment to a greener approach, the pain will have to get *alot* higher before channel I/O grabs a foothold* *yes, there are certainly spots where the pain threshold is indeed a lot higher.

20. 20 Unified fabrics? Doesn’t OpEx/TCO conquer all? Well…

21. 21 “Nobody ever lost a bet on Ethernet” Ethernet continues to chug along… The Ethernet community is thinking about a lossless wire  Congestion management reducing the burden on the transport, thus reducing the traditional achilles heel of TCP/IP  Virtual lanes to support multiple streams on a single fabric This is starting to sound like a credible converged fabric…

22. 22 … a converged fabric which is aimed squarely at the commercial space…i.e., those environments which can easily harvest the TCO benefits of a converged fabric.

23. 23 Application transport network switch phy It’s a completely pragmatic, ‘wire-centric’ approach…intended to make the wire sufficiently capable such that it will support multiple streams of traffic. A single-minded focus on creating a unified wire.

24. 24 Nobody believes that Ethernet will ever go away. So we have a few choices. 1.Cede the converged fabric, commercial multi-protocol space to Ethernet’s emerging enhancements. 2.Battle it out for ownership of the converged fabric. And probably lose. 3.Look for niche enclaves in the enterprise where IB dominates - the ‘toehold’ strategy. 4.Drive channel I/O in above the wire. How?

25. 25 1. Don’t depend on the emergence of a ‘pure IB’ environment, it won’t happen (except in HPC). Instead… Leverage the heavy lifting Ethernet is proposing to do by reducing the impedance mismatch between networks wherever possible. How about an IB transport on Ethernet? LRHGRHBTHETHpayload MACIPv6TCPpayload (not as simple as that of course, but maybe worth a look…)

26. 26 2. Enable IB in every toehold that exists in the data center filesystem transport switch phy FC-4 SCSI Make sure an application can use the channel I/O interface to access system resources, regardless of the protocol. FCoIB, anyone?

27. 27 3. Reduce impediments to accessing the features of the IB wire IB already has the features now being proposed by the Ethernet community, in spades. Let’s reduce the impediments to accessing those features. Is there any good reason for continuing to support different bit rates/encoding schemes?

28. 28 Apply industry leadership to make sure our customers aren’t swamped in a quagmire of I/O protocols, wire protocols, wires. Remember that a key attraction of a converged fabric is its simplicity.

29. 29 We need to ensure that the promise of channel I/O can be delivered…no matter what the wire. We need to lower the pain threshold of adopting RDMA-based networks (aka channel I/O).

30. 30 Conclusions  Channel I/O is a simple and powerful concept  The trend toward convergence in the ‘multi-fabric’ space is accelerating; Ethernet is quietly but aggressively attacking this space.  Channel I/O is in danger of being relegated to the supercomputer and HPC niches.  Expanding RDMA networks into the ‘multi-fabric’ space requires that we keep moving channel I/O forward

31. 31 A paid advertisement… System Fabric Works is dedicated to ensuring that its customers derive maximum competitive advantage from their information technology investment. Our focus is on accelerating adoption of advanced data center networks (“Fabric Computing”) as a foundation technology. We provide: 1)strategic and architectural consulting services directly to end user clients, 2)design and software engineering services to our vendor and integrator partners, 3)custom integrated computing, networking and storage products.

32. 32 Abstract: Channel I/O delivers an embarrassment of riches in terms of the features it provides. Despite that, significant inroads are only being made primarily in the supercomputer, scientific and commercial HPC spaces…spaces characterized as being purpose-built on top of a single primary fabric. The commercial space is characterized as being built on top of a combination of fabrics (as opposed to the single-fabric character of the HPC space). This space, where most of the gold is, remains more-or-less elusive. There are a few emerging exceptions. Why is this? The values delivered by channel I/O do not yet out-weigh the costs of transitioning. Meanwhile, Ethernet is quietly addressing those markets by providing a path toward a converged fabric. Call to action: Lower the hurdles preventing end users from harvesting the values of channel I/O Provide leadership in helping select customers through the “I/O quagmire”