Interconnect Trends in High Productivity Computing Actionable Market Intelligence for High Productivity Computing Addison Snell, VP/GM, April 2008

Tabor: “High Productivity Computing” Productivity analysis studies: How do users define and measure productivity? Site census studies: How do users configure their systems over time? Budget maps: What are the components of TCO? New online Productivity Analysis Tool Productivity analysis studies: How do users define and measure productivity? Site census studies: How do users configure their systems over time? Budget maps: What are the components of TCO? New online Productivity Analysis Tool Tabor Research is studying how HPC transcends price/performance and server sales.

Expanding Technology Contexts in HPC Technology differentiation pushed outside the server –HPC systems are no longer “self-contained” –Interconnects, processors, OS, storage all may come from different vendors Only a third of HPC budget goes to the server –A third goes to other products and services –Remaining third to facilities, staffing: much of facilities spending is “NIB” Network spending is about 10% of hardware (not including bundled system interconnect) Technology differentiation pushed outside the server –HPC systems are no longer “self-contained” –Interconnects, processors, OS, storage all may come from different vendors Only a third of HPC budget goes to the server –A third goes to other products and services –Remaining third to facilities, staffing: much of facilities spending is “NIB” Network spending is about 10% of hardware (not including bundled system interconnect)

Expanding Usage Contexts in HPC Tabor Research is doing a study on “Edge HPC” HPC technology or application profiles outside the traditional areas of engineering, science, analytics –Complex event processing: Wide-scale, sensor- based, event-driven, real-time or near real-time –Organization optimization: BI, data mining, logistics, inventory / supply chain management –Virtual environments: Online games, Second Life, “augmented reality,” virtual economies –Ultra-scale: Other usage of supercomputers Tabor Research is doing a study on “Edge HPC” HPC technology or application profiles outside the traditional areas of engineering, science, analytics –Complex event processing: Wide-scale, sensor- based, event-driven, real-time or near real-time –Organization optimization: BI, data mining, logistics, inventory / supply chain management –Virtual environments: Online games, Second Life, “augmented reality,” virtual economies –Ultra-scale: Other usage of supercomputers

System (Node-to-Node) Interconnects Data from Tabor Research HPC Site Census: Almost even thirds distribution between Ethernet, Infiniband, and other high-end cluster interconnects (Myrinet, Quadrics) –Almost no 10GbE as system fabrics –Infiniband seems to have had more impact on Ethernet than other high-end interconnects Average 200 nodes/system (SMP = 1 node) Data skewed a bit towards academic Data from Tabor Research HPC Site Census: Almost even thirds distribution between Ethernet, Infiniband, and other high-end cluster interconnects (Myrinet, Quadrics) –Almost no 10GbE as system fabrics –Infiniband seems to have had more impact on Ethernet than other high-end interconnects Average 200 nodes/system (SMP = 1 node) Data skewed a bit towards academic

LAN (Compute Room) Interconnects Data from Tabor Research HPC Site Census: Ethernet dominates as LAN fabric –Almost a third of Ethernet LANs have at least some 10GbE –10GbE outnumbers Infiniband by two-to-one Very little usage of anything other than Ethernet or Infiniband –Some mentions of wireless, FC, Myrinet –Here Infiniband seems to have taken over from fast non-Ethernet technologies Data from Tabor Research HPC Site Census: Ethernet dominates as LAN fabric –Almost a third of Ethernet LANs have at least some 10GbE –10GbE outnumbers Infiniband by two-to-one Very little usage of anything other than Ethernet or Infiniband –Some mentions of wireless, FC, Myrinet –Here Infiniband seems to have taken over from fast non-Ethernet technologies

Storage Interconnects Data from Tabor Comprehensive Research Study: Over one-third of HPC users implement IB as a storage interconnect –More common as storage infrastructure grows –Native IB protocol most common, followed closely by FC, with SATA a distant third –IB also most common implementation of RDMA On average, users place 25%-30% premium value on doubling the storage bandwidth Data from Tabor Comprehensive Research Study: Over one-third of HPC users implement IB as a storage interconnect –More common as storage infrastructure grows –Native IB protocol most common, followed closely by FC, with SATA a distant third –IB also most common implementation of RDMA On average, users place 25%-30% premium value on doubling the storage bandwidth

Other Thoughts Converged Fabric Strategies –About a third are “likely” to implement –Despite Ethernet position in LAN, much more likely to consider converged fabric on Infiniband Impact of multi-core at low end –In the near term, could create interest in SMP –Is MPI equipped to handle new level of parallelism at the socket? –Will changes in workload management or job scheduling create new interconnect requirements? Converged Fabric Strategies –About a third are “likely” to implement –Despite Ethernet position in LAN, much more likely to consider converged fabric on Infiniband Impact of multi-core at low end –In the near term, could create interest in SMP –Is MPI equipped to handle new level of parallelism at the socket? –Will changes in workload management or job scheduling create new interconnect requirements?

Interested in Tabor Research? Please, help us in our research efforts! Join the HPC User Views Advisory Council –Access to research –Rewards for participation Please, help us in our research efforts! Join the HPC User Views Advisory Council –Access to research –Rewards for participation

Interconnect Trends in High Productivity Computing Actionable Market Intelligence for High Productivity Computing Addison Snell, VP/GM, April 2008