Infiniband subnet management Discuss the Infiniband subnet management system Discuss fat tree and subnet management in an Infiniband with a fat tree topology.

Slides:



Advertisements
Similar presentations
EdgeNet2006 Summit1 Virtual LAN as A Network Control Mechanism Tzi-cker Chiueh Computer Science Department Stony Brook University.
Advertisements

1 Routing Protocols I. 2 Routing Recall: There are two parts to routing IP packets: 1. How to pass a packet from an input interface to the output interface.
Presentation of Designing Efficient Irregular Networks for Heterogeneous Systems-on-Chip by Christian Neeb and Norbert Wehn and Workload Driven Synthesis.
Packet Switching COM1337/3501 Textbook: Computer Networks: A Systems Approach, L. Peterson, B. Davie, Morgan Kaufmann Chapter 3.
Jaringan Komputer Lanjut Packet Switching Network.
Case study IBM Bluegene/L system InfiniBand. Interconnect Family share for 06/2011 top 500 supercomputers Interconnect Family CountShare % Rmax Sum (GF)
Advanced Networking Wickus Nienaber Daniel Beech.
High Performance Router Architectures for Network- based Computing By Dr. Timothy Mark Pinkston University of South California Computer Engineering Division.
1 Switching and Forwarding Bridges and Extended LANs.
CMPE 150- Introduction to Computer Networks 1 CMPE 150 Fall 2005 Lecture 22 Introduction to Computer Networks.
Shivkumar KalyanaramanRensselaer Q1-1 ECSE-6600: Internet Protocols Quiz 1 Time: 60 min (strictly enforced) Points: 50 YOUR NAME: Be brief, but DO NOT.
Chapter 10 Introduction to Wide Area Networks Data Communications and Computer Networks: A Business User’s Approach.
1 Lecture 25: Interconnection Networks Topics: communication latency, centralized and decentralized switches, routing, deadlocks (Appendix E) Review session,
CMPE 150- Introduction to Computer Networks 1 CMPE 150 Fall 2005 Lecture 21 Introduction to Computer Networks.
A Scalable, Commodity Data Center Network Architecture Mohammad Al-Fares, Alexander Loukissas, Amin Vahdat Presented by Gregory Peaker and Tyler Maclean.
1 Switching and Forwarding Bridges and Extended LANs.
Introduction to Computer Networks 09/23 Presenter: Fatemah Panahi.
Storage area network and System area network (SAN)
Layer-3 Routing Natawut Nupairoj, Ph.D. Department of Computer Engineering Chulalongkorn University.
Modeling and Evaluation of Fibre Channel Storage Area Networks Xavier Molero, Federico Silla, Vicente Santonia and Jose Duato.
Switching, routing, and flow control in interconnection networks.
Interconnect Network Topologies
Copyright © 2007 InfiniBand ® Trade Association. Other names and brands are properties of their respective owners. IB Cross-Subnet Communication OpenFabrics.
29-Aug-154/598N: Computer Networks Switching and Forwarding Outline –Store-and-Forward Switches.
Network Redundancy Multiple paths may exist between systems. Redundancy is not a requirement of a packet switching network. Redundancy was part of the.
Interconnect Networks
Introduction to Routing and Routing Protocols By Ashar Anwar.
University of the Western Cape Chapter 11: Routing Aleksandar Radovanovic.
1/29/2002 CS Distributed Systems 1 Infiniband Architecture Aniruddha Bohra.
Current major high performance networking technologies InfiniBand 10G-Ethernet.
Day 12 Chapter 13 WAN Technologies and Routing. Classification of Networks LAN – spans a single building or campus MAN – Spans a city WAN – Spans multiple.
QoS Support in High-Speed, Wormhole Routing Networks Mario Gerla, B. Kannan, Bruce Kwan, Prasasth Palanti,Simon Walton.
1 © 2003, Cisco Systems, Inc. All rights reserved. CCNA 2 Module 9 Basic Router Troubleshooting.
NUS.SOC.CS2105 Ooi Wei Tsang Application Transport Network Link Physical you are here.
Chi-Cheng Lin, Winona State University CS 313 Introduction to Computer Networking & Telecommunication Chapter 5 Network Layer.
Logical Topology Design
1 Network Layer Lecture 13 Imran Ahmed University of Management & Technology.
Cisco Systems Networking Academy S2 C 11 Routing Basics.
Interconnect Networks Basics. Generic parallel/distributed system architecture On-chip interconnects (manycore processor) Off-chip interconnects (clusters.
McGraw-Hill©The McGraw-Hill Companies, Inc., 2000 CH. 8: SWITCHING & DATAGRAM NETWORKS 7.1.
1 Switching and Forwarding Sections Connecting More Than Two Hosts Multi-access link: Ethernet, wireless –Single physical link, shared by multiple.
Prepaid by: Guided by: ashwin goswami.
Mobile IP THE 12 TH MEETING. Mobile IP  Incorporation of mobile users in the network.  Cellular system (e.g., GSM) started with mobility in mind. 
1 Packet Switching Outline Switching and Forwarding Bridges and Extended LANs.
Configuration for routing example
Network Layer COMPUTER NETWORKS Networking Standards (Network LAYER)
Chapter 3: Packet Switching (overview)
ROUTING.
Modeling and Evaluation of Fibre Channel Storage Area Networks
Infiniband Architecture
Chapter 4 Introduction to Network Layer
Advanced Computer Networks
Scaling the Network: The Internet Protocol
3. Internetworking (part 1)
Chapter 4: Routing Concepts
Switching and Forwarding Bridges and Extended LANs
Chapter 3 Switching.
CCNA 2 v3.1 Module 6 Routing and Routing Protocols
Chapter 4 Introduction to Network Layer
ECE 544 Protocol Design Project 2016
Switching, routing, and flow control in interconnection networks
Storage area network and System area network (SAN)
Routing protocol in all three types of networks - Phase 1
Switch controller: Routing
Data Center Architectures
Scaling the Network: The Internet Protocol
Bridges Neil Tang 10/10/2008 CS440 Computer Networks.
Switching, routing, and flow control in interconnection networks
Data Communication: Routing algorithms
Virtual LAN (VLAN).
Presentation transcript:

Infiniband subnet management Discuss the Infiniband subnet management system Discuss fat tree and subnet management in an Infiniband with a fat tree topology. References A. Bermudez, R. Casado, F.J. Quiles, T. M. Pinkston, J. Duato, “Evaluation of a Subnet Management Mechanism for Infiniband Networks”, ICPP A. Vishnu, A. R. Mamidala, H. Jin, D. K. Panda, “Performance Modeling of Subnet Management on Fat Tree Infiniband Networks using OpenSM”, Workshop on System Management Tools on Large Scale Parallel Systems, Held in Conjunction with IPDPS 2005 X. Lin, Y. Chung, T. Huang, “A Multiple LID Routing Scheme for Fat-Tree- Based Infiniband Networks.” IPDPS 2004.

Infiniband devices and entities related to subnet management Devices: Channel Adapters (CA), Host Channel Adapters, switches, routers Subnet manager (SM): discovering, configuring, activating and managing the subnet A subnet management agent (SMA) in every device generates, responses to control packets (subnet management packets (SMPs)), and configure local components for subnet management SM exchange control packets with SMA with subnet management interface (SMI).

Subnet management packets (SMP) –256 bytes of data –Use unreliable datagram service on the management virtual lane (VL 15) –LID routed: use lookup table for forwarding Use after the subnet is setup. E.g. Check the status of an active port –Direct routed: has the information of the output port for each intermediate hop. Subnet discovery for the subnet is setup

Subnet management packets (SMP) –Define the operation to be performed by SM –Get: get the information about CA, switch, port –Set: set the attribute of a port (e.g. LID) –GetResp: get response –Trap: inform SM about the state of a local node A SMA stop sending Trap message until it receives TrapRepress packet. Topology information can be obtained by a sweep and by peridical Traps.

Subnet Management phases: –Topology discovery: sending direct routed SMP to evert port and processing the responses. –Path computation: computing valid paths between each pair of end node –Path distribution phase: configuring the forwarding table

Subnet discovery –SM starts by sending a direct routed Get SMP to its local node. Upone receiving response, SM sends SMPs with additive depth.

Path computation: –Compute paths between all pair of nodes –For irregular topology: Up/Down routing does not work directly –Need information about the incoming interface and the destination and Infiniband only uses destination –Potential solution: »find all possible paths »remove all possible down link following up links in each node »find one output port for each destination –Why is that still working? No clear to me. –Other solutions: destination renaming –Fat tree topology: What is the best that can be achieved is also not clear.

Path distribution: –Ordering issue: the network may be in an inconsistent state when partially updated, which may result in deadlock during this period. Traditional solution, no data packets for a period of time deadlock free reconfiguration schemes.

Fat Tree: –A way to build large scale clusters

Fat Tree: –Routing in a complete fat tree: a up phase and a down phase (always contention free).

Fat Tree: –The complete fat tree has the scalability problem The root has a very large nodal degree –How to build a fat tree with nodes that have a constant nodal degree. –M-port n-tree FT(m, n) m is the number of port per switch n+1 is the height of the tree The tree consists of 2*(m/2)^n processing nodes and (2n-1)*(m/2)^(n-1) switches.

–How is an m-port n-tree FT(m, n) connected? m is the number of port per switch n+1 is the height of the tree The tree consists of 2*(m/2)^n processing nodes and (2n-1)*(m/2)^(n-1) switches. A processing node is labeled as P(p_0 … p_{n-1}), –P_0 = 0..m-1, p_i (i!=0) = 0..m/2-1 A switch is labeled as SW –l = 0..n-1, –When l=0, w_i = 0..m/2-1 –When l!=0, w_0 = 0..m-1, w_I (I!=0) = 0..m/2-1

Figure 5: a 4-port 3-tree

How is the tree connected? –SW _k be the kth port of SW(w, l). –SW(w, l)_k and SW(w’, l’)_k’ is connected iff l’ = l + 1 w_0…w_{n-3} = w_0’…w_{l-1}’w_{l+1}’…w_{n-2}’ k=w_l’, k’ = w_{n-2} + m/2 –Question: which switches are connected to SW ? How the ports are connected?

Fat tree properties: –Multiple routes between two nodes Deterministic routing: one path between two nodes, how to map? –What is a good mapping? In the case when the traffic pattern is unknown, common practice is to minimize the maximum load on a link. –Do we know how to do it? Not clear even when there is no restriction on the routing. It is likely that an optimal solution exists for a particular FAT tree topology In infiniband, destination based routing put some restriction on which path can be used.