CS 258 Parallel Computer Architecture Lecture 3 Introduction to Scalable Interconnection Network Design January 30, 2008 Prof John D. Kubiatowicz

Slides:



Advertisements
Similar presentations
Comparison Of Network On Chip Topologies Ahmet Salih BÜYÜKKAYHAN Fall.
Advertisements

What is Flow Control ? Flow Control determines how a network resources, such as channel bandwidth, buffer capacity and control state are allocated to packet.
1 Lecture 12: Interconnection Networks Topics: dimension/arity, routing, deadlock, flow control.
1 Lecture 23: Interconnection Networks Topics: communication latency, centralized and decentralized switches (Appendix E)
ECE669 L12: Interconnection Network Performance March 9, 2004 ECE 669 Parallel Computer Architecture Lecture 12 Interconnection Network Performance.
EECC756 - Shaaban #1 lec # 9 Spring Network Definitions A network is a graph V = {switches and nodes} connected by communication channels.
CS 258 Parallel Computer Architecture Lecture 4 Network Topology and Routing February 4, 2008 Prof John D. Kubiatowicz
CS252 Graduate Computer Architecture Lecture 21 Multiprocessor Networks (con’t) John Kubiatowicz Electrical Engineering and Computer Sciences University.
CS 258 Parallel Computer Architecture Lecture 5 Routing February 6, 2008 Prof John D. Kubiatowicz
Generic Multiprocessor Architecture
1 Lecture 24: Interconnection Networks Topics: topologies, routing, deadlocks, flow control Final exam reminders:  Plan well – attempt every question.
CS252 Graduate Computer Architecture Lecture 15 Multiprocessor Networks (con’t) March 15 th, 2010 John Kubiatowicz Electrical Engineering and Computer.
ECE669 L25: Final Exam Review May 6, 2004 ECE 669 Parallel Computer Architecture Lecture 25 Final Exam Review.
EECC756 - Shaaban #1 lec # 8 Spring Generic Multiprocessor Architecture Generic Multiprocessor Architecture Node: processor(s), memory system,
1 Lecture 24: Interconnection Networks Topics: communication latency, centralized and decentralized switches (Sections 8.1 – 8.5)
Issues in System-Level Direct Networks Jason D. Bakos.
EECS 570: Fall rev1 1 Chapter 10: Scalable Interconnection Networks.
EECC756 - Shaaban #1 lec # 9 Spring Generic Multiprocessor Architecture Generic Multiprocessor Architecture Node: processor(s), memory system,
Interconnection Network Topology Design Trade-offs
1 Lecture 24: Interconnection Networks Topics: topologies, routing, deadlocks, flow control.
1 Lecture 25: Interconnection Networks Topics: communication latency, centralized and decentralized switches, routing, deadlocks (Appendix E) Review session,
CS252/Patterson Lec /28/01 CS162 Computer Architecture Lecture 16: Multiprocessor 2: Directory Protocol, Interconnection Networks.
1 Static Interconnection Networks CEG 4131 Computer Architecture III Miodrag Bolic.
EECC756 - Shaaban #1 lec # 10 Spring Generic Multiprocessor Architecture Generic Multiprocessor Architecture Node: processor(s), memory system,
Computer Architecture II 1 Computer architecture II Network topologies.
ECE669 L16: Interconnection Topology March 30, 2004 ECE 669 Parallel Computer Architecture Lecture 16 Interconnection Topology.
CS252 Graduate Computer Architecture Lecture 14 Multiprocessor Networks March 9 th, 2011 John Kubiatowicz Electrical Engineering and Computer Sciences.
John Kubiatowicz Electrical Engineering and Computer Sciences
Storage area network and System area network (SAN)
Switching, routing, and flow control in interconnection networks.
Interconnect Network Topologies
CS252 Graduate Computer Architecture Lecture 15 Multiprocessor Networks March 14 th, 2011 John Kubiatowicz Electrical Engineering and Computer Sciences.
CS252 Graduate Computer Architecture Lecture 15 Multiprocessor Networks March 12 th, 2012 John Kubiatowicz Electrical Engineering and Computer Sciences.
1 The Turn Model for Adaptive Routing. 2 Summary Introduction to Direct Networks. Deadlocks in Wormhole Routing. System Model. Partially Adaptive Routing.
Interconnect Networks
On-Chip Networks and Testing
Winter 2006 ENGR 9861 – High Performance Computer Architecture March 2006 Interconnection Networks.
CSE Advanced Computer Architecture Week-11 April 1, 2004 engr.smu.edu/~rewini/8383.
1 Lecture 7: Interconnection Network Part I: Basic Definitions Part II: Message Passing Multicomputers.
1 Scalable Interconnection Networks. 2 Scalable, High Performance Network At Core of Parallel Computer Architecture Requirements and trade-offs at many.
Multiprocessor Interconnection Networks Todd C. Mowry CS 740 November 3, 2000 Topics Network design issues Network Topology.
ECE669 L21: Routing April 15, 2004 ECE 669 Parallel Computer Architecture Lecture 21 Routing.
Network-on-Chip Introduction Axel Jantsch / Ingo Sander
Anshul Kumar, CSE IITD ECE729 : Advanced Computer Architecture Lecture 27, 28: Interconnection Mechanisms In Multiprocessors 29 th, 31 st March, 2010.
Interconnection Networks Alvin R. Lebeck CPS 220.
Interconnect Networks Basics. Generic parallel/distributed system architecture On-chip interconnects (manycore processor) Off-chip interconnects (clusters.
Super computers Parallel Processing
Networks: Routing, Deadlock, Flow Control, Switch Design, Case Studies Alvin R. Lebeck CPS 220.
1 Lecture 24: Interconnection Networks Topics: communication latency, centralized and decentralized switches, routing, deadlocks (Appendix F)
Spring EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Advanced Computer Architecture Lecture 22 Distributed computer Interconnection.
1 Lecture 14: Interconnection Networks Topics: dimension vs. arity, deadlock.
1 Lecture 22: Interconnection Networks Topics: Routing, deadlock, flow control, virtual channels.
Interconnect Networks
Lecture 23: Interconnection Networks
Multiprocessor Interconnection Networks Todd C
Azeddien M. Sllame, Amani Hasan Abdelkader
Prof John D. Kubiatowicz
Interconnection Network Routing, Topology Design Trade-offs
John Kubiatowicz Electrical Engineering and Computer Sciences
Interconnection Network Design Contd.
Introduction to Scalable Interconnection Network Design
Switching, routing, and flow control in interconnection networks
Lecture 14: Interconnection Networks
Interconnection Network Design Lecture 14
Introduction to Scalable Interconnection Networks
Static Interconnection Networks
Interconnection Network Design
Interconnection Networks Contd.
Embedded Computer Architecture 5SAI0 Interconnection Networks
Networks: Routing and Design
Presentation transcript:

CS 258 Parallel Computer Architecture Lecture 3 Introduction to Scalable Interconnection Network Design January 30, 2008 Prof John D. Kubiatowicz

Lec 3.2 1/30/08Kubiatowicz CS258 ©UCB Spring 2008 Scalable, High Perf. Network At Core of Parallel Computer Arch. Requirements and trade-offs at many levels –Elegant mathematical structure –Deep relationships to algorithm structure –Managing many traffic flows –Electrical / Optical link properties Little consensus –interactions across levels –Performance metrics? –Cost metrics? –Workload? => need holistic understanding

Lec 3.3 1/30/08Kubiatowicz CS258 ©UCB Spring 2008 Requirements from Above Communication-to-computation ratio  bandwidth that must be sustained for given computational rate –traffic localized or dispersed? –bursty or uniform? Programming Model –protocol –granularity of transfer –degree of overlap (slackness)  job of a parallel machine network is to transfer information from source node to dest. node in support of network transactions that realize the programming model

Lec 3.4 1/30/08Kubiatowicz CS258 ©UCB Spring 2008 Goals latency as small as possible as many concurrent transfers as possible –operation bandwidth –data bandwidth cost as low as possible

Lec 3.5 1/30/08Kubiatowicz CS258 ©UCB Spring 2008 Outline Introduction Basic concepts, definitions, performance perspective Organizational structure Topologies

Lec 3.6 1/30/08Kubiatowicz CS258 ©UCB Spring 2008 Basic Definitions Network interface –Processor (or programmer’s) interface to the network –Mechanism for injecting packets/removing packets Links –Bundle of wires or fibers that carries a signal –May have separate wires for clocking Switches –connects fixed number of input channels to fixed number of output channels –Can have a serious impact on latency, saturation, deadlock

Lec 3.7 1/30/08Kubiatowicz CS258 ©UCB Spring 2008 Links and Channels transmitter converts stream of digital symbols into signal that is driven down the link receiver converts it back –tran/rcv share physical protocol trans + link + rcv form Channel for digital info flow between switches link-level protocol segments stream of symbols into larger units: packets or messages (framing) node-level protocol embeds commands for dest communication assist within packet Transmitter...ABC123 => Receiver...QR67 =>

Lec 3.8 1/30/08Kubiatowicz CS258 ©UCB Spring 2008 Clock Synchronization? Receiver must be synchronized to transmitter –To know when to latch data Fully Synchronous –Same clock and phase: Isochronous –Same clock, different phase: Mesochronous »High-speed serial links work this way »Use of encoding (8B/10B) to ensure sufficient high- frequency component for clock recovery Fully Asynchronous –No clock: Request/Ack signals –Different clock: Need some sort of clock recovery? Data Req Ack Transmitter Asserts Data t0 t1 t2 t3 t4 t5

Lec 3.9 1/30/08Kubiatowicz CS258 ©UCB Spring 2008 Formalism network is a graph V = {switches and nodes} connected by communication channels C  V  V Channel has width w and signaling rate f =  –channel bandwidth b = wf –phit (physical unit) data transferred per cycle –flit - basic unit of flow-control Number of input (output) channels is switch degree Sequence of switches and links followed by a message is a route Think streets and intersections

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 What characterizes a network? Topology (what) –physical interconnection structure of the network graph –direct: node connected to every switch –indirect: nodes connected to specific subset of switches Routing Algorithm(which) –restricts the set of paths that msgs may follow –many algorithms with different properties »gridlock avoidance? Switching Strategy(how) –how data in a msg traverses a route –circuit switching vs. packet switching Flow Control Mechanism(when) –when a msg or portions of it traverse a route –what happens when traffic is encountered?

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Topological Properties Routing Distance - number of links on route Diameter - maximum routing distance Average Distance A network is partitioned by a set of links if their removal disconnects the graph

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Interconnection Topologies Class of networks scaling with N Logical Properties: –distance, degree Physical properties –length, width Fully connected network –diameter = 1 –degree = N –cost? »bus => O(N), but BW is O(1) - actually worse »crossbar => O(N 2 ) for BW O(N) VLSI technology determines switch degree

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Example: Linear Arrays and Rings Linear Array –Diameter? –Average Distance? –Bisection bandwidth? –Route A -> B given by relative address R = B-A Torus? Examples: FDDI, SCI, FiberChannel Arbitrated Loop, KSR1

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Example: Multidimensional Meshes and Tori d-dimensional array –n = k d-1 X...X k O nodes –described by d-vector of coordinates (i d-1,..., i O ) d-dimensional k-ary mesh: N = k d –k = d  N –described by d-vector of radix k coordinate d-dimensional k-ary torus (or k-ary d-cube)? 2D Grid 3D Cube

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Properties of Routing –relative distance: R = (b d-1 - a d-1,..., b 0 - a 0 ) –traverse ri = b i - a i hops in each dimension –dimension-order routing? Adaptive routing? Average DistanceWire Length? –d x 2k/3 for mesh –dk/2 for cube Degree? Bisection bandwidth?Partitioning? –k d-1 bidirectional links Physical layout? –2D in O(N) spaceShort wires –higher dimension?

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 On Chip: Embeddings in two dimensions Embed multiple logical dimension in one physical dimension using long wires When embedding higher-dimension in lower one, either some wires longer than others, or all wires long 6 x 3 x 2

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Real Machines Wide links, smaller routing delay Tremendous variation

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Administrivia First set of readings posted for Monday –“The Future of Wires,” Ron Ho, Kenneth W. Mai, and Mark A. Horowitz. –“An Adaptive and Fault Tolerant Wormhole Routing Strategy for k-ary n-cubes,” Daniel H. Linder and Jim C. Harden,

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Typical Packet Format Two basic mechanisms for abstraction –encapsulation –fragmentation

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Communication Perf: Latency per hop Time(n) s-d = overhead + routing delay + channel occupancy + contention delay Channel occupancy = (n + n e ) / b Routing delay? Contention?

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Store&Forward vs Cut-Through Routing Time:h(n/b +  ) vsn/b + h  OR(cycles): h(n/w +  ) vsn/w + h  what if message is fragmented? wormhole vs virtual cut-through

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Contention Two packets trying to use the same link at same time –limited buffering –drop? Most parallel mach. networks block in place –link-level flow control –tree saturation Closed system - offered load depends on delivered –Source Squelching

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Bandwidth What affects local bandwidth? –packet densityb x n/(n + n e ) –routing delayb x n /(n + n e + w  ) –contention »endpoints »within the network Aggregate bandwidth –bisection bandwidth »sum of bandwidth of smallest set of links that partition the network –total bandwidth of all the channels: Cb –suppose N hosts issue packet every M cycles with ave dist »each msg occupies h channels for l = n/w cycles each »C/N channels available per node »link utilization for store-and-forward:  = (h l /M channel cycles/node)/(C/N) = Nh l /MC < 1! »link utilization for wormhole routing?

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Saturation

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Organizational Structure Processors –datapath + control logic –control logic determined by examining register transfers in the datapath Networks –links –switches –network interfaces

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Link Design/Engineering Space Cable of one or more wires/fibers with connectors at the ends attached to switches or interfaces Short: - single logical value at a time Long: - stream of logical values at a time Narrow: - control, data and timing multiplexed on wire Wide: - control, data and timing on separate wires Synchronous: - source & dest on same clock Asynchronous: - source encodes clock in signal

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Example: Cray MPPs T3D: Short, Wide, Synchronous(300 MB/s) –24 bits »16 data, 4 control, 4 reverse direction flow control –single 150 MHz clock (including processor) –flit = phit = 16 bits –two control bits identify flit type (idle and framing) »no-info, routing tag, packet, end-of-packet T3E: long, wide, asynchronous (500 MB/s) –14 bits, 375 MHz, LVDS –flit = 5 phits = 70 bits »64 bits data + 6 control –switches operate at 75 MHz –framed into 1-word and 8-word read/write request packets Cray BlackWidow High-Radix Clos Network: long, narrow, mesochronous (800MHz) –3 bits each direction/port, 6.25Gbps per link, clock recovery –64 ports –High-radix butterfly/fat tree network Cost = f(length, width) ?

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Switches

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Switch Components Output ports –transmitter (typically drives clock and data) Input ports –synchronizer aligns data signal with local clock domain –essentially FIFO buffer Crossbar –connects each input to any output –degree limited by area or pinout Buffering Control logic –complexity depends on routing logic and scheduling algorithm –determine output port for each incoming packet –arbitrate among inputs directed at same output

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Properties of Routing Algorithms Deterministic –route determined by (source, dest), not intermediate state (i.e. traffic) Adaptive –route influenced by traffic along the way Minimal –only selects shortest paths Deadlock free –no traffic pattern can lead to a situation where no packets move forward

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Deadlock Freedom How can it arise? –necessary conditions: »shared resource »incrementally allocated »non-preemptible –think of a channel as a shared resource that is acquired incrementally »source buffer then dest. buffer »channels along a route How do you avoid it? –constrain how channel resources are allocated –ex: dimension order How do you prove that a routing algorithm is deadlock free? –More next time: See Linder and Harden reading

Lec /30/08Kubiatowicz CS258 ©UCB Spring 2008 Summary TopologyDegreeDiameterAve DistBisectionD (D P=1024 1D Array2N-1N / 31huge 1D Ring2N/2N/42 2D Mesh42 (N 1/2 - 1)2/3 N 1/2 N 1/2 63 (21) 2D Torus4N 1/2 1/2 N 1/2 2N 1/2 32 (16) k-ary n-cube2nnk/2nk/4nk/415 Hypercuben =log Nnn/2N/210 (5)