1. NoC Switch: Basic Design Principles &
Tunis, December 2015
NoC Switch: Basic Design Principles &
Intra-Switch Performance Optimization
Instructor: Davide Bertozzi

2. Acknowledgement Many slides have been taken or adapted from
Prof. Giorgos Dimitrakopoulos
Electrical and Computer Engineering Democritus University of Thrace (DUTH), Greece
NoCS 2012 Tutorial:
Switch design: A unified view of
microarchitecture and circuits

3. Switch Building Blocks

4. Wormhole switch operation
The operations can fit in the same cycle or they can be pipelined
Extra registers are needed in the control path
Body/tail flits inherit the decisions taken by the head flits, yet they cannot bypass RC, and SA
Simply, there is nothing to do for them in those stages
Operation latency is an issue!
For single-cycle switches, the head flit is anyway the one which determines the critical timing path! Body/tail flits would have slack!

5. Look-ahead routing
Routing computation is based only on packet’s destination
Can be performed in switch A and used in switch B
Look-ahead routing computation (LRC)
Does it really need to be a separate pipeline stage?

6. Look-ahead routing optimization
The LRC can be performed in parallel to SA
LRC should be completed before the ST stage in the same switch
The head flit needs to embed the output port requests for the next switch before leaving

7. Look-ahead routing details
The head flit of each packet carries the output port requests for the next switch together with the destination address

8. Low-latency organizations
Baseline
SA precedes ST (no speculation)
SA decoupled from ST
Predict or Speculate arbiter’s decisions
Trick: crossbar control not from switch allocator
When prediction is wrong replay all the tasks (same as baseline)
Do in different phases
Circuit switching
Arbitration and routing at setup phase
Transmit phase: contention-free ST
Bypass switches
Reduce latency under certain criteria
When bypass not enabled, same as baseline SA ST LT
LRC SA LT ST
LRC SA LT
Setup
LRC ST LT
Transmit

9. At runtime the prediction accuracy is verified.
ST in parallel with SA SA ST
Prediction criteria
Target:
LRC
It is likely that a packet coming from the east (if any) will go to the west because of xy routing in a 2D mesh
Target Output
West
East
Input
packet
During idle time the predictor pre-sets the I/O connection of the West output port through the crossbar multiplexer
At runtime the prediction accuracy is verified.
Mis-
prediction
Arbiter South
East
Arbiter West
Arbiter West
East
East
East
East
North
North
East
South
South
West
West
Local
Local

10. Mux control signals are set on-the-fly by fast speculation logic
ST in parallel with SA SA ST
Speculation
Target:
LRC
Mux control signals are set on-the-fly by fast speculation logic
At the end of the cycle, arbiter computation results are compared with the outcome of speculation logic
At the beginning of the cycle, requests are fed to the allocator arbiter and to speculation logic
East
East
arbiter
arbiter ??
Local
Local
Local
Mis-
speculation
East
East
Mask
East
East
North
East
North
East
South
South
West
West
Local
Local
Next step: switch traversal from Local

11. Prediction-based ST: HitSA ST
Target:
Assumption: RC is a pipeline stage (no LRC) RC
Crossbar is reserved
Idle state: Output port X+ is selected and reserved for Input X+
1st cycle: Incoming flit is transferred to X+ without RC and SA
Correct
1st cycle: RC is performed  The prediction was correct!
Outcome: SA, ST and RC were actually performed in parallel!
PREDICTOR
ARBITER
Buffer X+ X+ X- X- Y+ Y+ Y- Y-
Crossbar

12. Prediction-based ST: Miss
Idle state: Output port X+ is selected and reserved
1st cycle: Incoming flit is transferred to X+ without RC and SA
Correct
Dead flit
1st cycle: RC is performed  The prediction is wrong!
(X- is correct)
KILL
Kill signal to X+ is asserted
2nd/3rd cycle: Dead flit is removed; Move on with SA.
PREDICTOR
ARBITER
Buffer X+ X+ X- X- Y+ Y+
@Miss: tasks replayed as the baseline case
(that is, at least RC was done, now let us move to SA) Y- Y-
Crossbar

13. Speculative ST Assumption: RC is done (LRC) Speculation criteria:
Switch
Fabric
Control
B Wins
A Wins
Speculation criteria:
assume contention doesn’t happen!
If correct then flit transferred directly to output port without waiting SA
In case of contention SA was done, move on with ST accordingly
Wasted cycle in the event of contention
Generation and management of event abort A A A A A ? B B B
clk
port 0
port 1
grant
valid out
data out 1 4
cycle 2 3 A p0 A B p1
??? A p0 B A A B
Only SA done

14. Efficient recovery from mispeculation: xor-based recovery
Switch
Fabric
Control
B Wins
Assume contention never happens
If correct then flit transferred directly to output port
If not then bitwise=XOR all the competing flits and send the encoded result to the link
At the same time arbitrate and mask (set to 0) the winning input
Repeat on the next cycle
In the case of contention, encoded outputs (due to contention) are resolved at the receiver
Can be done at the output port of the switch too A A A A A
A^B B B 1 4
cycle 2 3
clk
port 0
port 1
grant
valid out
data out A p0 A B p1
B^A A
No Contention
Contention

15. XOR-based recovery Works upon simple XOR property.
Coded
Flit Buffer 1
B^C
A^B^C
A^B^C A
B^C C
B^C C B A
Works upon simple XOR property.
(A^B^C) ^ (B^C) = A
Always able to decode by XORing two sequential values
Performs similarly to speculative switches
Only head-flit collisions matter
Maintains previous router’s arbitration order

16. Bypassing intermediate nodes
Virtual bypassing paths
SRC
1-cycle
Bypassed
DST
3-cycle
3-cycle
3-cycle
3-cycle
3-cycle
Switch bypassing criteria:
Frequently used paths
Packets continually moving along the same dimension
Most techniques can bypass some pipeline stages only for specific packet transfers and traffic patterns
Not generic enough

17. Speculation-free low-latency switches
SAT
Prediction and speculation drawbacks
On miss-prediction (speculation) the tasks should be replayed
Latency not always saved. Depends on network conditions
Merged Switch Allocation and Traversal (SAT)
Latency always saved – no speculation
Delay of SAT smaller than SA and ST in series

18. How can we increase throughput?
Green flow is blocked until red passes the switch. Physical channel left idle
The solution is to have separate buffers for each flow

19. Virtual Channels
Decouple output port allocation from next-hop buffer allocation
Contention present on:
Output links (crossbar output port)
Input port of the crossbar
Contention is resolved by time sharing the resources
Mixing words of two packets on the same channel
The words are on different virtual channels
A virtual channel identifier should be in place
Separate buffers at the end of the link guarantee no blocking between the packets

20. Virtual Channels Virtual-channel support does not mean extra links
They act as extra street lanes
Traffic on each lane is time shared on a common channel
Provide dedicated buffer space for each virtual channel
Decouple channels from buffers
Interleave flits from different packets
“The Swiss Army Knife for Interconnection Networks”
Reduce head-of-line blocking
Prevent deadlocks
Provide QoS, fault-tolerance, …

21. Datapath of a VC-based switch
Separate buffer for each VC
Separate flow control signals (credits/stalls) for each VC
The radix of the crossbar may stay the same (or may increase)
A higher number of input ports increases propagation delay through the crossbar
Input VCs may share a common input port of the crossbar
Alternatively, crossbars can be replicated
On each cycle at most one VC will receive a new word

22. Per-packet operation of a VC-based switch
A switch connects input VCs to output VCs
Routing computation (RC) determines the output port
Can be shared among VCs of an input port
May restrict the usable output VCs (e.g., based on msg type or dst ID)
An input VC should allocate first an output VC
Allocation is performed by a VC allocator (VA)
RC and VA are done per packet on the head flits and inherited to the remaining flits of the packet
Input VCs
Output VCs

23. Per-flit operation of a VC-based switch
Flits with an allocated output VC fight for an output port
Output port allocated by switch allocator
This entails 2 levels of arbitration
At input port
At output port
The VCs of the same input share a common input port of the crossbar
Each input has multiple requests (equal to the #input VCs)
The flit leaves the switch provided credits are available downstream
Credits are counted per output VC
Unfortunate case: VC & port are allocated to an input VC, but no credits available
Input VCs
Output VCs

24. Switch allocation
All VCs at a given input port share one crossbar input port
Switch allocator matches ready-to-go flits with crossbar time slots
Allocation performed on a cycle-by-cycle basis
N×V requests (input VCs), N resources (output ports)
At most one flit at each input port can be granted
At most one flit et each output port can be sampled
Other options need more crossbar ports (input-output speedup)

25. Switch allocation example
Bipartite graph
Inputs
Outputs
Request matrix
Outputs 1 2 1 1
Inputs 1 2 2 2
One request (arc) for each input VC
Example with 2 VCs per input
At most 2 arcs leaving each input
At most 2 requests per row in the request matrix
The allocation is a Matching problem:
Each grant must satisfy a request
Each requester gets at most one grant
Each resource is granted at most once

26. Separable allocation
Matchings have at most one grant per row and per column
Two phases of arbitration
Column-wise and row-wise
Perform in either order
Arbiters in each stage are independent
But the outcome of each one affects the quality of the overall match
Fast and cheap
Bad choices in first phase can prevent second stage from generating a good matching
Multiple iterations required for a good match
Iterative scheduling converges to a maximal schedule
Unaffordable for high-speed networks
Input-first:
Output-first:

27. Input first allocation
Implementation
Input first allocation
(row-wise)

28. Output first allocation
Implementation
Output first allocation
(column-wise)

29. Centralized allocator
Wavefront allocation
Pick initial diagonal
Grant all requests on each diagonal
Never conflict!
For each grant, delete requests in same row, column
Repeat for next diagonal

30. Switch allocation for adaptive routing
Input VCs can request more than one output ports
Called the set of Admissible Output Ports (AOP)
This adds an extra selection step (not arbitration)
Selection mostly tries to load balance the traffic
Input-first allocation
For each input VC select one request of the AOP
Arbitrate locally per input and select one input VC
Arbitrate globally per output and select one VC from all fighting inputs
Output-first allocation
Send all requests of the AOP of each input VC to the outputs
Arbitrate globally per output and grant one request
Arbitrate locally per input and grant an input VC
For this input VC select one out of the possibly multiple grants of the AOP set

31. VC allocation
Input VCs
Output VCs
Virtual channels (VCs) allow multiple packet flows to share physical resources (buffers, channels)
Before packets can proceed through router, need to claim ownership of VC buffer at next router
VC acquired by head flit, is inherited by body & tail flits
VC allocator assigns waiting packets at inputs to output VC buffers that are not currently in use
N×V inputs (input VCs), N×V outputs (output VCs)
Once assigned, VC is used for entire packet’s duration in the switch

32. VC allocation example Requests Grants Inputs VCs Output VCs Inputs VCs
In#0
Out#0
In#0
Out#0 1 1 1 1 2 2 2 2
In#1
Out#1
In#1
Out#1 3 3 3 3 4 4 4 4
Out#2
In#2
Out#2
In#2 5 5 5 5
An input VC may require any of the VCs of a given output port
In case of adaptive routing, an input VC may require VCs from different output ports
No port constraints as in switch allocators
Allocation can be both separable (2 arbitration steps) or centralized
At most one grant per input VC
At most one grant per output VC

33. Input – output VC mapping
Any-to-any flexibility in VC allocator is unnecessary
Partition set of VCs to restrict legal requests
Different use cases for VCs restrict possible transitions:
Message class never changes
VCs within a packet class are functionally equivalent
Can take advantage of these properties to reduce VC allocator complexity!

34. Single cycle VA or pipelined organization
Header flits see longer latency than body/tail flits
RC, VA decisions taken for head flits and inherited to the rest of the packet
Every flit fights for SA
Can we parallelize SA and VA?

35. The order of VC and switch allocation
VA first SA follows
Only packets with an allocated output VC fight for SA
VA and SA can be performed concurrently:
Speculate that waiting packets will successfully acquire a VC
Prioritize non-speculative requests over speculative ones for SA
Speculation holds only for the head flits (The body/tail flits always know their output VC) VA SA
Description
Win
Everything OK!! Leave the switch
Lose
Allocated a VC
Retry SA (not speculative - high priority next cycle)
Does not know the output VC
Allocated output port (grant lost – output idle)
Retry both VA and SA

36. Free list of VCs per output
Can assign a VC non-speculatively after SA
A free list of output VCs exists at each output
The flit that was granted access to this output receives the first free VC before leaving the switch
If no VC available output port allocation slot is missed
Flit retries for switch allocation
VCs are not unnecessarily occupied for flits that don’t win SA
Optimizations feasible:
Flits are allowed to win SA for a target port only provided there are empty VCs at that output port