1. Low-Latency Virtual-Channel Routers for On-Chip Networks Robert Mullins, Andrew West, Simon Moore
Presented by
Sailesh Kumar

2. Outline Motivation Comparison to Packet Networks
Why Network-on-chip (NoC)
Comparison to Packet Networks
Similarities
Differences
Design Constraints
Topology and Routing/Switching techniques for NoC
Mesh, fat-tree, honey-comb
Greedy, Deflection, Wormhole, Virtual-Channels
Start with the paper – Design of a Low-Latency Virtual-Channel Router

3. Why NoC Billion transistor era has arrived
Several such SoC are in pipeline, Inter-connection is critical
A generic inter-connection architecture ensures
Reduced design time
IP reuse
Predictable backend (versus ad-hoc wiring)
Bus based inter-connects were sufficient until now
But not now
Shared bus is slow (arbitrates between several requesters)
More components increase loading => speed drops further
Ad-hoc routing of wires results in backend complications, lower performance and higher power consumption

4. Why NoC (cont) Recently Dally proposed an idea
“Route packets not wires” as in data networks
Point to point communication
Point to point links are faster
Create a chip wide network (Like a regular IP WAN)
A router at every node
Links connecting all routers
Messages encapsulated in packets, which are routed
Challenges
Topologies, Routing protocol
Network and router design with small footprint and low latency

5. Some more motivations
The need to put repeaters into long wires allows us to add the switching needed to implement a network at little additional cost
Makes efficient use of critical global wiring resources by sharing them across different senders and receivers
Simplifies overall design
Design a single router and do copy-paste in both dimension

6. A typical NoC node Layered Design of reconfigurable micronetworks.
Exploits methods and tools used for general network.
Micronetworks based on the ISO/OSI model.
NoC architecture consists of Physical, Data link, and Network layers.

7. Implemented in cores, enables end-to-end reliable transport
A typical NoC
Layered Design of reconfigurable micronetworks.
Exploits methods and tools used for general network.
Micronetworks based on the ISO/OSI model.
NoC architecture consists of Physical, Data link, and Network layers.
Implemented in cores, enables end-to-end reliable transport

8. A typical NoC Layered Design of reconfigurable micronetworks.
Exploits methods and tools used for general network.
Micronetworks based on the ISO/OSI model.
NoC architecture consists of Physical, Data link, and Network layers.
Implemented in cores, enables end-to-end reliable transport
Multi hop route setup, packet addressing, etc

9. A typical NoC Layered Design of reconfigurable micronetworks.
Exploits methods and tools used for general network.
Micronetworks based on the ISO/OSI model.
NoC architecture consists of Physical, Data link, and Network layers.
Implemented in cores, enables end-to-end reliable transport
Multi hop route setup, packet addressing, etc
Contention issues, reliability issues, grouping of physical layer bits, e.g. “flits”

10. A NoC topology Cores Communicates With Each Other Using NoC
NoC Consists of Routers (R) and Network Interfaces (NI)
A NI linked to Router by Non-Pipelined Wires
One or More Cores Connected to a NI

11. Another NoC topologies
Fat tree
Mesh
Multi hop route setup, packet addressing, etc

12. Routing protocols We will only consider mesh topology
Objective is to find a path from
a source to a destination
Greedy Algorithms (deterministic)
Choose shortest path (e.g. X-Y)
Adaptive routing
If congestion, choose alternative path
Deflection routing
Is adaptive better than greedy => NOT REALLY (when only local information is used)
Adaptive routing can also result in livelock

13. Switching techniques
Circuit Switching: A control message is sent from source to destination and a path is reserved. Communication starts. The path is released when communication is complete.
Store-and-forward policy (Packet Switching): each switch waits for the full packet to arrive in switch before sending to the next switch
Cut-through routing or worm hole routing: switch examines the header, decides where to send the message, and then starts forwarding it immediately
In worm hole routing, when head of message is blocked, message stays strung out over the network, potentially blocking other messages (Needs only buffer the piece of the packet that is sent between switches).
Cut through routing lets the tail continue when head is blocked, storing the whole message into an intermediate switch. (Need buffer large enough to hold the largest packet).

14. Wormhole Routing – Good fit for NoC
Wormhole routing is good for NoC
Low latency
Less buffering requirements
Suffers from deadlock
[example from Li and McKinley,
IEEE Computer v26n2, 1993]

15. Adding Virtual Channels
With virtual channels, deadlock can be avoided
Move message and reply on different channels => Will never have loop on a single channel

16. Designing Virtual Channel Routers
Design Constraints in NoC
Minimize Latency
Minimize Buffering
Minimal footprint
Can exploit far greater number of pins and wires
May use fat data and flow control wires
Objective: Design routers with minimal latency
This will also result in smaller buffers
This paper presents design of a low latency router
Cycle time of 12 FO4
Single cycle routing/switching

17. Designing Virtual Channel Routers
Design Constraints in NoC
Minimize Latency
Minimize Buffering
Minimal footprint
Can exploit far greater number of pins and wires
May use fat data and flow control wires
Objective: Design routers with minimal latency
This will also result in smaller buffers
This paper presents design of a low latency router
Cycle time of 12 FO4
Single cycle routing/switching

18. A Virtual Channel Router

19. Designing Virtual Channel Routers
Every VC of every input port has buffers to hold arriving flits
Arriving flits are placed into the buffers of corresponding VC

20. Designing Virtual Channel Routers
Every VC of every input port has buffers to hold arriving flits
Routing logic assigns set of outgoing VC on which flit can go
Arbitrates between competing input VC & allocates output VC
Arriving flits are placed into the buffers of corresponding VC

21. Designing Virtual Channel Routers
Every VC of every input port has buffers to hold arriving flits
Routing logic assigns set of outgoing VC on which flit can go
Arbitrates between competing input VC & allocates output VC
Arriving flits are placed into the buffers of corresponding VC
Matches successful input ports (allocated VC) to output ports
Flits at input VCs getting grants are passed to output VCs

22. Routing Logic Three possibilities Look ahead routing
Return a single VC
Return set of VCs on a single port
Return any VCs
Look ahead routing
Routing performed at the previous router
Good for X-Y deterministic (non adaptive) routing
A SGI routing chip first implemented it

23. VC Allocation Complexity of VC allocation depends on routing range
Routing returns single VC
Needs PxV input arbiter for every outgoing VC
Routing returns multiple VC at single port
Additional V:1 arbiter at every input VC to reduce potential outgoing VC to 1
Routing returns any set of VCs
Needs two cascaded PxV input arbiters
We consider multiple VC at single port case

24. VC Allocation Logic
At every outgoing VC following logic is needed

25. Switch Allocation
Individual flits at input VCs arbitrate for access to the crossbar port
Arbitration can be performed in two stages
First stage
A VC among V possible VCs at every input port is selected
V:1 arbiter at every input port
Second stage
Winning VC at every input port is matched to the output port
P:1 arbiter at every output port
This scheme doesn’t guarantee a maximal/maximum/good matching
But simple to implement

26. Switch Allocation

27. Issues VC allocation and Switch allocation are serialized
A flit will either take 2 clocks to get through
Else clock speed will be low
Solution: Speculative switch allocation

28. Speculative Switch Allocation
Dally proposed speculative switch allocation
Perform switch and VC allocation in parallel
Assume that participating VC in switch allocation will get the output VC
If not then wasted cycle
An even better idea is to perform speculative and non-speculative switch allocation in parallel
Non-speculative allocation has higher priority
Note that non-speculative allocation is done for input VCs which has already been allocated an output VC
Mostly one cycle delay under light load
Mostly one cycle delay under heavy load
Speculative will work
Non-speculative will work

29. Further Enhancement
Is it possible to have zero cycle VC/switch allocation
YES, Most of the time, that’s what this paper is about!

30. Idea 1: Free Virtual Channel Queue
Keep queue of free VC at every outgoing port
Also bit mask with one set bit
Thus First stage of VC allocation where an output VC is selected will be removed

31. Idea 1: Free Virtual Channel Queue
Keep queue of free VC at every outgoing port
Also bit mask with one set bit
Thus First stage of VC allocation where an output VC is selected will be removed

32. Idea 2: Pre-computing arbitration decisions
If somehow, you know the arbitration results before flits actually arrive and fight for the VC and switch
I mean, every arbitration decision
VC allocation
Switch allocation
Etc
Then the router can be made to run in zero cycle
The arriving flit route/switch in the same clock they arrive
Also, clock speed may be pretty good
Data path and control path are no more in series
That’s what the idea 2 is.

33. Some preliminaries before going into detail
Tree Arbiters
Implements large arbiters using tree of small arbiters
Matrix Arbiters
Fair and Fast arbiter implementation

34. VC allocation using a Tree Arbiter

35. A Matrix Arbiter

36. Pre-computing arbitration decisions
An alternative arbiter design

37. Pre-computing arbitration decisions
An alternative arbiter design
Generate grant enables one cycle prior and latch them
Grants are product of latched enables and the requests

38. Pre-computing arbitration decisions
An alternative arbiter design
Grants are generated in same clock as request arrives
If at least one request remains
Generate grant enables one cycle prior and latch them
Grants are product of latched enables and the requests

39. Pre-computing arbitration decisions
An alternative arbiter design
However, when no request remains, it is difficult to generate grant enables ???
Generate grant enables one cycle prior and latch them
Grants are product of latched enables and the requests

40. Generating grant enables
Safe Environment
Only one request may arrive in a cycle
Thus it is safe to assert all grant enables
Thus grant can still be generated in same cycle
Unsafe Environment
Multiple request may arrive in same cycle
Can still assert all grants
But need to abort when multiple requests arrive in same cycle
All first stage V:1 arbiters operate under safe environment
However P:1 arbiters doesn’t

41. Generating grant enables
Even in unsafe environments, assert all grants
May need to abort when multiple requests arrive
Note that after an abort, a correct arbitration is ensured in the next cycle
Why will it work?
Because in lightly loaded network, multiple requests for same VC/port will not arrive (few aborts)
In heavily loaded network flits will remain buffered and Non-speculative arbitration (higher priority) will happen most of the time
Few aborts again

42. I will skip the design details now
Since it is confusing and complex
Will jump to critical path analysis

43. Analysis of critical path
Generates VC/switch grants from pre-computed grant enables

44. Analysis of critical path
Generates VC/switch grants from pre-computed grant enables
Crossbar traversal is aborted once invalid grants are detected

45. Analysis of critical path
In case of an abort, the correct control signals are ensured in the next cycle
Generates VC/switch grants from pre-computed grant enables
Crossbar traversal is aborted once invalid grants are detected

46. Final design Control path critical delay is 12 FO4
Until now, the best design had 20 FO4 delays
They have sampled a NoC based ASIC last week using this idea
Runs at several GHz speeds
Note that fast cycle time is possible by
Running VC allocation and Switch allocation in parallel
Must use speculation, else delay will be higher (1 more cycle)

47. Simulation results

48. If (doubts) Then
Ask;
Else
Thank you;
Goto Discussion;
End if;