1. OpenSoC Fabric An open source, parameterized, network generation tool
CoDEx
An open source, parameterized, network generation tool
Farzad Fatollahi-Fard, David Donofrio,
George Michelogiannakis, John Shalf
SoC for HPC Programmatic Meeting
August 25-26, Denver, CO.

2. 1 2 3 4 5 Motivation State of the Art in SoC OpenSoC Overview
Power, parallelism, and data movement drive the need for SoC 1
State of the Art in SoC
What technology is being used to build SoCs? What can we leverage? 2
OpenSoC Overview
An open source network generator using a new HDL - Chisel 3
OpenSoC Code Deep Dive and Demo
OpenSoC module walk through and network output 4
Conclusion and Future Work
New features for OpenSoC 5

3. Power: The New Design Constraint
Trends beginning in 2004 are continuing…
Power densities have ceased to increase
No power efficiency increase with smaller transistors
So, why aren’t we at exascale yet?

4. Power: The New Design Constraint
On-chip parallelism increasing to maintain performance increases…
We have come to the end of clock frequency scaling
Moore’s Law is alive and well
Now seeing core count increasing

5. Parallelism increasing
NERSC Trends
Franklin
Hopper
Edison
Cori (NERSC 8)
Core Count 4 24
48 (logical)
>60
Clock Rate
2.3GHz
2.1GHz
2.4 GHz
~1.5GHz
Memory
8GB
32GB
64GB
64-128GB +On package
Peak Perf
.352 PF
1.288 PF
2.57 PF
> 3 TF
Revisit of how SoC, increased parallelism in general is playing out in a real world system

6. Hierarchical Power Costs
Data movement is the dominant power cost
120 pJ
2000 pJ
250 pJ
~2500 pJ
100 pJ
6 pJ
Cost to move data off chip to a neighboring node
Cost to move data off chip into DRAM
Cost to move off-chip, but stay within the package (SMP)
Cost to move data 20 mm on chip
Typical cost of a single floating point operation
Cost to move data 1 mm on-chip
Let’s put aside cooling, building efficiency and focus on one part of data center power consumption – the processor. Data movement is the dominant power consumer, driving the benefit / need for SoCs

7. 1 2 3 4 5 Motivation State of the Art in SoC OpenSoC Overview
Power, parallelism, and data movement drive the need for SoC 1
State of the Art in SoC
What technology is being used to build SoCs? What can we leverage? 2
OpenSoC Overview
An open source network generator using a new HDL - Chisel 3
OpenSoC Code Deep Dive and Demo
OpenSoC module walk through and network output 4
Conclusion and Future Work
New features for OpenSoC 5

8. Current Hardware Challenges
Embracing embedded designs
Power is now limiting factor for leading-edge chips
Moore’s Law continues… but now reliant to more exotic technologies such as 3D transistors, FinFET etc.
Design Validation / Verification dominating development costs
Solution: Smaller is Better
Simpler, 5 to 9 stage pipeline cores
Parallel is key to energy efficiency: CV2F
Large arrays of small simple cores are easier to verify, more resilient
Vibrant commodity market in IP components

9. Building an SoC from IP Logic Blocks It’s Legos with a some extra integration and verification cost
Processor Core (ARM, Tensilica, MIPS deriv)
With extra “options” like DP FPU, ECC
OpenSoC Fabric (on-chip network)
(currently proprietary ARM or Arteris)
Mem
Control
DDR memory controller
(Denali/Cadence, SiCreations)
+ Phy & Programmable PLL
DRAM
Mem
Control
Memory
PCIe
FLASH Control
IB or GigE
IB or GigE
DRAM
PCIe Gen3 Root complex IO
Integrated FLASH Controller
10GigE or IB DDR 4x Channel

10. SoC – What’s out there? A few examples… AMD Opteron Intel Crystalwell
CPU, Gfx and 128MB eDRAM
Xeon Phi
KNC, KNL
Intel Integrated IO + Data Direct IO
PCI Controller integrated on die
7W power savings by bypassing processor cache and memory
ARM
Apple A7
AMD Opteron
CPU, PCIe + SATA Controller

11. SoC – What’s out there? Some common features Cost driven
CPUs integrated with IO and Gfx to reduce BOM cost, decrease total PCB area
Die power density / pin constraint driven
Homogeneous cores
Simple Networks
Most SoCs rely on ring or cross-bar interconnect
Increasing core count will drive need for more complex topologies
KNL present in 2016 NERSC machine will have mesh

12. What Interconnect Provides the Best Power / Performance Ratio?
What tools exist to answer this question?

13. SoC - Interconnect Examples
Some common topologies

14. The Importance of Network Topology
Network topology can greatly influence application performance
An analysis of on-chip interconnection networks for large-scale chip multiprocessors
ACM Transactions on computer architecture and code optimization (TACO), April 2010

15. The Importance of Networks
Networks consume a large fraction of total chip power...
A 5-GHz Mesh Interconnect for a Teraflops Processor.
IEEE Micro. 2007

16. What tools exist for SoC research
What tools do we have to evaluate large, complex networks of cores?
Software models
Fast to create, but plagued by long runtimes as system size increases
Hardware emulation
Fast, accurate evaluate that scales with system size but suffers from long development time
A complexity-effective architecture for accelerating full-system multiprocessor simulations using FPGAs. FPGA 2008

17. Software Models C++ based on-chip network simulators Booksim
Cycle-accurate
Verified against RTL
Few thousand cycles per second
Garnet
Event driven
Simulation speed limits designs to 100’s of cores
Booksim ISPASS 2013
GARNET ISPASS 2009

18. Hardware Models Stanford open- source NoC router
HDL network generators and implementations
Stanford open- source NoC router
Verilog
Precise but long simulation times
Connect network generation
Bluespec
FPGA Optimized
CONNECT: fast flexible FPGA-tuned networks-on-chip. CARL 2012

19. 1 2 3 4 5 Motivation State of the Art in SoC OpenSoC Overview
Power, parallelism, and data movement drive the need for SoC 1
State of the Art in SoC
What technology is being used to build SoCs? What can we leverage? 2
OpenSoC Overview
An open source network generator using a new HDL - Chisel 3
OpenSoC Code Deep Dive and Demo
OpenSoC module walk through and network output 4
Conclusion and Future Work
New features for OpenSoC 5

20. Chisel: A New Hardware DSL
Using Scala to construct Verilog and C++ descriptions
Chisel provides both software and hardware models from the same codebase
Object-oriented hardware development
Allows definition of structs and other high- level constructs
Powerful libraries and components ready to use
Working processors fabricated using chisel

21. Recent Chisel Designs Chisel created cores successfully boot Linux
Clock test site
SRAM test site
DCDC test site
Processor Site
Raven core – 28nm

22. Chisel Overview Not “Scala to Gates” Describe hardware functionality
How does Chisel work?
Not “Scala to Gates”
Describe hardware functionality
Chisel creates graph representation
Flattened
Each node translated to Verilog or C++

23. Chisel Overview All bit widths are inferred Clock and reset implied
How does Chisel work?
All bit widths are inferred
Clock and reset implied
Multiple clock domains possible
IOs grouped into convenient bundles

24. OpenSoC Fabric Part of the CoDEx tool suite, written in Chisel
An open source, flexible, parameterized, NoC generator
Part of the CoDEx tool suite, written in Chisel
Dimensions, topology, VCs all configurable
Fast functional C++ model for functional validation
SystemC ready
Verilog based description for FPGA or ASIC
Synthesis path enables accurate power / energy modeling
AXI Based endpoints
Ready for ARM integration

25. OpenSoC Fabric
An open source, flexible, parameterized, NoC generator

26. OpenSoC: Current Status
Projected v1.0 release date of October 1st
Available now:
2-D mesh or Flattened Butterfly network of arbitrary size
Wormhole routing
In Development
Virtual Channels
AXI Interface

27. 1 2 3 4 5 Motivation State of the Art in SoC OpenSoC Overview
Power, parallelism, and data movement drive the need for SoC 1
State of the Art in SoC
What technology is being used to build SoCs? What can we leverage? 2
OpenSoC Overview
An open source network generator using a new HDL - Chisel 3
OpenSoC Code Deep Dive and Demo
OpenSoC module walk through and network output 4
Conclusion and Future Work
New features for OpenSoC 5

28. OpenSoC Code Examples
Walkthrough of Switch and Full Network Tester

29. 1 2 3 4 5 Motivation State of the Art in SoC OpenSoC Overview
Power, parallelism, and data movement drive the need for SoC 1
State of the Art in SoC
What technology is being used to build SoCs? What can we leverage? 2
OpenSoC Overview
An open source network generator using a new HDL - Chisel 3
OpenSoC Code Deep Dive and Demo
OpenSoC module walk through and network output 4
Conclusion and Future Work
New features for OpenSoC 5

30. Future additions Photonics and circuit switched networks
Towards a full set of features
Photonics and circuit switched networks
Integrated NIC model
More diverse topologies and routing functions
Validation against RTL and other simulators
Standardized (AXI) interfaces at the endpoints
More powerful synthetic traffic and trace replay support
Power modeling in the C++ model

31. Acknowledgements UCB Chisel US Dept of Energy Ke Wen Columbia LRL
John Bachan
Dan Burke
BWRC

32. More Information