1. Richard Dorrance Literature Review: 1/11/13
Chapter 11 in Systems and Circuit Design for Biologically-Inspired Intelligent Learning
FPGA Coprocessor for Simulation of Neural Networks Using Compressed Matrix Storage
Richard Dorrance
Literature Review: 1/11/13

2. Overview Binary neuron model for unsupervised learning
arranged into minicloumns and macrocloumns
Sparse matrix vector multiplication (SpMxV)
compressed row format
software optimizations
FPGA coprocessor
exploiting matrix characteristics for a “novel” architecture
benchmarks and scalability (or lack there of)

3. Neural Network Example

4. Binary Neuron Model Modeled as McCulloch-Pitts neurons (i.e. binary):
only 2 states: firing (1) and not firing (0)
fixed time step t
refractory period of one time step

5. Macrocolumn Dynamics

6. Cortical Macrocolumn Connection Matrix

7. Feature Extraction: Bars Test
Introduce learning with “Hebbian plasticity” (i.e. updating neuronal links)

8. Compress Row Format Sparse matrix representation (w/ 3 vectors): value
column index
row pointer

9. SpMxV is the Bottleneck
Theoretically SpMxV should be memory bound
Reality: lots of stalling for data due to irregular memory access patterns
Coding Strategies:
cache prefetching
matrix reordering
register blocking (i.e. N smaller, dense matrices)
CPU: 100 GFLOPS (theoretical), 300 MFLOPS (reality)
GPU: 1 TFLOPS (theoretical), 10 GFLOPS (reality)

10. Simplifications and Optimizations
Matrix elements are binary: value vector is dropped
Strong block-like structure to matrices: compress column index vector

11. FPGA Coprocessor Architecture

12. Resource Usage

13. Scalability

14. Conclusions
SpMxV is the major bottleneck to simulating neural networks
Architectures of CPUs and GPUs limit performance
FPGAs can increase performance efficiency
Specialized formulation of SpMxV  limited scalability