1. Ghent University Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks September 13, 2006 - ICANN 2006 Michiel D'Haene, Benjamin Schrauwen and Dirk Stroobandt Electronics and Information Systems Department Ghent University www.elis.UGent.be/SNN/

2. 2 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Outline Introduction Threshold models Delayed firing problem Efficient processing of multi-synapse neurons Results Conclusion

3. 3 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Introduction Spiking Neural Networks: Neurons use spikes to communicate Intrinsically able to process temporal information Computationally more powerfull as analog neurons [Maass] Biologically more plausible w1w1 w2w2 w3w3 inputoutputneuron

4. 4 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Introduction: threshold models Incoming spikes influence the membrane potential When certain threshold θ is reached: reset and fire Ex. Spike Response Model 0 th orde (SRM 0 ) with exp. synapses t input output membr θ

5. 5 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Introduction Two ways of simulating SNN: Time-step driven Event-driven Time-step driven simulations Still most used: simple and generally applicable Calculate a new state for each neuron at each time-step Asynchronous nature of spikes: small time-steps required to reach adequate accuracy Low activity: inefficient

6. 6 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Introduction Event-driven simulation: Usually: only external behaviour of a neuron is considered Thus: evaluate neurons only when necessary: new spike arrival firing a spike to other neurons Benefits from the typical low activity in neural networks Very high accuracy Ex. ESSpiNN (our event-based simulator) is approx. 60 times faster than the time-step based CSIM

7. 7 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Event-driven simulation Neurons must have discontinuous behaviour Spikes must be events with a well defined time-stamp No Hodgkin-Huxley neurons (without concessions) instantanious firing membr t t t input output The simulator has to predict the time a neuron will fire In a Leaky Integrate and Fire (LIF) model, this is rather simple θ

8. 8 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Delayed firing t input output θ m(t) Event-simulator schedules the next fire time of neurons Each time a spike is received: ➔ Update the synapse values ➔ Calculate next fire time (assuming no other spikes arriving) Solve the membrane function m(t) for time with m(t) = θ  In many cases not analytically solvable! When using more common neuron models, one has to deal with the so called delayed fire problem

9. 9 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Delayed firing Several techniques developed to solve this problem Developed for one or restricted number of synapse models neuron model Multiple synapses: Each time a spike arrives on one of the inputs: update current synapse potential for each synapse Estimate new fire timestamp using the dynamic functions of all synapses. w1w1 w2w2 w3w3 s 2 (t ) s 1 (t ) s 3 (t ) w1w1 w2w2 w3w3 s 1 (t )

10. 10 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Multi-synapse neurons Update time grows lineary with the number of synapses Neurons with 100 or 1000 synapses or not unusual Estimating new fire-timestamp has to be done iteratively Requires many calculations Next: optimizations applied on LIF neurons with exponential decaying synapses Also applicable to other neuron models

11. 11 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Optimizations Don't calculate synapses with very small values (also used in advanced time-step based simulators like CSIM) Use lookup tables for exponentials, Newton-Rhapson for threshold crossing

12. 12 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Optimizations Spikes are rather sparse, in most cases a neuron will not fire when a spike arives Approx. max. influence of each synapse function Incoming spike: only update this synapse Possibility for neuron to fire? Yes: start updating other synapses to current time Meanwhile, adapt max. influence of other synapses £ t total membr. pot. t t S1 S2

13. 13 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Optimizations In many cases, max. approx. drops again below the threshold value Updating the oldest synapses first gives most change to decrease the max. approx. Keep neurons sorted by the last update time Circular doubly linked list: no sorting action needed! Array pointing to each element or to NULL: fast access contains also active synapse structure s = 8 t = 1 s = 7 t = 3 s = 2 t = 4 s = 1 t = 7 s = 4 t = 9 1 start 2345678 NULL end

14. 14 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Results (1) Tested on randomly interconnected networks 1000 neurons (, spectral radius of 0.9***) Input spikes applied on 10% of the neurons We compared our model with a neuron model of Olaf Booij Single synapse model Exponential decay of synapse with τ s and membrane with τ m τ m = 2x τ s -> analytical solvable as quadratic equation

15. 15 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Results (2) 100 synapse inputs per neuron less efficient for very high firing rates (>1000 spikes/sec) Booij model and LIF model are equally fast!

16. 16 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Results (3) Scaling in number of synapses per neuron

17. 17 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Conclusions Multi-synapse neurons require much computational power in an event-driven scheme Our optimisations show significant speedup (> 5 times) Moreover the scalability in number of synapses is better Future work: Adding more complex models (ex. conductance-based) Further accelerate the simulator by using parallel simulators Hardware acceleration www.elis.UGent.be/SNN/

18. 18 /15 Accelerating Event Based Simulation for Multi-Synapse Spiking Neural Networks ICANN 2006 – September 13, 2006 Delayed firing Several techniques developed to solve this problem Use only very restricted single-synapse models Precalculated lookup-tables -> size grows linear with number of synapses and exponential with number of parameters Iterative techniques -> computational demanding operations