Insect neural networks as a visual collision detection mechanism in automotive situations Richard Stafford (1), Matthias S. Keil (2), Shigang Yue (1), Jorge Cuadri- Carvajo (2), F. Claire Rind (1) 1) School of Biology, Ridley Building, University of Newcastle upon Tyne, NE1 7RU, UK. 2) Instituto de Microelectronica de Sevilla (IMSE), Centro Nacional de Microelectronica (CNM), Avda. Reina Mercedes, 41012, Sevilla, Spain

Structure of the talk  Introduction  Improvements to the LGMD model Light on and light off pathways  Testing the LGMD model Methods and Results  Further improvements Biologically inspired filtering of images by lateral inhibition Analysing the filtered images using EMD like structures  Conclusions

Locusts as collision detectors  The Lobula Giant Movement Detector (LGMD) neuron responds most vigorously when objects of certain speeds and sizes approach, as if on a direct collision course  This has been linked to a predator avoidance, gliding behaviour in flying locusts

Predator avoidance caused by the LGMD Angular subtense of object LGMD Spikes

Inputs and structure of the LGMD

Why use the Locust LGMD to detect automotive collisions?  Evolutionary honed collision avoidance system  Efficient circuit – based on insect neurons  Neural architecture well studied  Responds optimally to imminent collisions  Simulated networks respond in a similar manner to real locust

Limitations of existing model (e.g. Rind and Bramwell, 1996; Blanchard et al., 2000)  Simulations only tested in simple closed environment  Model needs to work in real automotive situations  Biology of the LGMD is not fully used – model only responds to change in light

Structure of the talk  Introduction  Improvements to the LGMD model Light on and light off pathways  Testing the LGMD model Methods and Results  Further improvements Biologically inspired filtering of images by lateral inhibition Analysing the filtered images using EMD like structures  Conclusions

Model Improvements – Light on and Light off Pathways  Small scale spatial antagonism between the pathways helps eliminate noise caused by vibration etc.  Larger scale antagonism can interfere with collision alerts

Model Improvements – Light on and Light off Pathways and Block Sum Cells Input Image‘S’ units – light on ~ light off Block Sum Cells Allow small scale antagonism of pathways only

Location of BSC in model Block sum cells occur here

Model Improvements - Block Sum Cells Sum light on (-ve) and light off (+ve) excitation to obtain net excitation Excitation (+ve only) is passed to the LGMD from the BSC Block sum cells obtain excitation from a 10x10 section of the array of ‘S’ units Light on and Light off excitation from ‘S’ units Block Sum Cells LGMD

Structure of the talk  Introduction  Improvements to the LGMD model Light on and light off pathways  Testing the LGMD model Methods and Results  Further improvements Biologically inspired filtering of images by lateral inhibition Analysing the filtered images using EMD like structures  Conclusions

Testing the model in automotive situations Input video sequences 8 – 25 Hz Input via frame- grabber of Playstation images 8.3 Hz

Detecting collisions  Membrane potential of LGMD is obtained from sum of BSC  If a threshold is exceeded then the LGMD produces spikes  If > 2 spikes in 3 timesteps then collision detected

Results: LGMD model Results show % of times collision was detected even if no collision occurred Stationary car 100 % Moving Car 90 % Head on with moving Car 100 % Entering Tunnel 0 % General Driving 0 % Driving in close proximity 0 % Translating cars 70 %

Why do translating cars prove problematic? Excitation is much higher in the LGMD for translating objects Locust LGMD ignores translating objects partially due to differences in mathematics of object approach

Structure of the talk  Introduction  Improvements to the LGMD model Light on and light off pathways  Testing the LGMD model Methods and Results  Further improvements Biologically inspired filtering of images by lateral inhibition Analysing the filtered images using EMD like structures  Conclusions

Image Filtering by LGMD network ‘S’ units only excited by objects moving in close proximity to car e.g. Colliding or translating objects No threatThreat Input Image ‘S’ units

Analysing the biologically filtered images  Analysing patterns of excitation in ‘S’ or ‘BSC’ layers over time shows: No or little excitation – no threat. LGMD does not reach threshold Excitation moving in one direction over time – no threat, translating object. LGMD spikes can be suppressed Excitation moving in all directions over time – collision threat, object on collision course is expanding in all directions. LGMD spikes and produces collision mitigation response

Structure of the talk  Introduction  Improvements to the LGMD model Light on and light off pathways  Testing the LGMD model Methods and Results  Further improvements Biologically inspired filtering of images by lateral inhibition Analysing the filtered images using EMD like structures  Conclusions

Incorporation of simple Elementary Movement Detector like units (EMDs) into the model  EMD like units take input from the Block Sum Cells – simplified visual environment  One detected ‘Right’ movement and one ‘Left’ movement  If membrane potential of ‘left’ EMDs was > 5 x potential of ‘right’ EMDs at time t or time t-1 then LGMD spikes were suppressed for time t, t+1 & t+2

Location of EMD like units BSC EMDs Suppression of LGMD spikes

Results: LGMD incorporating EMDs Results show % of times collision was detected even if no collision occurred Stationary car 85 % Was 100 % Moving Car 80 % Was 100 % Head on with moving Car 50 % Was 100 % Entering Tunnel 0 % Unchanged General Driving 0 % Unchanged Driving in close proximity 0 % Unchanged Translating cars 20 % Was 70 %

Results: LGMD and EMDs  Incorporation of EMDs reduce false collision alerts  Real collision detection was also reduced  EMD model was very simple. Using a more advanced (adaptive) model may improve the responses  Non bio-inspired image analysis could also be used on the biologically filtered ‘S’ units to improve model performance

Conclusions  Locust based LGMD model can be used for automotive collision detection  In some situations modifications are needed as the LGMD’s function in automotive situations is quite different to the evolved function in the locust  The biologically filtered image can be analysed to further assess the threat of collision

Acknowledgements  Project funded by Future and Emerging Technologies Grant from European Union (LOCUST – IST )  We would like to thank Marrti Soininen of Volvo Car Corporation for supplying the video footage of car crashes

Other Improvements to the LGMD model  On-Off cells look at absolute change in image  Lateral inhibition has a greater potential spread to eliminate more non threatening situations  Spiking threshold of LGMD is self variable to allow a greater range of visually complex scenes to be investigated  Model parameters tuned using a Genetic Algorithm to automotive situations

Differences between automotive collisions and predator avoidance in locusts  Locusts respond to small, fast moving predators  Final excitation, just before predator strikes, is much higher  This can be used to distinguish between different object types  Small translating objects produce less excitation than larger objects