Ghent University Pattern recognition with CNNs as reservoirs David Verstraeten 1 – Samuel Xavier de Souza 2 – Benjamin Schrauwen 1 Johan Suykens 2 – Dirk Stroobandt 1 – Joos Vandewalle 2 1 ELIS, Ghent University 2 ESAT, KU Leuven Belgium NOLTA 2008 – September 8, 2008

2/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Outline Introduction: what is Reservoir Computing? Reservoir Computing with CNNs Simulation results using grid-search optimization On-chip results for optimization using CSA Conclusions and future challenges

3/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Reservoir Computing Novel method for training recurrent neural networks Internal weights and input weights are randomly drawn (gaussian), globally scaled and left untrained Linear output layer is trained using e.g. ridge regression Input layer Output layer Reservoir

4/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Reservoir Computing Functionality is similar to a kernel, but works for dynamic signals Reservoir offers rich pool of nonlinear dynamic transformations of the input to the linear output layer Applied to: speech recognition, time series prediction, robot control,... Input layer Output layer Reservoir

5/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Reservoir Computing with CNNs ACE16k: very fast hybrid aVLSI implementation. Limitations compared to traditional reservoirs: 2D topology Uniform internal weight template Dynamic node behavior For computational reasons, we only use the central 8x8 cells

6/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Grid search template optimization Search space for traditional reservoirs is huge: N^2-dimensional for N neurons Search space for CNN templates is smaller, especially when we restrict it Diagonal Lateral Self

7/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Academic task, rather easy: proof of concept Input signal switches randomly between sawtooth and square wave Main difficulty lies at transitions between signals Input Output Experiment 1: simple signal classification

8/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Experiment 1: simple signal classification

9/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Isolated spoken digits, ‘zero’ to ‘nine’ Subset of TI46 Two different female speakers Every digit uttered 10 times  200 samples Preprocessing using biological model of cochlea. Experiment 2: spoken digit classification Readout + post-processing

10/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Experiment 2: spoken digit classification

11/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Template optimization using CSA CSA: Coupled Simulated Annealing Extension of traditional simulated annealing Better convergence and less sensitivity to initial conditions

12/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Template optimization using CSA : results Signal classificationDigit recognition No reservoirN/A11% Classic RC1%2% Simulated CNN RC1%3.6% On-chip CNN RC0.1%6%

13/13 Pattern recognition with CNNs as reservoirs – David Verstraeten Ghent University - Faculty of Engineering – Electronics and Information Systems Dept. Conclusions and future challenges Novel use of CNNs for computation: Reservoir Computing Proof-of-concept : academic and real world task Both in simulation and on-chip Future challenges: On-chip readout? How to get it trained (large dataset)? Can we use the dynamic behavior of the nodes?