1. Treating Epilepsy via Adaptive Neurostimulation Joelle Pineau, PhD School of Computer Science, McGill University Congress of the Canadian Neurological Sciences Foundation June 9, 2010

2. Adaptive Neurostimulation 2 Joelle Pineau Learning objectives Review the neurostimulation hypothesis for treating epilepsy. Understand the basic principles of adaptive neurostimulation. Study a mathematical framework for optimizing the choice of neurostimulation parameters. Observe results from applying adaptive neurostimulation in vitro.

3. Adaptive Neurostimulation 3 Joelle Pineau Disclosure statement This research was supported by the Natural Sciences and Engineering Research Council of Canada and the Canadian Institutes of Health Research. This is joint work with Massimo Avoli, MD, PhD; Keith Bush, PhD; Arthur Guez; Gabriella Panuccio, MD, PhD; and Robert Vincent.

4. Adaptive Neurostimulation 4 Joelle Pineau Epilepsy Epilepsy is a neurological disorder marked by spontaneous seizures. –Affects ~1% of world's population. –Up to 20-25% of those do not benefit from standard treatments (anti- convulsants, surgery). Causes are varied (pre-disposition, head trauma, fever, tumor, etc.) What is a seizure? –Abnormal electrical activity in the brain, may produce physical convulsions, or other symptoms. www.chse.louisville.edu/graphics/epilepsy07.jpg

5. Adaptive Neurostimulation 5 Joelle Pineau Anatomy of a seizure (in vitro)

6. Adaptive Neurostimulation 6 Joelle Pineau Neurostimulation hypothesis External perturbation of an epileptic neural system can alter dynamics away from excitability.

7. Adaptive Neurostimulation 7 Joelle Pineau Deep brain stimulation (DBS) Implanted electrodes electrically stimulate brain tissue. Recent clinical trials of DBS: –Stimulation of the Anterior Nucleus of the Thalamus in Epilepsy (SANTÉ) »157 people; 17 sites in US; 2003-2008; sponsored by MedtronicNeuro –Randomized Controlled Trial of Hippocampal Stimulation for Temporal Lobe Epilepsy (METTLE) »90 people; 1 site in Canada; 2008-2011; sponsored by U. of Calgary –RNS TM System Long-Term Treatment Clinical Investigation: »280 people; 28 sites in US; 2006-2013; sponsored by NeuroPace »Closed-loop stimulation “detect-then-stimulate” Many parameters to control: stimulation site, stimulation frequency/intensity, stimulation pattern, … www.medgadget.com www.onemedplace.com

8. Adaptive Neurostimulation 8 Joelle Pineau Electrophysiology results from in vitro model The literature repeatedly shows 1Hz, 10-200  A, fixed stimulation successfully suppresses seizures in vitro. [D’Arcangelo et al., Neurob of Disease. 2005].

9. Adaptive Neurostimulation 9 Joelle Pineau A few interesting open questions What parameter settings (stimulation site, frequency, intensity, pattern) achieve maximal suppression? Can we reduce the number and/or intensity of stimulations, while maintaining suppression efficacy? How can we customize parameters for different subjects?

10. Adaptive Neurostimulation 10 Joelle Pineau Adaptive neurostimulation paradigm Objective: create a stimulation device which is 1.Optimal: maximize seizure reduction + minimize stimulation. 2.Responsive: strategy evolves as a function of the observation. 3.Automatic: stimulation strategy learned from data.

11. Adaptive Neurostimulation 11 Joelle Pineau Adaptive neurostimulation example

12. Adaptive Neurostimulation 12 Joelle Pineau Methods: Data collection in vitro Electrophysiological recording in the Entorhinal Cortex (B), with stimulation at fixed frequencies in the Subiculum (A).

13. Adaptive Neurostimulation 13 Joelle Pineau Methods: Data collection and labeling Experimental protocol: Control (min. 3 seizures) Periodic pacing at 0.2Hz (min. 20 minutes). Recovery (until interval between ictal events stabilizes). Etc. with 0.5Hz, 1.0Hz, 2.0Hz. Then, manually identify: Seizure occurrences Neurostimulation parameters: {0Hz, 0.2Hz, 0.5Hz, 1.0Hz, 2.0Hz}

14. Adaptive Neurostimulation 14 Joelle Pineau Methods: Signal processing Select decision window duration: 1 sec. Select observation window duration: 13 sec. Extract observation features using signal processing techniques: –Range, energy, multi-scale Fourier transform

15. Adaptive Neurostimulation 15 Joelle Pineau Methods: Training data Form an input vector, x t, for each decision window, t : x t = {z t, a t, c t, z t+1 } where z t = observation features at t a t = neurostimulation parameters at t c t = cost function at t The cost function depends on the occurrence of seizures and stimulation delivered: c t = c t seizure +  c t stim wherec t seizure = {1 if seizure occurred at time t, 0 otherwise} c t stim = {1 if stimulation occurred at time t, 0 otherwise}  is a free parameter.

16. Adaptive Neurostimulation 16 Joelle Pineau Methods: Minimizing the cost function The objective is to select actions such as to minimize the expected cumulative cost: E [ c t + c t+1 + c t+2 + … + c T | z t ] Use regression analysis to estimate the cost for different action choices from the training data: Q k (z t, a t ) = c t + max a  A Q k-1 (z t+1, a) Select the action which minimizes the expected cost: a t := argmax a  A Q k (z t, a)

17. Adaptive Neurostimulation 17 Joelle Pineau Experimental protocol for validation 1.Control period (min. 3 seizures). 2.Periodic pacing at 1.0 Hz (min. 20 minutes). 3.Recovery period. 4.Adaptive stimulation strategy (min. 20 minutes). 5.Recovery period, no stimulation. 6.Periodic pacing at effective frequency f = n s /T wheren s =# stimulations during adaptive protocol T=duration of adaptive protocol

18. Adaptive Neurostimulation 18 Joelle Pineau Proportion of time spent in seizure Proportion of time spent in seizure, averaged over N=11 slices. * = statistically significant at p=0.05

19. Adaptive Neurostimulation 19 Joelle Pineau Effective frequency of the adaptive protocol

20. Adaptive Neurostimulation 20 Joelle Pineau Suppression efficacy for slices with eff > 1Hz * = statistically significant at p=0.1 * N=11N=4

21. Adaptive Neurostimulation 21 Joelle Pineau Adaptive protocol example #1 (a) Adaptive controller suppresses a seizure by increasing the frequency of stimulation.

22. Adaptive Neurostimulation 22 Joelle Pineau Adaptive protocol example #2 (a) Adaptive controller suppresses a seizure by increasing the frequency of stimulation. (b) A short seizure develops, stimulation is applied to shorten its duration.

23. Adaptive Neurostimulation 23 Joelle Pineau Adaptive protocol example #3 (a) Adaptive controller suppresses a seizure by increasing the frequency of stimulation. (b) A short seizure develops, stimulation is applied to shorten its duration. (c) Adaptive controller increases frequency to suppress seizure, then decreases frequency.

24. Adaptive Neurostimulation 24 Joelle Pineau In vivo: Challenges Limited slice-to-slice variation. Short lifespan. In vitro model has known periodic pacing strategy. Restricted parameter space. Larger variance between subjects. Longer lifetime; disease can evolve over time. No known open-loop strategies. Higher-dimensional action space (more electrodes, intensity settings, etc.) In vitro modelIn vivo model

25. Adaptive Neurostimulation 25 Joelle Pineau Discussion Animal models of epilepsy provide a rich framework for investigating adaptive neurostimulation strategies. Most adaptive neurostimulation approaches adopt a “detect-then-stimulate” paradigm. Our work leverages techniques from the control literature. –Goal is to directly minimize a cost function. –Explicit seizure prediction (or detection) is not required. Results show good suppression in vitro, in some cases using significantly less stimulation than periodic pacing. Preliminary evidence suggests that neurostimulation can be used to probe the excitability of the system.

26. Adaptive Neurostimulation 26 Joelle Pineau References J. Pineau, A. Guez, R. Vincent, G. Panuccio & M. Avoli, Treating epilepsy via adaptive neurostimulation: A reinforcement learning approach. Int. J. of Neural Systems. 19(4). 2009. M. Avoli, M. D’Antuono, J. Louvel, R. Kohling, G. Biagini, R. Pumain, G. D’Arcangelo, & V. Tancredi, Network and pharmacological mechanisms leading to epileptiform synchronization in the limbic systems in vitro. Prog. Neurobiol. 68(3). 2002. G. D’Arcangelo, G. Panuccio, B. Tancredi, M. Avoli. Repetitive low- frequency stimulation reduces epileptiform synchronization in limbic neuronal networks. Neurobiology of Disease. 19(1-2). 2005. A. Guez, R. Vincent, M. Avoli, J. Pineau. Adaptive treatment of epilepsy via batch-mode reinforcement learning. Innovative applications of Artificial Intelligence. 2008.

27. Adaptive Neurostimulation 27 Joelle Pineau Contrasting approaches to adaptive neurostimulation 1.Use observation window to detect seizures; when a seizure is detected, start neurostimulation. 2.Use observation window to characterize state of the system; apply neurostimulation as necessary to minimize seizure occurrence.

28. Adaptive Neurostimulation 28 Joelle Pineau Results: Proportion of decision windows with seizures

29. Adaptive Neurostimulation 29 Joelle Pineau Results: Proportion of decision windows with stimulation

30. Adaptive Neurostimulation 30 Joelle Pineau Adaptive neurostimulation deployed in vitro