1. 1 High Transfer Rate, Real-time Brain-Computer Interface Machine-based learning techniques towards a practical spelling device for the completely paralyzed

2. April, 2005ThinQ Innovation2 Agenda Brain Computer Interfaces – brief intro. Our system –Overview, technical details –Machine learning – Support Vector Machines –Additional Bandwidth – Word Prediction –Results Future Improvements, Q&A Demonstration at Psychology Lab

3. April, 2005ThinQ Innovation3 BCIs – the Need ‘Locked-in’ patients Example: J.D. Bauby, “The Diving Bell and the Butterfly” Persistence of life –“butterfly” Extreme physical disability – “diving bell”

4. April, 2005ThinQ Innovation4 BCIs – the Need Amyotrophic Lateral Sclerosis (ALS), aka Lou Gherig’s –Degeneration of motor neurons, paralysis of voluntary muscles –120,000 diagnosed each year worldwide –2000 Canadians live with ALS right now –Can leave patients ‘locked-in’ –Cognitive and sensory functions remain intact

5. April, 2005ThinQ Innovation5 BCI(1): Slow Cortical Potentials (SCPs) Extensive training ~ 3 months using biofeedback mechanism Tested on ALS patients, learned to control SCPs Ref: N. Birbaumer et al., “The thought translation device (TTD) for completely paralyzed patients,” IEEE Trans. Rehab. Eng., Vol. 8, pp. 190-193,June 2000.

6. April, 2005ThinQ Innovation6 BCI(1): SCPs cont. Most successful subject – artificially fed and respirated for 4 years After 3 months of training, wrote letter below -Took 16 hours to write ~ 2 letters/minute -Expresses thanks, wants to have a party

7. April, 2005ThinQ Innovation7 BCI(2): Implants - Cyberkinetics Inc. BrainGate Neural Interface System: Mkt. cap ~$45mil. Control of cursor on PC using implant in motor cortex Undergoing limited clinical trials Limb movement possibilities

8. April, 2005ThinQ Innovation8 P300 Spelling Device – the P300 Event Related Potential Known as ‘oddball’ or ‘surprise’ paradigm Inherent 300ms 8-40 uV avg. deflection

9. April, 2005ThinQ Innovation9 P300 Spelling Device – the System Non-invasive Inherent Response

10. April, 2005ThinQ Innovation10 P300 Speller Terminology Epoch = One flash of any row or column Trial = 1 complete set of epochs - all rows and columns Symbol = Alphanumeric characters or pictures

11. April, 2005ThinQ Innovation11 BCI Competition 2003 Provided pre-collected data for competition P300 Spelling Paradigm: -Winners included Kaper et al. -Used Support Vector Machines -Achieved high transfer rate with real-time implementation possibilities

12. April, 2005ThinQ Innovation12 System Operation Steps –Training (approximate 1hr) Provide visual stimuli (flashing of rows/columns) Record data with known classification label Run data through pattern recognition algorithm (SVM) Create customized models for each individual –Spelling Load customized model for individual Provide visual stimuli (flashing of rows/columns) Record data with unknown classification label Run data through SVM classifier Sum up decision values Feedback most probable letter

13. April, 2005ThinQ Innovation13 Display Flexible matrix size Flexible matrix contents –Alphanumeric Characters –Words –Symbols

14. April, 2005ThinQ Innovation14 Display cont… Random and exhaustive flashing of all of the rows and columns on display Flashing cycle: 300ms –100ms intensification period –200ms de-intensification period 10 second rest period at the end of each symbol

15. April, 2005ThinQ Innovation15 Data Collection Collect data from DAQ sampled at 240Hz 600ms after intensification Buffer overlap Flexible data collection delay Flexible data recording time

16. April, 2005ThinQ Innovation16 Data Collection – cont. 10 channels collected simultaneously Data from each channel concatenated together Data stored into program memory Collected until end of a symbol –Converted to array –Memory cleared for next symbol System is timing critical

17. April, 2005ThinQ Innovation17 Timing Issue Purpose –Process within 300ms window Bottleneck –Online SVM processing Old design = 340ms/Epoch New design = 17.67ms/Epoch Requirement –Pentium4 or equivalent is sufficient

18. April, 2005ThinQ Innovation18 Matlab Interface Why we use Matlab? VB–Matlab interface using APIs Common functions –Pass matrix array to Matlab workspace –Get matrix array from Matlab workspace –Execute command line or script

19. April, 2005ThinQ Innovation19 Support Vector Machines Pattern recognition Algorithm SVM used for: –Creating models for different individuals (train) –Getting discriminant scores (spelling) Detailed information covered later

20. April, 2005ThinQ Innovation20 Score Matrix

21. April, 2005ThinQ Innovation21 Word Prediction Idea: predict intended words based on previous spelling. Similar to cellular phone ‘smart text’ Extract top ranked words –SQL for fast searching –Dynamic database Selection updated on the bottom of the display Words chosen same way

22. April, 2005ThinQ Innovation22 System Design Modular Design Approach

23. April, 2005ThinQ Innovation23 What is SVM? Developed by Vapnik in 1992 at Bell Labs Broad applications Based on concept of ‘learn from examples’ Key concepts: –Linear Decision Boundary with Margin –Nonlinear feature transformation

24. April, 2005ThinQ Innovation24 Basic Concept {x 1,..., x n } be our training data set y i  {1,-1} be the class label of x i then, Find a decision boundary Make a decision on disjoint test data

25. April, 2005ThinQ Innovation25 Decision Boundary (linear) Infinite possibility Class -1 Class 1

26. April, 2005ThinQ Innovation26 Bad Decision Boundary Class -1 Class 1 Class -1 Class 1

27. April, 2005ThinQ Innovation27 Good Decision Boundary Class -1 Class 1 m Want to maximize m Boundary found using constrained optimization problem

28. April, 2005ThinQ Innovation28 Optimization Problem

29. April, 2005ThinQ Innovation29 After Training x i ’s on the decision boundary are called SUPPORT VECTORS Support vectors and b defines the decision boundary

30. April, 2005ThinQ Innovation30 Geometrical Interpretation  6 =1.4 Class -1 Class 1  1 =0.8  2 =0  3 =0  4 =0  5 =0  7 =0  8 =0.6  9 =0  10 =0

31. April, 2005ThinQ Innovation31 Non-separable Samples Use of Soft Margin Separation Kernel Transformation

32. April, 2005ThinQ Innovation32 Soft Margin Separation Class -1 Class 1

33. April, 2005ThinQ Innovation33 Soft Margin Separation Idea: simultaneous maximization of margin and minimization of training error

34. April, 2005ThinQ Innovation34 Nonlinear Samples Some Samples are inherently nonlinear in input space No linear boundary is sufficiently accurate

35. April, 2005ThinQ Innovation35 Solution?

36. April, 2005ThinQ Innovation36 Kernel Transformation Idea: map input space into feature space such that samples become linearly separable

37. April, 2005ThinQ Innovation37 Gaussian Kernel

38. April, 2005ThinQ Innovation38 SVM Implementation Matlab interface to libsvm Kernel: RBF with  = 6.6799e-4 C parameter: 20.007

39. April, 2005ThinQ Innovation39 SVM Implementation Average Method (61.538%) Multi-Model Method (65.22%) Concatenation Method (82.418%) Weighted Concatenation Method (max. 86.264%)

40. April, 2005ThinQ Innovation40 Possible Improvements Weighted concatenation method Customized Kernel Parameters

41. April, 2005ThinQ Innovation41 Measure of Performance Bit Rate –N: number of available symbols –p: prediction accuracy –t: number of seconds taken to choose one symbol Letters per minute

42. April, 2005ThinQ Innovation42 Cont… Resulting Transfer Rates –Without using dictionary –With using dictionary

43. April, 2005ThinQ Innovation43 More Accurate Measure Resulting Transfer Rates –Without using dictionary –With using dictionary

44. April, 2005ThinQ Innovation44 Cont… Mechanism –Receives a chosen letter from control module –Appends the letter to current letters in the word –Searches SQL database –Return list of most probable target words based on ranking

45. April, 2005ThinQ Innovation45 Result Analysis 1.Accuracy across subjects 2.Accuracy over time, same subject 3.Accuracy over number of trials 4.Accuracy versus model size

46. April, 2005ThinQ Innovation46 Accuracy Across Subjects DateSubject# of TrialsAccuracy (letters) Percentage March 26Jack1512/1486% March 28Min1512/2157% March 30Brian1519/19100% March 31 st Jyh-Liang1526/26100% April 2Lucky1525/2889%

47. April, 2005ThinQ Innovation47 Accuracy Across Subjects DateSubject# of TrialsAccuracy (letters) Percentage March 31 st Jyh-Liang313/13100% March 31 st Jyh-Liang218/2090% March 31 st Jyh-Liang110/2148%

48. April, 2005ThinQ Innovation48 Accuracy Over Time, Same Subject Subject: Jack DateTime Change from Model Made # of TrialsAccuracy (letters) Percentage March 2601512/1486% March 282 days1519/19100% March 315 days1519/2286%

49. April, 2005ThinQ Innovation49 Accuracy Over Number of Trials Subject: Jyh-Liang

50. April, 2005ThinQ Innovation50 Accuracy Versus Model Size Subject: Jyh-Liang

51. April, 2005ThinQ Innovation51 Questions?