1. UofR: July 27, 2001 A Flexible Brain-Computer Interface Jessica D. Bayliss Advisor : Prof. Dana Ballard Committee Members : Prof. Christopher Brown, Prof. Mary Hayhoe, and Prof. David Loiselle Chair : Prof. Edward Titlebaum

2. University of Rochester Computer Science Dept. What is a BCI? External BCI BCI - an interface which seeks to enable a user to communicate directly with a computer via the electrical signals from the brain Internal BCI

3. University of Rochester Computer Science Dept. Motivations l Locked in patients n Spinal Cord Injury n Amyotrophic Lateral Sclerosis – also known as ALS or Lou Gehrig’s Disease –5,000 newly diagnosed cases/yr. –300,000 living in the US now will eventually die from ALS

4. University of Rochester Computer Science Dept. Why not use Eye Movement Control? l Not all patients have eye movement control. l Since ALS is a degenerative disease, patients with eye movement control may lose it. l Even for those with eye movement control, a BCI presents a viable alternative/addition for control.

5. University of Rochester Computer Science Dept. Thesis Statements l A BCI should be designed for flexibility. l Virtual environments can be used in evoked potential experiments and the P3 component of the evoked potential in particular is robust among different environments. l Signal artifacts have a smaller effect on recognition of the P3 component of the evoked potential than commonly assumed. l Signal recognition ability only accounts for part of the performance and usability of a BCI system.

6. University of Rochester Computer Science Dept. Contributions l Creation of a flexible BCI with flexibility in signal recognition, user applications, and communication. l Experimental results showing the presence of evoked potentials in virtual environments and the robustness of the P3 component of the evoked potential in different environments. l Examination of the effects that common artifacts have on P3 recognition algorithms. l Experimental results showing that there is more to BCIs than just signal recognition.

7. University of Rochester Computer Science Dept. A BCI should be designed for flexibility? Why? l Existing BCI’s are monolithic or designed for speed rather than flexibility l Existing BCI’s are slow, error prone, and hardly useable.  These two attributes lead to systems that are problematic and hard to change!

8. University of Rochester Computer Science Dept. Thought Translation Device [Birbaumer, Nature vol. 399, 1999] l 75-85% correct recognition l alphabet is split into two halves and the subject picks one half until a single letter is picked l 2 characters chosen per minute l the first letter written this way took 16 hours!

9. University of Rochester Computer Science Dept. The First BCI [Vidal 1973] l “The main computing power is provided by the [UCLA] campus IBM 360/91, which is equipped with an exceptionally large core memory of 4 MBytes.”

10. University of Rochester Computer Science Dept. BCI for Cursor Control (Wolpaw et. al.)

11. University of Rochester Computer Science Dept. Sutter’s 1991 Brain Response Interface

12. University of Rochester Computer Science Dept. A Comparison of Several Applications

13. University of Rochester Computer Science Dept. System Architecture l Once the system runs fast enough, we would like it to be flexible l Fast enough means –Continuous EEG data acquisition –Analysis that presents results within a reasonable time frame to the user within the context of the task (fast enough for interactive use)

14. University of Rochester Computer Science Dept. Basic Requirements l Basic Needs n use the available hardware in the lab n flexibility in signal processing and user applications n the BCI exists on an inexpensive computer n use as much “off the shelf” software as possible

15. University of Rochester Computer Science Dept. BCI Design

16. University of Rochester Computer Science Dept. Implementation l BCI Backend mostly in Visual C++ l Lightweight threads for the GUI (for control setup), communications, EEG Acquisition l Separate processes for: Matlab engine, the BCI backend, the user app. l User app communicates via numerical codes over a serial port l Important variables that vary from experiment to experiment are set in a configuration file.

17. University of Rochester Computer Science Dept. Current Applications l Playing QBasic Pong and Java Tetris with eye movements l Driving in a virtual environment using the P3 EP l Controlling items in a virtual apartment with the P3 EP

18. University of Rochester Computer Science Dept. Thesis Statements l A BCI should be designed for flexibility. l Virtual environments can be used in evoked potential experiments and the P3 component of the evoked potential in particular is robust among different environments. l Signal artifacts have a smaller effect on recognition of the P3 component of the evoked potential than commonly assumed. l Signal recognition ability only accounts for part of the performance and usability of a BCI system.

19. University of Rochester Computer Science Dept. Why Virtual Environments? l Can control most aspects of the environment l Represents a complex environment and opens up new vistas for experimentation l Safe l Can be used for training l Motivational

20. University of Rochester Computer Science Dept. Evoked Potentials l changes in voltage potential that occur after a stimulus

21. University of Rochester Computer Science Dept. P3 [Chapman 1964, Sutton 1965] l a positive deflection in the EEG signal peaking approximately 300- 400ms after a task relevant stimulus

22. University of Rochester Computer Science Dept. Task Relevant Stimulus

23. University of Rochester Computer Science Dept. P3 Character Recognition l Farwell and Donchin in 1988 l You stare at a letter on the screen and count the number of times it flashes in order to pick the letter l [demo]

24. University of Rochester Computer Science Dept. Other P3 Application Results l 56% on-line and 90% off-line recognition (Spencer et. al. 1999) –Using a discrete stepwise linear analysis –Used an average of around 4 trials l 50% off-line (Polikoff et. al.1995) –Used single trials l 80% off-line (Jung et. al. 1999) –Only looked at true positives (P3’s), not true negatives (non-P3’s) –Used single trials

25. University of Rochester Computer Science Dept. Driving Experiment l Subjects were told to drive around the VR town and stop at red lights. l In addition, all red lights are preceded by yellow lights, so the red light should elicit a P3 and the yellow light should not elicit a P3.

26. University of Rochester Computer Science Dept. Results

27. University of Rochester Computer Science Dept. Results - alternate lights

28. University of Rochester Computer Science Dept. VR Driving (1998) l Can evoked potentials be reliably collected in a virtual environment? –Yes! l Is the signal clean enough to do single trial analysis? –Yes!: 84.5% recognition accuracy l 90% of non-P3’s recognized correctly l 65% of P3’s recognized correctly

29. University of Rochester Computer Science Dept. Environmental Control

30. University of Rochester Computer Science Dept. Main Experimental Question l Is there a difference in control while immersed in a virtual environment vs. just staring at a computer screen?

31. University of Rochester Computer Science Dept. The Application l 9 subjects in an experiment in a VR apartment living room l 5 possible actions: turn on/off 3 objects (light, stereo, tv) and say Hi or Bye to a graphics figure l All feedback is visual l On-line rec. and eye movement reduction l In order to obtain P3’s for the item that needs to be controlled, we have a clear button on each item that occasionally flashes red

32. University of Rochester Computer Science Dept. How the P3 is evoked

33. University of Rochester Computer Science Dept. The Experiment l Training: 5 minutes sitting in the room and counting the occurrences of the red button flash on the light l 5 minutes trying to obtain given goals with the following conditions given in a random order: –showing the apartment on a monitor –in the VR helmet with a fixed head view –fully immersed in the VR helmet

34. University of Rochester Computer Science Dept. Grand Averages for Different Environments

35. University of Rochester Computer Science Dept. Throughput (items/minute) in Different Environments

36. University of Rochester Computer Science Dept. Grand Averages over Time of Task

37. University of Rochester Computer Science Dept. Throughput (items/min) over Time of Task

38. University of Rochester Computer Science Dept. Thesis Statements l A BCI should be designed for flexibility. l Virtual environments can be used in evoked potential experiments and the P3 evoked potential in particular is robust among different environments. l Signal artifacts have a smaller effect on recognition of the P3 than commonly assumed. l Signal recognition ability only accounts for part of the performance and usability of a BCI system.

39. University of Rochester Computer Science Dept. Nonbiological Artifacts

40. University of Rochester Computer Science Dept. Biological Artifacts

41. University of Rochester Computer Science Dept. Artifact Experiment l 5 minutes of each of the following artifacts were recorded: breathing, chewing, foot movement, forehead movement, horizontal eye movement, jaw movement, talking, and vertical eye movement l The goal: Assume there are no P3 signals in the recordings and see how often various algorithms classify data as a P3.

42. University of Rochester Computer Science Dept. Example Artifact Recordings

43. University of Rochester Computer Science Dept. Recognition Algorithms Used l Peak picking: If the highest peak difference in the signal time window is above a threshold then the signal is a P3. l Correlation with the average P3 and non-P3 signal. The largest correlation determines classification. l Robust Kalman filter as a preprocessor: Statistically robust template matching

44. University of Rochester Computer Science Dept. Results of Artifact Experiment

45. University of Rochester Computer Science Dept. Thesis Statements l A BCI should be designed for flexibility. l Virtual environments can be used in evoked potential experiments and the P3 evoked potential in particular is robust among different environments. l Signal artifacts have a smaller effect on recognition of the P3 than commonly assumed. l Signal recognition ability only accounts for part of the performance and usability of a BCI system.

46. University of Rochester Computer Science Dept. Recognition: Is there always a P3?

47. University of Rochester Computer Science Dept. Pilot Experiment with Eye Tracking l The results indicated that subjects are not always paying attention to the control goal l [video]

48. University of Rochester Computer Science Dept. There are Tradeoffs between Performance and Recognition

49. University of Rochester Computer Science Dept. Two Dimensional Cursor Task (Polikoff et. al.) l If single trials yield 50% recognition n Subject would need 30 steps to reach a goal of 10 steps in the target direction (2 minutes if 4 seconds/step). l If an average of 3 trials yields 60% recognition n Subject would need 24 steps to reach the same goal (4.8 minutes at 12 seconds/step) l If an average of 8 successive trials yields 85% recognition n Subject would need 14 steps to reach the same goal (7.5 minutes at 32 seconds/step)

50. University of Rochester Computer Science Dept. Can the Interface Affect Throughput?

51. University of Rochester Computer Science Dept. Thesis Statements l A BCI should be designed for flexibility. l Virtual environments can be used in evoked potential experiments and the P3 evoked potential in particular is robust among different environments. l Signal artifacts have a smaller effect on recognition of the P3 component of the evoked potential than commonly assumed. l Signal recognition ability only accounts for part of the performance and usability of a BCI system.

52. University of Rochester Computer Science Dept. Future Work l Areas for system expansion: –Release system as open source –Allowing stand alone signal processing programs to run instead of just Matlab –Increase robustness l Recent off-line results indicate that different signal processing alg’s recognize different trials correctly. Can these alg’s be used together? l Different signals (even eye movements!) may be used together for the best system for a particular patient.

53. University of Rochester Computer Science Dept. The Different Algorithms Recognize Different Trials Correctly

54. University of Rochester Computer Science Dept. Contributions l Creation of a flexible BCI with flexibility in signal recognition, user applications, and communication. l Experimental results showing the presence of evoked potentials in virtual environments and the robustness of the P3 component of the evoked potential in different environments. l Examined the effects that common artifacts had on P3 recognition algorithms. l Experimental results showing that signal recognition is not necessarily the cause of problems in a BCI system.

55. University of Rochester Computer Science Dept. Visual Continuous Performance Task l Pictures are flashed on a computer screen in front of the subject. The subject is told to press a mouse button or count the occurrences of an infrequent picture.

56. University of Rochester Computer Science Dept. True Pos. vs. False Pos.

57. University of Rochester Computer Science Dept. In order to Increase Recognition with Correlation l Variable Averaging n If the trial correlates well with the P3 or non-P3 average, then use it as is n If the trial does not correlate well, wait until the next trial of the same stimulus type. n If the next trial correlates well, then use it as is. n If not, then do a weighted average with the correlations from the previous trial and use this for classification.