1. Carnegie Mellon 1

2. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN www.cs.cmu.edu/~listen Carnegie Mellon University 2 This work was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305A080157 and R305A080628 to Carnegie Mellon University. The opinions expressed are those of the authors and do not necessarily represent the views of the Institute or U.S. Department of Education.

3. Carnegie Mellon Motivation: peer into student’s mind Identify mental states –Cognitive: effort, recognition, understanding, learning, … –Affective: attention, engagement, frustration, … –Motor: speech, motion, expression, … Decide accordingly –What to teach next –How to teach it –…–… 3

4. Carnegie Mellon Peering Into Minds: EEG (electroencephalography) 4

5. Carnegie Mellon EEG in the Lab 5 Cons: Impractical for schools! Wet electrodes (gel or saline) Expert application and monitoring Money for equipment & personnel Magnetically-shielded room Pros: Controlled experimental conditions Temporally fine-grained data Reflects widespread brain activity Detects relevant mental states: attention (Marosi et al. ’02), engagement (Lutsyuk et al. ’06), frustration (Berka et al. ’06)

6. Carnegie Mellon New Portable EEG Devices 6 Pros: Feasible to use in schools Inexpensive Dry electrodes No expert needed Headphones and microphone Cons: 1-2 electrodes; not whole brain What can such devices detect?

7. Carnegie Mellon Detect brain states related to learning? 7 1.Can EEG detect when reading is difficult? 2.Can EEG detect differences between words? 3.Which EEG features detect differences best?

8. Carnegie Mellon Easy vs Hard (Grade K-1) (GRE level) 8 We need water, land, and air to live. Earth has all these things. Water covers much of Earth. Most of this water is not safe to drink. Many people are running out of fresh drinking water. In regard to propaganda the early advocates of universal literacy and a free press envisaged only two possibilities: the propaganda might be true, or it might be false. They did not foresee what in fact has happened… Carnegie Mellon 1. Detect when reading is difficult?

9. Carnegie Mellon 1. Detect when reading is difficult? For adults / children –10 adults in Project LISTEN lab –11 children ages 9-10, at school Reading connected text / isolated words Aloud / silently 9

10. Carnegie Mellon Child reader Adult reader Pilot study setup 10 Carnegie Mellon Project LISTEN’s Reading Tutor Reader

11. Carnegie Mellon Collecting EEG and Reading Tutor Data 11 Time Student Text 07-12-2011-09:51:00.10 Kai-min We … 07-12-2011-09:51:00.20 Kai-min need … 07-12-2011-09:51:00.35 Kai-min need … 07-12-2011-09:51:00.55 Kai-min water… 07-12-2011-09:51:01.01 Kai-min land … 07-12-2011-09:51:01.50 Kai-min air … Reading Tutor Log Time Student Raw Attention 07-12-2011-09:51:00.01 Kai-min -33 11 … 07-12-2011-09:51:00.02 Kai-min 351 17 … 07-12-2011-09:51:00.03 Kai-min 117 20… 07-12-2011-09:51:00.04 Kai-min 66 1 14 … 07-12-2011-09:51:00.05 Kai-min -451 26… 07-12-2011-09:51:00.06 Kai-min -3 43… MindSet (EEG) Log Time Student Text Raw 07-12-2011:09:51:00.10 Kai-min We -33 … 07-12-2011:09:51:00.20 Kai-min need 351 … 07-12-2011:09:51:00.35 Kai-min need 117 … 07-12-2011:09:51:00.55 Kai-min water 661 … 07-12-2011:09:51:00.10 Kai-min land -451 … 07-12-2011-09:51:01.50 Kai-min air 43,,, Combined Log

12. Carnegie Mellon MindSet (EEG) Features 12 Raw EEG signal, reported at 512 Hz Filtered EEG signal, 512 Hz Proprietary “attention” measure, 1 Hz Proprietary “meditation” measure, 1 Hz Proprietary signal quality measure, 1Hz Theta band (4-7 Hz), 8Hz Alpha band (8-11 Hz), 8Hz Beta band (12-29 Hz), 8Hz Gamma band (30-100 Hz), 8Hz Gamma+ band (101-256 Hz), 8Hz Delta band (1-3 Hz), 8Hz

13. Carnegie Mellon 13 Machine Learning Approach Train classifiers to detect mental states associated with stimuli –f = Binary Logistic Regression Classifier –X = MindSet (EEG) Features (averaged over stimulus interval) –Y = Easy or hard sentences N 1 1 F Y = f( X ) N 1

14. Carnegie Mellon 14 Train on: Test on: Train on: Test on: Trial 1 Trial 2 Trial 3 Trial 4 Trial 1 Trial 2 Trial 3 Trial 4 Trial 1 Trial 2 Trial 3 Trial 4 Trial 1 Trial 2 Trial 3 Trial 4 Reader-Specific Classifiers Reader-Independent Classifiers

15. Carnegie Mellon Class Size Imbalance 15 More easy sentences than hard ones Made the two sets equal in size using 3 approaches: 1.Random Oversampling (with replacement)

16. Carnegie Mellon Class Size Imbalance 16 More easy sentences than hard ones Made the two sets equal in size using 3 approaches: 1.Random Oversampling (with replacement) 2.Random Undersampling

17. Carnegie Mellon Class Size Imbalance 17 More easy sentences than hard ones Made the two sets equal in size using 3 approaches: 1.Random Oversampling (with replacement) 2.Random Undersampling 3.Directed Undersampling (Truncating)

18. Carnegie Mellon Detect when reading is difficult: Reader-specific classifier accuracy 18

19. Carnegie Mellon Detect when reading is difficult: Reader-independent classifier accuracy 19 chance p <.05

20. Carnegie Mellon 2. Detect differences between words? 20 Easy Words Bedroom Chicken Station Hard Words Cologne Chassis Brocade Non-Words KOF CUN WAF Illegal String FFS GHT NKL

21. Carnegie Mellon 21 Machine Learning Classifiers Multinomial logistic regression classifiers Measure classifier’s rank accuracy (Mitchell et al. ’04) –Use classifier to rank-order possible class labels –Rank accuracy = percentile rank of correct label; 0.5 = chance –More sensitive than % correct rank accuracy = 0.67 Classifier’s Ranking 1. Non-Word 2. Illegal String 3. Hard Word 4. Easy Word 1.0 0.5 0.0 True label 

22. Carnegie Mellon Detect differences between words: Reader-specific rank accuracy 22 chance p <.05

23. Carnegie Mellon Detect differences between words: Reader-independent rank accuracy 23 chance p <.05

24. Carnegie Mellon 3. Which features detect text difficulty best? Train classifier using each feature in isolation Average accuracy across subjects, CV folds –Higher = better

25. Carnegie Mellon 3. Which features detect word differences best? Train classifier using each feature in isolation Average rank accuracy across subjects, CV folds –Higher = better

26. Carnegie Mellon Which EEG features are sensitive to which lexical properties? 26 Delta (1-3 Hz) Theta (4-7 Hz) Alpha (8-11 Hz) Beta (12-29 Hz) Gamma (30-100 Hz) Gamma+ (101-256 Hz) Concreteness Imageability Colorado Meaningfulness-0.08-0.10 Familiarity-0.08 Age of acquisition Brown verbal frequency-0.12-0.13-0.09-0.12 Kucera and Francis written frequency0.070.10 Thorndike-Lorge frequency0.09 Number of letters0.110.130.10 0.140.16 Do EEG spectra reflect natural lexical variance among sentences? –If so, those bands may carry different information for a tutor. Correlate with MRC Psycholinguistic word properties (Coltheart 81) –Blank = not statistically significant; bold = passes False Discovery Rate test

27. Carnegie Mellon 4. Conclusions and Future Work 1-electrode EEG tells easy from hard better than chance. Frequency bands tap different properties a tutor may use. Detect mental states such as attention or frustration Use longitudinal EEG in schools to: –Instrument authentic behavior –Label data based on normal tutor use, not artificial experiments –Detect longer-term learning, not just recency effects –Combat EEG noise with “big data” –Inform tutor redesign and make student-specific models 27

28. Carnegie Mellon 28 Questions?

29. Carnegie Mellon 29

30. Carnegie Mellon 3. Which features detect word differences best? Train classifier using each feature in isolation Average rank accuracy across subjects, CV folds –Higher = better

31. Carnegie Mellon Which features predict word differences best?