Affective Computing and Intelligent Interaction Ma. Mercedes T. Rodrigo Ateneo Laboratory for the Learning Sciences Department of Information Systems and Computer Science Ateneo de Manila Univeristy

Page 2 Affective computing Computing that relates to, arises from or deliberately influences emotion - Picard, 1997, Affective Computing

Page 3 Affective computing  Emotion recognition  Emotion expression  Intelligent response to emotion

Page 4 Significance: Towards more humane interfaces How can we enable computers to better serve people’s needs--adapting to you, vs. treating you like some fictionalized ideal user, and recognizing that humans are powerfully influenced by emotion, even when they are not showing any emotion? - Picard, 2003

Page 5 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

Page 6 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

Page 7 Methods: Aplusix

Page 8 Methods: Ecolab / MEcolab

Page 9 Methods: Cognitive Tutor

Page 10 Methods: BlueJ

Page 11 Methods: The Incredible Machine

Page 12 Methods: Math Blaster

Page 13 SQL/EER Tutors

Page 14 SimStudent

Page 15 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

Page 16 Biometrics instruments

Page 17 Biometrics instruments

Page 18 Biometrics instruments

Page 19 Sample Log: Brainfingers [Header V2035] 7/31/2009 7:49:05 PM UserFile = C:\Nia Data\\__ usr [Data] Sample,Event,GlanceMagJs,GlanceDirJs,A1Js,A2Js,A3Js,B1Js,B2Js,B3Js,MuscleJs 1,0,0.1,0.065,0.0195, , E-07,0.5815,0.5311,0.6048,0.6782,0.7665,0.7074,0 1,0,-0.029,-0.122,0.021,0.2056, E-07,0.5774,0.5251,0.5892,0.6723,0.7598,0.7125,0 1,0,0.015,-0.167,0.0205, , E-07,0.5743,0.5187,0.5737,0.6683,0.7534,0.7157,0 1,0, , ,0.0205, , E-07,0.573,0.5118,0.5596,0.6656,0.7468,0.7159,0 1,0,-0.285,-0.163,0.0205, , E-07,0.5733,0.5043,0.5477,0.6622,0.7398,0.7168,0 1,0, ,-0.206,0.022, , E-07,0.5745,0.4966,0.5371,0.6567,0.7321,0.7221,0 1,0,-0.125,-0.158,0.0225, , E-07,0.5772,0.4885,0.5268,0.6483,0.7242,0.7262,0

Page 20 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

Page 21 Sample Log: Aplusix %;ACTIONS;#Date=1/16/2007#Heure=14:57:59;#TypeProbleme=TpbDevelopper 0;0.0;structure;();0;();();();();();(); 2 derriere);rien;;N1; 2 derriere);rien;V1;N1; 6;1.5;BackSpace;();1;?;();(dedans);rien;V-;N-; 7;2.0;-;();1;-?;();(0 dedans);rien;V-;N-; 8;3.7;4;();1;-4;();(0 0 derriere);rien;V0;S0; 9;0.2;9;();1;-49;();(0 1 derriere);rien;V0;S0; 10;2.7;x;();1;-49x;();(0 1 derriere);rien;V0;S0; 1 1 dedans);rien;V-;N-;

Page 22 Sample Log: Ecolab New Activity Toolbar ButtonClick0 6 Activity 16 Activity Chosen: Food 46 Suggested Help 06 Suggested Challenge 16 Challenge Accepted18 ViewWeb change13 ViewWeb change14 ActionShow19

Page 23 Sample Log: Scatterplot Tutor *000:03:781 READY. *000:59:503 APPLY-ACTION WINDOW; ALGEBRA-2-TRANSLATOR::VARIABLE-TYPE-MODEL, CONTEXT; SPLOT-DB-C , SELECTIONS; (|var-1val-1|), ACTION; SUBSTITUTE-TEXT-INTO-BLANK, INPUT; ("Numerical"),. *000:59:503 UPDATE-P-KNOW META; META-VALUING-NUM-FEATURES, PRODUCTION; (CHOOSE-VAR-TYPE-NUM MIDSCH-VARIABLE-TYPING), SUCCESS?; T, P-KNOW; ,..

Page 24 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

Page 25 consolidation analysis Methods

Page 26 Methods

Page 27 Methods

Page 28 analysis Methods

Page 29 analysis consolidation analysis Methods

Affective states

Page 31 Boredom

Page 32 Confusion

Page 33 Delight

Page 34 Frustration

Page 35 Engaged concentration

Page 36 Others  Neutrality  Surprise

Behaviors

Page 38 On-task, solitary

Page 39 On-task, giving and receiving answers

Page 40 Other on-task conversation (probably)

Page 41 Off-task solitary

Page 42 Others  Off-task, conversation  Inactive  Gaming the system

Page 43 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

Page 44 Talk to me %;ACTIONS;#Date=1/16/2007#Heure=14:57:59;#TypeProbleme=TpbDevelopper 0;0.0;structure;();0;();();();();();(); 2 derriere);rien;;N1; 2 derriere);rien;V1;N1; 6;1.5;BackSpace;();1;?;();(dedans);rien;V-;N-; 7;2.0;-;();1;-?;();(0 dedans);rien;V-;N-; 8;3.7;4;();1;-4;();(0 0 derriere);rien;V0;S0; 9;0.2;9;();1;-49;();(0 1 derriere);rien;V0;S0; 10;2.7;x;();1;-49x;();(0 1 derriere);rien;V0;S0; 1 1 dedans);rien;V-;N-;

The problem: 5(-9x+7)+3(4x-3)

Step #Text typed by student 21-45x 22-45x=? 23-45x 24-45x+? 25-45x x x+35+? 28-45x x x+35+12x 31-45x+35+12x-? 32-45x+35+12x-9

The problem: 8x 2 -2x+6-(-5x 2 +8x+3)

Page 48 Step #Text typed by student 563c 564ch 565chr 566chri 567chris 568christ 569christi 570christin 571christine 572christine+? 573christine+c 574christine+cy 575christine+cyr

Step #Text typed by student 576christine+cyri 577christine+cyril 578christine+cyril=? 579christine+cyril=a 580christine+cyril=ab 581christine+cyril=abi 582christine+cyril=abig 583christine+cyril=abiga 584christine+cyril=abigai 585christine+cyril=abigail

Page 50 I’m not a math genius but I’m pretty sure that 8x 2 -2x+6-(-5x 2 +8x+3) != christine+cyril=abigail

Page 51 Analysis methods  Clean the data  Define the different features  Distill new features  Define desired range of values  Select an appropriate statistical test or data mining algorithm  Validate the findings

Page 52 Analysis techniques  Statistical methods  Data mining techniques

Page 53 Persistence of affective states  Regardless of software, boredom tends to persist  Affect and behavior detection

Page 54 Affect and behavior  Students who attempt the most difficult problems experience flow the most  Students who try the lowest levels experience more boredom and confusion.  Students who take the longest time in solving the problems experience confusion the most  Students who take the shortest time experience confusion the least.  Students who use the most number of steps to solve a problem experience confusion and boredom the most.  Students who take the least number of steps experience more flow.

Page 55 Carelessness  Creation of models that generalize across different datasets and countries  Students who are bored and confused are less likely to be careless  Students who are engaged are more likely to be careless

Page 56 Novice programmer behaviors  Students who are persistently confused do worse on the midterm  Students who are able to resolve their confusion do better  Students with low error quotients do better on the midterm

Page 57 Aplusix, Scatterplot, Ecolab, BlueJ, etc. Student Interaction Logs Observation Logs Analysis Affect detectors Behavior detectors Novice programmer errors Interventions Intelligent agents Improved error messages Sensor data

Page 58 Example

Page 59 New version

Page 60 Affect-sensitive game

Page 61 BlueJ Browser

Page 62 BlueJ Browser

Page 63 EDM Workbench

Page 64 Closing the loop  We continue to create new models and integrate the models with software  We still have a long way to go

My thanks  Ryan Baker  Jessica Sugay  Thor Collin Andallaza  Rina Joy Jimenez  Jason King Li  Tricia Monsod  Diane Lee  Minmin Lagud  Sweet San Pedro  Thomas Dy  And all the “cast of thousands” who constitute the ALLS

Thank you! Questions?