The Need for Assistants that Monitor Cognitive Abilities Bart Peintner, Bill Jarrold SRI International November 26, 2015November 26, 2015November 26, 2015 Interaction Challenges for Intelligent Assistants, AAAI Spring Symposium

Bart Peintner, Bill Jarrold 2 Main Idea Application 1 Application 2 Application 3 Interaction Data Cognitive Abilities Executive function, motor skills, visuospatial abilities, long- and short-term memory, verbal abilities, physical changes in the brain 

Bart Peintner, Bill Jarrold 3 Applications: Reducing Errors Air traffic control Aircraft control Military apps Changes in interaction data may reflect increased likelihood of error Changes in interaction data may reflect increased likelihood of error Time horizon: hours Time horizon: hours

Bart Peintner, Bill Jarrold 4 Apps: Detecting Cognitive Decline Internet Games Early detection is critical Early detection is critical Time horizon: days/months Time horizon: days/months In 2006, 32% of adults > 65 use the Internet 71% of adults 50-64

Bart Peintner, Bill Jarrold 5 Data shown to reflect cognitive change Mouse movements (Jimison et al, 2005) Mouse movements (Jimison et al, 2005) Game performance (Jimison et al, 2005) Game performance (Jimison et al, 2005) Time between keystrokes/variation (Jimison et al, 2004) Time between keystrokes/variation (Jimison et al, 2004) Linguistic content analysis Linguistic content analysis Verb use changes with damage to frontal lobe (Rhee et al, 2001) Verb use changes with damage to frontal lobe (Rhee et al, 2001) Noun and proper name use change with damage to temporal lobe (Ash et al, 2006) Noun and proper name use change with damage to temporal lobe (Ash et al, 2006) Two systems already developed (Gottschalk et al, 2002, Pennebaker et al, 2001) Two systems already developed (Gottschalk et al, 2002, Pennebaker et al, 2001) Speech Speech Speech-to-text for content analysis Speech-to-text for content analysis Prosodic features already used for speaker ID Prosodic features already used for speaker ID

Bart Peintner, Bill Jarrold 6 Challenges: How to Evaluate? No gold standard for cognitive abilities No gold standard for cognitive abilities The standard: neuropsychological tests The standard: neuropsychological tests Limitation: one-time snapshot Limitation: one-time snapshot User studies expensive User studies expensive Ideally in-home Ideally in-home For periods long enough for change to occur For periods long enough for change to occur

Bart Peintner, Bill Jarrold 7 Challenges: Creating Models Person-to-person variability Person-to-person variability Practice effects Practice effects People improve at games, mouse use, etc. People improve at games, mouse use, etc. Confounding factors Confounding factors Measures influenced by changes in sight, hearing, level of interest Measures influenced by changes in sight, hearing, level of interest

Bart Peintner, Bill Jarrold 8 Other Challenges When to act on perceived decline in cognitive abilities? When to act on perceived decline in cognitive abilities? How to collect data for research while retaining privacy and validity? How to collect data for research while retaining privacy and validity?