Some Reflections on Augmented Cognition Eric Horvitz ISAT & Microsoft Research November 2000 Some Reflections on Augmented Cognition Eric Horvitz ISAT & Microsoft Research November 2000

Toward Augmented Cognition Challenge: Volume of information, overall complexity of defense tasks continue to grow at a rapid pace, in stark contrast to human cognitive abilities, which remain static: Attention Attention Memory Memory Learning Learning Comprehension Comprehension Sensory bandwidth Sensory bandwidth Visualization abilities Visualization abilities Judgment and decision making Judgment and decision making In contrast, computational capabilities have continued to grow rapidly Apply computation to support / augment human cognition

Developing new powers of understanding, remembering, and decision making via new technologies custom-tailored for human computer collaboration, symbiosis Augmented Cognition Methods, designs that harness computation and explicit knowledge about human limitations to open bottlenecks, address biases, deficits in cognition. Methods, designs that harness computation and explicit knowledge about human limitations to open bottlenecks, address biases, deficits in cognition. Techniques and interfaces that allow for fluid mixed- initiative interaction to solve problems. Techniques and interfaces that allow for fluid mixed- initiative interaction to solve problems. Continual background sensing, learning, inference to understand trends, patterns, situations relevant to a users context and goals. Continual background sensing, learning, inference to understand trends, patterns, situations relevant to a users context and goals.

Modern Psychology: Cognition as information processing Human cognition: Vast abilities coupled with highly characterized / characterizable limitations & bottlenecks Human cognition: Vast abilities coupled with highly characterized / characterizable limitations & bottlenecks Mind as an architecture composed of different subsystems that form, remember, transmit representations of the world. Mind as an architecture composed of different subsystems that form, remember, transmit representations of the world. Work on distinct dimensions of cognition: Attention, memory, learning, concept attainment, visualization, judgment Work on distinct dimensions of cognition: Attention, memory, learning, concept attainment, visualization, judgment

Results in Cognitive Psychology Limitations in serial processing Limitations in serial processing Attention Attention Concept attainment Concept attainment Learning and memory Learning and memory Visualization Visualization Judgment and decisions Judgment and decisions Collaboration, group decisions Collaboration, group decisions Psychophysics Psychophysics HCI expertise, results HCI expertise, results Understanding of human as resource-limited reasoner, well-characterized bottlenecks, capabilities, biases Timing: Is Now the Time to Undertake Such an Effort?

Computational power Computational power Raw computational resources (CPU, memory) Raw computational resources (CPU, memory) Connectivity Connectivity Sensing Sensing Sensor fusion Sensor fusion Inference and classification Inference and classification Learning Learning User modeling User modeling Speech recognition Speech recognition Vision Vision Interaction, input Interaction, input UI, display design UI, display design Leaps in computation, memory, algorithmic prowess

Augmented Cognition Efforts ExistingPsychological Results on Cognitive Limitations Target Cognitive Tasks Memory ConceptAttainment DividedAttention Computation Abilities & efficiencies Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive limitations Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive limitations Identify, apply results from cognitive psychology to build computation solutions Identify, apply results from cognitive psychology to build computation solutions Identify new cognitive psychology research Identify new cognitive psychology research Identify new joint CScognitive psychology efforts Identify new joint CScognitive psychology efforts

ExistingPsychological Results on Cognitive Limitations Target Cognitive Tasks Memory ConceptAttainment DividedAttention Abilities & efficiencies Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive limitations Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive limitations Identify, apply results from cognitive psychology to build computation solutions Identify, apply results from cognitive psychology to build computation solutions Identify new cognitive psychology research Identify new cognitive psychology research Identify new joint CScognitive psychology efforts Identify new joint CScognitive psychology efforts * New Cog. Psych. Research * New Cog. Psych. Research * HCI, Aug. Cognition Research * HCI, Aug. Cognition Research Augmented Cognition Efforts ?

Enhance learning and memory via reminder systems & methods Enhance learning and memory via reminder systems & methods Automate specific aspects of problem solving & filtering Automate specific aspects of problem solving & filtering Modulate / triage communications Modulate / triage communications Develop new visualizations, other info. rendering to increase rate of concept attainment---raising effective human-computer bandwidth Develop new visualizations, other info. rendering to increase rate of concept attainment---raising effective human-computer bandwidth Augmented Cognition: Opportunities

Support sensory fusion, judgment, action under uncertainty Support sensory fusion, judgment, action under uncertainty Provide vigilant monitoring for situations requiring attention Provide vigilant monitoring for situations requiring attention Monitoring, knowledge about human and machine errors Monitoring, knowledge about human and machine errors Extend abilities to coordinate, collaborate with other people and systems Extend abilities to coordinate, collaborate with other people and systems Design multimodal, context-aware systems for more efficient, natural humancomputer collaboration Design multimodal, context-aware systems for more efficient, natural humancomputer collaboration

Low-Hanging(?) Fruit Information filtering and triage Information filtering and triage Mixed-initiative interaction Mixed-initiative interaction Context-sensitive UI / computing Context-sensitive UI / computing Intelligent reminding Intelligent reminding Managing attention and disruption Managing attention and disruption Visualization & comprehension Visualization & comprehension Human-error--aware systems Human-error--aware systems Automated sensor fusion Automated sensor fusion Judgment de-biasing systems Judgment de-biasing systems

Risks in Building a Strong, Valuable Program Timing: Too early…? Timing: Too early…? Need more complete understanding of human bottlenecks in real world? Need more complete understanding of human bottlenecks in real world? Need more sophisticated computational procedures? Need more sophisticated computational procedures? Strength of research Strength of research Rigorous, foundational work may be needed Rigorous, foundational work may be needed Poor initial stabs may lead to premature discontinuation, disappointment Poor initial stabs may lead to premature discontinuation, disappointment Focus Focus Choosing some foci for early traction Choosing some foci for early traction Avoiding replication of work in commercial sector Avoiding replication of work in commercial sector

Several Sample Projects Vista: Models for limiting, controlling information based on coarse value of revealed information. Vista: Models for limiting, controlling information based on coarse value of revealed information. Handsfree decision support: Fluid interaction for providing recommendation, updates; context-sensitive speech. Handsfree decision support: Fluid interaction for providing recommendation, updates; context-sensitive speech. Lumiere: Context- and competence-based assistance Lumiere: Context- and competence-based assistance Lookout: Mixed-initiative interaction. Lookout: Mixed-initiative interaction. Priorities: Automated assignment of urgency, alerting, communication, caching by value. Priorities: Automated assignment of urgency, alerting, communication, caching by value.

Several Sample Projects Notification Platform: Attention in HCI, notification architectures, UI, psych studies of disruption, reminders. Notification Platform: Attention in HCI, notification architectures, UI, psych studies of disruption, reminders. Continual computation: Use background computation to forecast goals, attention; guide precomputing, caching. Background query, document tracking. Continual computation: Use background computation to forecast goals, attention; guide precomputing, caching. Background query, document tracking. Learning information goals: Learning about a users goals from behavior, log files. Learning information goals: Learning about a users goals from behavior, log files. Qualia: Models of attention, perception in multimedia rendering and communication Qualia: Models of attention, perception in multimedia rendering and communication Context-aware devices: Enhance UI with implicit gestures, context-sensing. Context-aware devices: Enhance UI with implicit gestures, context-sensing.

Project Drill Down & Demos