Denis Gouin Valérie Lavigne Alexandre Bergeron-Guyard Innovative Interfaces and Interactions Group Intelligence and Information Section DRDC Valcartier KSCO 2012 February 2012 Human-Computer Interaction with an Intelligence Virtual Analyst

1 iVAC initiative Intelligent Software Assistant (ISA) identified by MIT’s Technology Review as one of 2009’s most promising emerging technologies Conversational, computer-generated characters capable of providing guidance to a user in the conduct of his tasks Intelligence Virtual Analyst Capability (iVAC) R&D initiative at Defence R&D Canada 3-year initiative started in 2011 Knowledge System with important HCI component

2 Virtual Assistant / Virtual Advisor IBM's Watson supercomputerDARPA’s CALO Cognitive Assistant that Learns and Organizes SIRI Apps on IPhone 4s

3 DARPA’s Personal Assistant that Learns (PAL) Video Clip presentation

4 Cognitive Assistant that Learns and Organizes (CALO)

5 HCI Requirements - General HCI dialogue between iVAC and the analyst(s) –How to input knowledge into the system –How to raise questions –How to task the IVA –How does the IVA ask for precisions on the questions and the way of presenting information Optimization of the presentation of the results One and/or multiple virtual analysts can serve multiple users Virtual analysts can interact with each other within and across meeting spaces

6 Enabling HCI Technologies Smart Room Environments Multimodal Interaction Information Presentation and Adaptive Interfaces Augmented Cognition ISA Representation / Avatars Storytelling

7 Smart Room Environments Instrumented environments (e.g. cameras, microphones, biometry) Ubiquitous computing. The user can access the system from various locations Track users location Track users activities US Navy Command Center of the Future (CCoF) DSTO LiveSpaces

8 Multimodal Interaction Allow the user(s) to employ multimodal interaction (e.g. voice, pointing, gesture, eye/gaze, neural/brain interfaces, emotion detection) Resolve ambiguities –These are my priorities –We should move that milestone back 6 months Xbox Kinect AFRL Interactive Data Wall Emotion Detection, Valenti et al. 2007

9 Information Presentation In support of various tasks –Present tools and information –Answer questions –Ask for clarification –Remind the user of some tasks or procedures –Propose alternative possibilities –Give a briefing

10 Information Presentation Means –Voice output and/or information display –Highlight information elements –Display information in a new window, organized as the user wants to see it –Gather a set of documents –Filter information based on user requests –Capture information about a meeting (automatic transcription and semantic analysis of multiparty meetings )

11 Adaptive Interfaces Customized to the role and current tasks of the user Present tools and information based on the user preferences Suggest sequences of events based on learned tasks and preferences Understand the context / tasks Observe what the user is doing and his mental state Do not disturb The user is recognized using biometry The interface adapts to the distance of the user to the (large) display (text fonts and granularity of information)

12 Augmented Cognition Track the sensory and cognitive overload of the users Use of neural/brain interfaces, emotion detection Employ computational strategies to restore operational effectiveness –Intelligent interruption to improve limited working memory –Attention management to improve focus during complex tasks –Cued memory retrieval to improve situational awareness and context recovery –Modality switching (i.e., audio, visual) to increase information throughput –When not to disturb DARPA (2011a), DARPA Defense Science Office, Improving Warfighter Information Intake Under Stress (AugCog),

13 ISA Representation / Avatars Various representations –Speech output (voice) –A simple icon –A two- or three-dimensional representation of a character (avatar) PAL

14 ISA Representation / Avatars Adaptation of the avatar to the task and socio-cultural context –Facial and voice features of the avatar (serious / smiling, tone of the voice ) should communicate emotions, danger, risk, uncertainty –Consider socio-cultural factors (Gender / age / ethnicity / profession) –Use different avatars to support different ISA tasks (weather prediction or course of action) Embedding avatars in a video conferencing Be careful of the Spectre of the Uncanny Valley Need for experiences with avatar representations DSTO’s FOCAL

15 Storytelling Guide audience through a preconstructed narrative Use of tangible graphics and videos (TV news like) Richness of the storytelling approach Convey situation awareness about complex events Logo – VisWeek -Telling Stories with Data

16 ISA for Coalition Operations Improved interoperability between coalition forces in terms of disseminating information Translate information between languages Share differences in tactics, techniques and procedures (TTPs) Synchronizing coalition activities Improve cognitive assistance by providing shared coalition awareness and task support Enable coalition organizations to learn by managing a knowledge base Provide better and faster decision making through access to comprehensive knowledge developed through time

17 Conclusion ISA, a knowledge system component Solve the human cognitive overload Conducting a wide variety of tasks –Search and organize information –Learn procedures and preferences –Manage schedules –Track people HCI is Key –Multimodal interaction –Avatar is not mandatory –Assign tasks –Summarize documents –Mediate interactions –Guide and remind the user

18