Multimodal FooBilliard adding two multimodal user interfaces to an existing 3d billiard game Dominic Sina, Paul Frischknecht, Marian Briceag, Ulzhan Kakenova Spring 2015 Final presentation Paul

Outline Introduction The Game The Interfaces CASE CARE Model Research Questions Users Evaluation Results Demo Paul

Introduction: Objectives add two multimodal user-interfaces to an existing game pose and attempt to answer research questions related to these UIs by running a user-evaluation Paul

Introduction: The Game Open source, C & OpenGL, FooBilliard the Windows Platform’s Speech Recognition API in a C# component and the Windows Touch APIs (+custom swipe detection) in a C++ component Dominic

Introduction: The Interfaces Keyboard, Mouse, Voice Commands Interface 2: Touch, Voice Commands Dominic

CASE Model Concurrent - Using independent modalities in parallel for distinct tasks. The user can at the same time use voice to increase/decrease the power of the strike, and use touch gestures to change to zoom in/out; the same could be done by using the keyboard and mouse. Alternate - Using combined modalities in sequential order for one task. You can position the ball by dragging it and then say “Put Here” to confirm your placement. Synergistic - Using combined modalities in parallel for the same task. The ball will be placed at the fingers’ position when you hold your finger on the screen and say “Put Here”. Exclusive - Using independent modalities in sequential way for distinct tasks. Each shot can be considered a distinct task. You can always choose whether to use voice or keyboard/touch command to shoot. Concurrent - Using independent modalities in parallel for distinct tasks. The user can at the same time use voice to increase/decrease the power of the strike, and use touch gestures to change to zoom in/out; the same could be done by using the keyboard and mouse. Alternate - Using combined modalities in sequential order for one task. You can position the ball by dragging it and then say “Put Here” to confirm your placement. Synergistic - Using combined modalities in parallel for the same task. The ball will be placed at the fingers’ position when you hold your finger on the screen and say “Put Here”. Exclusive - Using independent modalities in sequential way for distinct tasks. Each shot can be considered a distinct task. You can always choose whether to use voice or keyboard/touch command to shoot.

CARE Model Complementarity - Multiple modalities are to be used within a temporal window to reach a given state. User touches the screen and holds the finger at a position and uses the voice command “Put Here”. The ball moves to the position where finger is pointing. Assignment - Only one modality can be used to reach a given state. The list of all voice commands is only accessible through the voice commands “commands/help/what can I say” Redundancy - Multiple modalities have the same expressive power and they are all used within the same temporal window - User can shoot by using voice and keyboard (only one shot is performed, the rest of the input is ignored). Equivalence - Necessary and sufficient to use any one of the available modalities. User can shoot the ball using gestures or voice or keyboard. Complementarity - Multiple modalities are to be used within a temporal window to reach a given state. User touches the screen and holds the finger at a position and uses the voice command “Put Here”. The ball moves to the position where finger is pointing. Assignment - Only one modality can be used to reach a given state. The list of all voice commands is only accessible through the voice commands “commands/help/what can I say”. Redundancy - Multiple modalities have the same expressive power and they are all used within the same temporal window - User can shoot by using voice and keyboard (only one shot is performed, the rest of the input is ignored). Equivalence - Necessary and sufficient to use any one of the available modalities. User can shoot the ball using gestures or voice or keyboard.

The Research Question 1. Which interface do users learn better without instructions? 2. The usage of which interface gives better results in playing the game? 3. Which interface is more attractive/satisfying for users?

Evaluation with users Objectives: Evaluations: Quantitative: to observe valuable problems to test theoretical questions Evaluations: Quantitative: ○ After how much time was a certain command from a set of commands used for the first time and how often was this command used?* ○ How much score did the user make (how many balls had disappeared from the table)? ○ Satisfaction score Qualitative: ○ Comments and answers from the user to the question “What would you improve?” ○ Videos of the users

Evaluation: quantitative Interface 1: Keyboard, Mouse, Voice Commands Interface 2: Touch, Voice Commands Score (the number of balls) Satisfaction (from 1 to 10) User 1 8 User 2 6 User 3 7 User 4 3 5 User 5 User 6* 12 10 Score (the number of balls) Satisfaction (from 1 to 10) User 1 8 User 2 5 4 User 3 2 7 User 4 9 User 5 11 User 6* 12 10

Evaluation: discovered commands Interface 1: Keyboard, Mouse, Voice Commands Command User 1 User 2 User 3 User 4 User 5 Shoot (Voice) 40s, 4x 13s, 8x 25s, 1x 80s, 4x 90s, 1x Shoot (Keyboard) 80s, 3x 34s, 7x 210s, 32s, 5x 20s, 56x Bird view (Voice) 80s, 10x 59s, 2x 220s, 1x Bird view (Keyboard) Stronger/Weaker (Mouse) 65s, 3x 6s, 4x Commands (Voice) 60s,1x 55s, 26s, 1x 70s, 1x Stronger/Weaker (Voice) 120s, 4x 20s, 2x Cue (Keyboard) Cue (Voice) 70s, 4x 180s, 2x 75s, 1x Help - F1 (Keyboard)

Evaluation: discovered commands Interface 2: Touch, Voice Commands Shoot (Voice) 20s, 4x 13s, 6x 2s, 8x Shoot (Gesture) 50s, 3x 15s, 14x 70s, 10x 4s, 35x Zoom (Gesture) 50s, 1x 120s, 2x Camera Move (Gesture) 15s, 16x 40s, 39x Place the Ball (Gesture) 20s,4x 1s, 3x 1s, 5x 5s, 5x Bird view (Voice) 80s,10x 180s, 2x Commands (Voice) 140s, 1x 1s, 1x Stronger/Weaker (Voice) 34s, 5x 130s, 2x 54s, 14x Cue (Voice) 55s, 1x 30s, 2x 33s, 1x Place the Ball (Voice) 28s, 2x

Results Which interface do people learn better without instructions? Interface 2 (Touch, voice commands) The usage of which interface gives better results in playing the game? Which interface is more attractive/satisfying for users? Interface 1 (touch, keyboard, mouse) Interface 2 (touch, voice commands) Overall Score (number of balls killed) 29 34 Average Satisfaction 6.8 7.5

Video Dominic