Interaction techniques for post-WIMP interfaces Lawrence Sambrooks Supervisor: Dr Brett Wilkinson

Summary Recap Project changes Timeline Issues Development of applications User testing Initial results Remaining tasks 2

Recap 3 Recent exposure and usage of alternative interaction devices for post-WIMP interfaces; for example, touch with tablets, Kinect gestures with Xbox 360 Investigate touch and gesture interaction techniques and compare them to the mouse and keyboard Develop applications to test performance and usability

Project changes 4 Added Fitts’ law style application to evaluate performance Replaced data visualisation application with 3D cube docking/alignment application Inclusion of speech for gestural interaction Studies suggest preference for using gestures and speech together (Hauptmann 1989) “Put-That-There” system by Bolt (1980)

Timeline 5

6

Issues 7 Performance of Windows-based tablet Poor touch responsiveness Not enough CPU and GPU “grunt” Effected development of application 4 in 3D (touch) Changes to Kinect SDK during development Beta 2 to version 1.0 Added new functionality but changed the public interface Getting a reasonable number of participants for testing

Development 8 4 applications Written in C# and WPF All applications run full-screen at 1280x

Development 9 Tools Visual Studio 2010 Expression Blend 4 Libraries.NET Framework 4.0 Kinect SDK beta2 / 1.0 Helix Toolkit Coding4Fun Kinect Toolkit Petzold.Media3D

Application 1 10 Performance evaluation using Fitts’ law 100 randomly sized and positioned squares Divided into groups of second break in-between each group Accepts mouse, touch, and Kinect (gesture) input

Screenshot 11 Application 1

Applications D cube docking/alignment task 10 pseudo-random targets Manipulation via Rotate-Scale-Transform (RST) operations Separate application for each interaction technique Each application shares common library (CubeEx.dll)

Gestures and speech 13 Gestures control manipulation One hand (left or right) for translation Two hands for scale and rotation Speech invokes commands Audible and visual feedback when command is recognised 70% confidence threshold for recognition Not perfect! Application 2

Screenshot 14 Application 2

Mouse and keyboard 15 Mouse controls manipulation of camera/cube Left-click-and-drag to translate cube Keyboard used to invoke scale and rotate modes Hold Z to scale Hold X to rotate Application 3

Multi-touch 16 Tablet performance prohibited creation of 3D application Poor touch responsiveness Not enough CPU and GPU “grunt” Due to time available, compromised on 2D version Still allows some evaluation of usability Application 4

Screenshot 17 Application 4

User testing 18 2 experiments utilising all 4 applications Experiment 1: Performance Experiment 2: Usability 15 participants Repeated-measures design Order of interaction techniques shuffled No practice time Questionnaire at the conclusion of each experiment Multiple-choice, short answer, and Likert scale

Initial results 19 On average, touch performed best (9.83 bits/s) followed by the mouse (7.76 bits/s) and gestures (1.23 bits/s) It took ~3.5 times longer using gestures to acquire targets compared with mouse/touch Participants missed an average of 6 targets (per 100) using the mouse, 12 using touch, and 32 using gestures Experiment 1 – Performance

Results browser 20 Experiment 1 – Performance

Initial results 21 Previous experience 85%+ with multi-touch (attributed to smartphone/tablet ownership) About 50% with gestures Participants overwhelmingly preferred the mouse Mouse had lowest number of “misses;” touch and gestures were both around 3 times worse Main reason cited was familiarity followed by speed and precision Experiment 2 – Usability

Initial results 22 Participant rating of performance for techniques used Mouse and touch best for translation Mouse best for rotation Mouse best for scaling Noted issues/frustrations Lack of precision when using gestural interaction Accuracy of voice recognition Gulf of execution for Kinect (gesture) task Experiment 2 – Usability

Results browser 23 Experiment 2 – Usability

Remaining tasks 24 In-depth analysis of results Prepare for expo (poster, application demo, etc.) Complete thesis

Questions?