Making sense out of recorded user-system interaction Dr Willem-Paul Brinkman Lecturer Department of Information Systems and Computing Brunel University

Topics  VIVID Research Centre  Motivation - Component-Based Software Engineering  Experiment 1: Searching for a component- specific measure  Experiment 2: Validating a component-specific measure  New and future research

VIVID Research Centre Based in the Department of Information Systems and Computing, Brunel University (London) Original focus on visualisation, but now also includes: - Mobile technology - Design for diverse user groups - Novel input/output devices 11 academics member of staff, 13 PhD Students disc.brunel.ac.uk/research/vivid/index.htm

Motivation Studying the usability of a system Work conducted together with Reinder Haakma (Philips), Don Bouwhuis (Eindhoven University of Technology)

Motivation ExternalComparison External Comparison relating difference in usability to differences in the systems InternalComparison Internal Comparison trying to link usability problems with parts of the systems

Component-Based Software Engineering  Multiple versions testing paradigm  Single version testing paradigm Manage Support Re-use Create Re-use

Motivation PROBLEM 1.Only empirical analysis of the overall system such as (task time, keystrokes, questionnaires etc) - not powerful 2.Usability tests, heuristic evaluations, cognitive walkthroughs where experts problems – unreliable SOLUTION Component-Specific usability measures: more powerful and reliable

Searching for a component- specific measure Questions 1.What is a component? 2.What interaction data should be recorded? 3.How do we link interaction data with the usability of a component?

Layered Protocol Theory Layered Protocol Theory (Taylor, 1988)

Interaction layers = 15+23= Add ProcessorEditor Control results Control equation UserCalculator

Experiment 1 – Fictitious Interface User Task: Rotate the Trumpet

Experiment 1 - Architecture Other symbols Rotator Map Selector Buttons Bike Aeroplane Rotate Change X Rotate(x)

Experiment 1 - Architecture Other symbols Rotator Map Selector Buttons Bike Aeroplane Rotate Change X Rotate(x) Low High Measures Task time #Rotate(T 0 ), #Rotate(T -1 ), #Rotate(T -2 ) #change, #rotate #bike,#aeroplane, #other #clicks

Experiment 1 - Training

Experiment 1 : Test Procedure 80 participants, all students of Eindhoven University of Technology 8 different trainings After training participants were asked to rotate, as fast as possible, a specific music instrument User interaction with the system was recorded in log file Once a task was complete the recording stops

Experiment 1 - Low-level Effect of Selector training Clicks on Number messages

Experiment 1 - High-level Effect Rotator Training #Rotate (X) Number messages

Experiment 1 – Control Loop Reliability: how do we link interaction data with the usability of a component? Evaluation Component User message Feedback Reference value User System Each message is a cycle of the control loop  Number of messages presents the user’s effort to control the component

Experiment 1 - Conclusion 1.What is a component? An interaction component is a unit within a device that directly or indirectly receives signals from the user. These signals enable the user to change the physical state of the interaction component 2.What interaction data should be recorded? Message exchange between the interaction components

Experiment 2 : Validation 80 users 8 mobile telephones 3 components were manipulated according to Cognitive Complexity Theory (Kieras & Polson, 1985) 1.Function Selector 2.Keypad 3.Short Text Messages

Architecture Mobile telephone Send Text Message Send Text Message Function Selector Function Selector Keypad

Experiment 2 – Function Selector Versions: Broad/shallow Narrow/deep

Experiment 2 – Keypad Versions Repeated-Key Method “L” Modified-Model-Position method “J”

Experiment 2 – Send Text Message Versions Simple Complex

Statistical Tests p-value: probability of making type I, or , error, wrongly rejecting the hypothesis that underlying distribution is same.

Results – Function Selector Results of two multivariate analyses and related univariate analyses of variance with the version of the Function Selector as independent between-subjects variable.

Results – Keypad Results of multivariate and related univariate analyses of variance with the version of the Keypad as independent between-subjects variable.

Results – Send Text Message Results of two multivariate analyses and related univariate analyses of variance with the version of the STM component as independent between-subjects variable

Power of number of messages as a usability measure Statistical Power: 1 - β

Results Average probability that a measure finds a significant (α = 0.05) effect for the usability difference between the two versions of FS, STM, or the Keypad components

Component-Based Software Engineering  Multiple versions testing paradigm  Single version testing paradigm Manage Support Re-use Create Re-use

Testing Different Components Component specific objective performance measure: 1.Messages received + Weight factor A common currency 2.Compare with ideal user A common point of reference Usability of individual components in a single device can be compared with each other and prioritized on potential improvements

Click {1} Click {1} Call <>{2} Set <Fill colour red, no border> {7} Right Mouse Button Menu Properties Assigning weight factors to represent the user’s effort in the case of ideal user

Total effort value Total effort =  MR i.W MR i.W : Message received. Weight factor Click {1} Click {1} Call <>{2} Right Mouse Button Menu Properties 5 2 = 7 + 2

Assigning weight factors in case of real user Correction for inefficiency of higher and lower components Visual Drawing Objects Properties Right Mouse Button Menu

Assigning weight factors in case of real user Assign weight factors as if lower components operate optimal Visual Drawing Objects Properties Right Mouse Button Menu Inefficiency of lower level components: need more messages to pass on a message upwards than ideally required

Assigning weight factors in case of real user Visual Drawing Objects Properties Right Mouse Button Menu Inefficiency of higher level components: more messages are requested than ideally required UE: User effort MR i.W : Message received. Weight factor #MSU real :Number of messages sent upward by real user #MSU ideal :Number of messages sent upward by ideal user  MR i.W #MSU real  #MSU ideal UE =

Ideal User versus Real User Extra User Effort = User Effort - Total effort The total effort an ideal user would make The total effort a real user made The extra effort a real user made Calculate for each component: Prioritize

Experiment 2 - Single version 40 users 4 mobile telephones 2 components were manipulated (Keypad only Repeated-Key Method) 1.Function Selector 2.Short Text Messages

Results Mobile phones Extra User Effort

Results MeasureFunction Selector Send Text Message Objective Extra keystrokes0.64**0.44** Task duration0.63**0.39** Perceived Overall ease-of-use-0.43**-0.26* Overall satisfaction-0.25*-0.22 Component-specific ease-of-use-0.55**-0.34** Component-specific satisfaction-0.41**-0.37** Partial correlation between extra user effort regarding the two components and other usability measures *p. <.05. **p. <.01.

Topics  VIVID Research Centre  Motivation - Component-Based Software Engineering  Experiment 1: Searching for a component-specific measure  Experiment 2: Validating a component-specific measure  New and future research -Extending the analysis outside the lab -Extending the analysis beyond only usability issues

New Projects - Field usability CD player, which 10 users will use at home Record interaction: online assignment of weigh factors, both optimal and real user, to messages Correlated interaction data with other data (questionnaire, dairy, interview) (Pui-Fong Man)

New Projects - PROSKIN Exciting Interface designed for the average user. However, the average user does not exist. Developing skins for specific user groups could be a way forward Question: How to identify user groups? What do user groups want? Work conducted together with Nick Fine User profiling for skinnable domestic technology

New Projects - PROSKIN Possible solution Recording online interaction, Identifying user groups, Developing skins for these user groups Question How to establish user groups that are relevant for designer? This time, how to make sense of the interaction data beyond usability? Work conducted together with Nick Fine User profiling for skinnable domestic technology

New Projects - PROSKIN Approach Interaction data User metrics User groups based on interaction data Design of Skins Online Validation

Conclusions and Final Remarks Interaction data can be used to study the usability of interaction components -External Comparison between different versions: More Powerful -Internal Comparison: prioritized on potential improvements Future questions -Usability analysis of everyday life interaction -Establishing new paradigms to understand interaction data beyond usability issues

Questions Thank you for your attention