1. Making sense out of recorded user-system interaction Dr Willem-Paul Brinkman Lecturer Department of Information Systems and Computing Brunel University (willem.brinkman@brunel.ac.uk)

2. Topics  VIVID Research Centre  Motivation - Component-Based Software Engineering  Experiment 1: Searching for a component- specific measure  Experiment 2: Validating a component-specific measure  New and future research

3. VIVID Research Centre Based in the Department of Information Systems and Computing, Brunel University (London) Original focus on visualisation, but now also includes: - Mobile technology - Design for diverse user groups - Novel input/output devices 11 academics member of staff, 13 PhD Students disc.brunel.ac.uk/research/vivid/index.htm

4. Motivation Studying the usability of a system Work conducted together with Reinder Haakma (Philips), Don Bouwhuis (Eindhoven University of Technology)

5. Motivation ExternalComparison External Comparison relating difference in usability to differences in the systems InternalComparison Internal Comparison trying to link usability problems with parts of the systems

6. Component-Based Software Engineering  Multiple versions testing paradigm  Single version testing paradigm Manage Support Re-use Create Re-use

7. Motivation PROBLEM 1.Only empirical analysis of the overall system such as (task time, keystrokes, questionnaires etc) - not powerful 2.Usability tests, heuristic evaluations, cognitive walkthroughs where experts problems – unreliable SOLUTION Component-Specific usability measures: more powerful and reliable

8. Searching for a component- specific measure Questions 1.What is a component? 2.What interaction data should be recorded? 3.How do we link interaction data with the usability of a component?

9. Layered Protocol Theory Layered Protocol Theory (Taylor, 1988)

10. Interaction layers 15 + 23 = 15+23= 01111 10111 Add 100110 38 ProcessorEditor Control results Control equation UserCalculator 15 15 + 15 + 23 38

11. Experiment 1 – Fictitious Interface User Task: Rotate the Trumpet

12. Experiment 1 - Architecture Other symbols Rotator Map Selector Buttons Bike Aeroplane Rotate Change X Rotate(x)

13. Experiment 1 - Architecture Other symbols Rotator Map Selector Buttons Bike Aeroplane Rotate Change X Rotate(x) Low High Measures Task time #Rotate(T 0 ), #Rotate(T -1 ), #Rotate(T -2 ) #change, #rotate #bike,#aeroplane, #other #clicks

14. Experiment 1 - Training

15. Experiment 1 : Test Procedure 80 participants, all students of Eindhoven University of Technology 8 different trainings After training participants were asked to rotate, as fast as possible, a specific music instrument User interaction with the system was recorded in log file Once a task was complete the recording stops

16. Experiment 1 - Low-level Effect of Selector training Clicks on Number messages

17. Experiment 1 - High-level Effect Rotator Training #Rotate (X) Number messages

18. Experiment 1 – Control Loop Reliability: how do we link interaction data with the usability of a component? Evaluation Component User message Feedback Reference value User System Each message is a cycle of the control loop  Number of messages presents the user’s effort to control the component

19. Experiment 1 - Conclusion 1.What is a component? An interaction component is a unit within a device that directly or indirectly receives signals from the user. These signals enable the user to change the physical state of the interaction component 2.What interaction data should be recorded? Message exchange between the interaction components

20. Experiment 2 : Validation 80 users 8 mobile telephones 3 components were manipulated according to Cognitive Complexity Theory (Kieras & Polson, 1985) 1.Function Selector 2.Keypad 3.Short Text Messages

21. Architecture Mobile telephone Send Text Message Send Text Message Function Selector Function Selector Keypad

22. Experiment 2 – Function Selector Versions: Broad/shallow Narrow/deep

23. Experiment 2 – Keypad Versions Repeated-Key Method “L” Modified-Model-Position method “J”

24. Experiment 2 – Send Text Message Versions Simple Complex

25. Statistical Tests p-value: probability of making type I, or , error, wrongly rejecting the hypothesis that underlying distribution is same.

26. Results – Function Selector Results of two multivariate analyses and related univariate analyses of variance with the version of the Function Selector as independent between-subjects variable.

27. Results – Keypad Results of multivariate and related univariate analyses of variance with the version of the Keypad as independent between-subjects variable.

28. Results – Send Text Message Results of two multivariate analyses and related univariate analyses of variance with the version of the STM component as independent between-subjects variable

29. Power of number of messages as a usability measure Statistical Power: 1 - β

30. Results Average probability that a measure finds a significant (α = 0.05) effect for the usability difference between the two versions of FS, STM, or the Keypad components

31. Component-Based Software Engineering  Multiple versions testing paradigm  Single version testing paradigm Manage Support Re-use Create Re-use

32. Testing Different Components Component specific objective performance measure: 1.Messages received + Weight factor A common currency 2.Compare with ideal user A common point of reference Usability of individual components in a single device can be compared with each other and prioritized on potential improvements

33. Click {1} Click {1} Call <>{2} Set <Fill colour red, no border> {7} Right Mouse Button Menu Properties Assigning weight factors to represent the user’s effort in the case of ideal user

34. Total effort value Total effort =  MR i.W MR i.W : Message received. Weight factor Click {1} Click {1} Call <>{2} Right Mouse Button Menu Properties 5 2 = 7 + 2

35. Assigning weight factors in case of real user Correction for inefficiency of higher and lower components Visual Drawing Objects Properties Right Mouse Button Menu

36. Assigning weight factors in case of real user Assign weight factors as if lower components operate optimal Visual Drawing Objects Properties Right Mouse Button Menu Inefficiency of lower level components: need more messages to pass on a message upwards than ideally required

37. Assigning weight factors in case of real user Visual Drawing Objects Properties Right Mouse Button Menu Inefficiency of higher level components: more messages are requested than ideally required UE: User effort MR i.W : Message received. Weight factor #MSU real :Number of messages sent upward by real user #MSU ideal :Number of messages sent upward by ideal user  MR i.W #MSU real  #MSU ideal UE =

38. Ideal User versus Real User Extra User Effort = User Effort - Total effort The total effort an ideal user would make The total effort a real user made The extra effort a real user made Calculate for each component: Prioritize

39. Experiment 2 - Single version 40 users 4 mobile telephones 2 components were manipulated (Keypad only Repeated-Key Method) 1.Function Selector 2.Short Text Messages

40. Results Mobile phones Extra User Effort

41. Results MeasureFunction Selector Send Text Message Objective Extra keystrokes0.64**0.44** Task duration0.63**0.39** Perceived Overall ease-of-use-0.43**-0.26* Overall satisfaction-0.25*-0.22 Component-specific ease-of-use-0.55**-0.34** Component-specific satisfaction-0.41**-0.37** Partial correlation between extra user effort regarding the two components and other usability measures *p. <.05. **p. <.01.

42. Topics  VIVID Research Centre  Motivation - Component-Based Software Engineering  Experiment 1: Searching for a component-specific measure  Experiment 2: Validating a component-specific measure  New and future research -Extending the analysis outside the lab -Extending the analysis beyond only usability issues

43. New Projects - Field usability CD player, which 10 users will use at home Record interaction: online assignment of weigh factors, both optimal and real user, to messages Correlated interaction data with other data (questionnaire, dairy, interview) (Pui-Fong Man)

44. New Projects - PROSKIN Exciting Interface designed for the average user. However, the average user does not exist. Developing skins for specific user groups could be a way forward Question: How to identify user groups? What do user groups want? Work conducted together with Nick Fine User profiling for skinnable domestic technology

45. New Projects - PROSKIN Possible solution Recording online interaction, Identifying user groups, Developing skins for these user groups Question How to establish user groups that are relevant for designer? This time, how to make sense of the interaction data beyond usability? Work conducted together with Nick Fine User profiling for skinnable domestic technology

46. New Projects - PROSKIN Approach Interaction data User metrics User groups based on interaction data Design of Skins Online Validation

47. Conclusions and Final Remarks Interaction data can be used to study the usability of interaction components -External Comparison between different versions: More Powerful -Internal Comparison: prioritized on potential improvements Future questions -Usability analysis of everyday life interaction -Establishing new paradigms to understand interaction data beyond usability issues

48. Questions Thank you for your attention