July 24, 2005UM 2005, Edinburgh, UK1 User Control over User Adaptation: A Case Study Xiaoyan Peng and Daniel L. Silver Jodrey School of Computer Science Acadia University Wolfville, Nova Scotia Canada B4P 2R6

2 Introduction UAI = HCI + UM Pro: Potential for more usable software interfaces Con: Complexity of system, user uncertainty Controllability – degree of human control over the system remains a key factor: Some researchers prefer maximum control Others suggest control can lead to distraction [1] An adaptive intelligent client is the application of choice: Focus on predicting the priority of incoming messages based on a learned UM The predicted priorities can be used to filter low priority “Spam” and identify high priority messages quickly

3 Background Prior work on UM for spam filtering [4] has focused on performance of the learning algorithms Less attention given to the usability of the UAI based system Lack of a user’s perspective has led to significant barrier against UAI technology [2] For example: automatic placement of legitimate into a spam folder can be unacceptable to some users [5] UAI can frustrate good HCI design – the interface may be perceived as a moving target that at times does not meet the expectations of the user [6] We present a theoretical model of the relationship between the expectations of a user and the changing state of a UAI

4 Theory of User Interaction Expectation and Adaptation Figure 1. Adaptation viewed as movement through an HCI state space Space of HCI states s R P R2 R3 R’ NowPastFuture

5 Theory of User Interaction Expectation and Adaptation Consider a space of HCI states as shown in Figure 1 Interaction states are topologically organized Similar states are proximal to each other System adaptation is trajectory, P, through the space Each point represents the system’s state of interaction with the user at a particular time A user has a region of interaction expectation, R Preferably R contains the systems current state of interaction, s |R|, is the number of interaction states within R If |R| = 1, then no variation from s will be tolerated by the user; this user is very conservative in terms of adaptation If |R| = n then there are n states that will be acceptable to the user; this user is more accepting of adaptation

6 Theory of User Interaction Expectation and Adaptation Ideally, as the system interface adapts, the user shifts her R so as to centre it on the new s This transition is not always in concert: If the system adapts too quickly, the user is left behind at R2 If the system adapts too slowly, the user may assume an interaction state too far in advance of the current s, at R3 In either case the user will not be satisfied with the system and task performance will suffer The worst case is R’, a region of interaction space through which adaptation will never pass - the user is continually dissatisfied

7 Theory of User Interaction Expectation and Adaptation A UAI should provide the user with control over adaptation: We advocate that user control should be exercised over the deployment of user models rather than their development Model deployment requires minimal knowledge of the UM subsystem Example: A user model for predicting incoming priority can be developed using information retrieval and machine learning methods [4] Control over spam filtering can be provided by adjustable cut-off values that determine when the predicted priority of a message classifies it as legitimate or Spam

8 The Intelligent Client We have created an intelligent client and developed an intuitive user interface for controlling adaptation (see Figure 3) The user model is trained to assign a value between 0 and 1 to each incoming message 0 is lowest priority, 1 is highest priority The default priority (no user model) is 0.5 The user model is developed with a BP ANN system using training examples from the current spam and legitimate folders (spam messages are assign priority 0, all others 1)

9 The Intelligent Client Figure 3. Major interface of the client

10 User Control over Adaptation There are two priority cut-off values, Suspect and Spam, controlled by two GUI sliders (Figure 4): with a priority value lower than the spam cut-off will be placed in the Spam folder. with a priority value equal to or higher than the suspect cut-off will be filed into the Inbox folder. with a priority equal to or higher than the spam cut-off and lower than the suspect cut-off will be put in a Suspect folder.

11 User Control over Adaptation Figure 4. Interface of cut-off controlling function

12 User Control over Adaptation Using the GUI sliders, a new or conservative user can select cut-off values that curtail the UAI’s automated classification of legitimate and spam messages (thus reducing risk) A more experienced user can establish cut-offs that give the UAI greater freedom to classify messages As the cut-offs are adjusted, the system automatically reallocates the messages to the Inbox, Suspect and Spam folders Provides immediate feedback to the user on their choice of cut-offs Adaptation of the systems interaction state can be kept within the user’s current region of interaction expectation. The approach will direct the most important legitimate to the Inbox folder The Suspect folder can be cleaned up periodically, sorting legitimate and spam - it is this process that provides data for improving the user model.

13 Empirical Study Objective: To demonstrate that user control over appropriate aspects of UAI can improve the usability and user satisfaction Scenario: Each subject pretends to be a secretary for a professor. He or she must classify the incoming (initially received in either the Inbox, Suspect or Spam folder) by moving the messages into one of six relevant folders including the Spam folder. Performance of the UAI is recorded in terms of: FP (false positives) = legitimate s placed in the Spam folder FN (false negatives) = spam placed in the Inbox folder Overall error = FP + FN Usability of the system is based on user surveys, post trial

14 Empirical Study - Method Three variants of the system were tried by each subject and compared as per [2]: Variant N - no user model; subjects must manually sort the Inbox to legitimate & Spam folders Variant F – user model with fixed cut-off values; user model automatically sorts to the Inbox (legitimate), Suspect and Spam folders based fixed cut-off values set to optimally performing values (as determined by preliminary trials) Variant A - user model with adjustable cut-off values; user model automatically sorts to the Inbox (legitimate), Suspect and Spam folders based adjustable spam and suspect cut-off values. Subjects have control over the deployment of user model.

15 Empirical Study - Method A within-subject experimental design selected: 28 subjects selected from the university campus (ages 18-38) Subjects expected to vary considerably in their use of the system and tolerance to adaptation Wanted to collect comparative results/comments over all 3 variants Overcoming experimental bias: Each subject used all three variants of the system in one of two possible orders: variant N, F, then A or variant N, A, then F A different subset of s was used for each variant to prevent subjects from memorizing the content of messages. Each subject was provided the same working environment. The instructions were provided by a power point presentation with minimal input by a researcher

16 Empirical Study - Method Procedure: data used was collected from a professor at Acadia over a 5 month timeframe in 2003 [4]. Three different subsets of 100 s used for each of the three variants of the system: 50 legitimate and 50 spam messages Variant N was always first - the subject was asked to manually sort the 100 s to their respective folders without assistance of the user model This acted as training data for developing a user model for predicting message priority for variants A and F The FP and FN statistics were recorded for each subject and used to determine each variant’s performance

17 Empirical Study – Results 92.86% of subjects found the client easier to use after the UM was developed Figure 6 shows the difference between the fixed cut-off and adjustable cut-off variants of the UAI for both orders of exp. The fixed cut-off variant performed better on average with fewer overall misclassifications (p = 0.01, paired, two-tailed T- test) However, 82.14% of subjects preferred the ability to adjust the Spam and Suspect cut-offs 78.57% felt that the cut-off adjustment increased the accuracy of classifications

18 Figure 6. Misclassifications of variant A and F. Order of system variants used is defined by order of N, F and A in bar labels Empirical Study – Results

19 Figure 7. Summary of survey results: preference for user control versus performance. Empirical Study – Results Subject ID: Brackets represents a subject who tried variant F before A; otherwise a subject who tried variant A before F.

20 Empirical Study – Results Those subjects above the line represent fewer misclassifications under variant A than variant F The distance between an ID and the line indicates the difference in performance between the two variants Examples: Subject 15 preferred variant A, had 51 misclassifications with variant A, 50 with variant F; a difference of 1 Subject 14 preferred variant A, had 49 misclassifications with variant A, 5 with variant F; a difference of 44 Subject 10 preferred variant F, had 11 misclassifications with variant A, 22 with variant F; a difference of -11

21 Empirical Study – Results 67.9% of the subjects (19/28) preferred variant A over F Subject 14 was an extreme case of where a user preferred control even though it reduced system performance This shows a strong desire to remain in control Typical responses for those who preferred adjustable cut-offs were: “It helps me to control how the s will be separated” “It is good to add user’s point view to the system” “I like the feeling of control”

22 Empirical Study – Results Majority of subjects liked adaptation as long as they felt in control: 95.65% of subjects who liked cut-off adjustment, preferred FN (spam messages classified as legitimate) Adjustable cut-offs allow the user to err on the side of FN classifications even if this reduces overall performance Of the subjects who responded “do not know” or “disagree” to cut-off adjustment: 80% preferred FP (legitimate s classified as spam) The fixed default cut-off values worked well for that purpose, the subjects recognized this and preferred it

23 Conclusions and Future Work Theory: Users will be satisfied with a UAI provided the interaction state of the system is maintained within the current region of user expectation If the system’s interaction falls outside of this region of expectation then user satisfaction and performance will degrade Solution: Give the user control over aspects of adaptation that limit changes in interaction state. In the case of the client, the user controls the cut-off at which s are considered legitimate or spam. The results of an empirical study using 28 subjects demonstrated that performance and user satisfaction is improved with user control over adaptation User satisfaction is higher even when control leads to reduced performance (greater numbers of misclassifications) We are currently working on a related problem of automatically classifying s to one of several category folders

24 References 1. Kay, J.: Learner control, Proceedings of User Modeling & User-Adapted Interaction (2001) 2. Jameson A., and Schwarzkopf, E.: Pros and Cons of Controllability: An Empirical Study, Adaptive Hypermedia and Adaptive Web-Based Systems: Proceedings of AH (2002) 3. Crawford, E., Kay, J., and McCreath E.: An Intelligent Interface for Sorting Electronic Mail, IUI’02, San Francisco, California, USA (2002) Fu, C.: User Modelling for an Adaptive System: an Intelligent Client, Master Thesis, Jodrey School of Computer Science, Acadia University, Wolfville, Canada (2003) 5. Crawford, E., Kay, J., and McCreath, E.: Automatic Induction of Rules for Classification, Proceedings of 6th Australian Document Computing Symposium (2001) 6. Cranor, L., F.: Designing a Privacy Preference Specification Interface: A Case Study, Proceedings of the Workshop on HCI and Security Systems, CHI2003, Florida (2003)

25 Introduction Objective: To improve usability and performance of User Adapted Interfaces (UAI) UAI = HCI + UM Pro: Potential for better software interfaces Con: Complexity of interface, user uncertainty Controllability – degree of human control over the system remains a key factor: Some researchers prefer maximum control Others suggest control can lead to distraction [1] There is a deficiency of evidence from users perspective on adaptation and controllability [2]

26 Introduction An adaptive intelligent client is the application of choice: Offers lots of functionality Example data is readily available Knowledgeable test subjects available Focus on predicting the priority of incoming messages based on a learned UM The predicted priorities can be used to: Filter low priority “spam” Identify high priority messages quickly

27 Synopsis Problem: The mix of automation and user control is a major issue for a successful UAI Theory: A UAI will be acceptable as long as the state of system interaction is within the current region of user interaction expectation Application: Spam filtering via a learned user model that predicts user’s sense of priority Solution: The user is provided with control over filtering via a novel method of adjusting priority cut-offs

28 The Intelligent Client Figure 2. System architecture of the intelligent client

29 Empirical Study - Subject Survey