Xiaoyan Peng and Daniel L. Silver Jodrey School of Computer Science Acadia University Wolfville, Nova Scotia Canada

Synopsis Problem: The mix of automation and user control is a major issue for a successful UAI Theory: A UAI will be acceptable as long as the state of system interaction is within the current region of user interaction expectation Application: Spam filtering via a learned UM that predicts user’s sense of priority Solution: The user is provided with control over spam classification by the UM via a novel priority cut-off adjustment method

User Adapted Interfaces (UAI) = Human-Computer Interaction (HCI) + User Modeling (UM) To improve the usability and performance of software systems Controllability is a major usability issue for UAI technology Most research has focused on the performance of the learning algorithm Deficiency of systematic gather evidence about what users think about adaptation and controllability Motivation

Space of HCI s R P R2 R3 R’ NowPastFuture P -- System adaptation through the space. Each point along P represents the system’s state of interaction with the user at a particular time. R, R2, R3, R’ – The user’s region of interaction expectation. The size of R, |R|, is the number of interaction states within R. If |R| = 1, then no variation from s will be tolerated by the user; this user is very conservative in terms of adaptation. If |R| = n then there are n states within R that will be acceptable to the user; this user is more accepting of adaptation. S -- The systems current state of interaction. Adaptation Viewed as Movement through as HCI State Space

A user has a region of interaction expectation, R, that preferably is centered on the systems current state of interaction, s, or at least contains the s. Ideally, as the system interface adapts, the user shifts her R so as to centre it once again on the new s. This transition is not always in concert. If the system adapts too quickly then the user is left behind at R2. If the system adapts too slowly then the user may assume an interaction state too far in advance of the current s, at R3. In either case the user will not be satisfied with the system and task performance will suffer. The worst case is when the user’s expectation region is R’, a region of interaction space through which adaptation will never pass; the user is continually dissatisfied. To be successful, a UAI must provide the user with control over adaptation. We advocate that user control should be exercised over the deployment of user models rather than their development. Adaptation Viewed as Movement through as HCI State Space Theory

Empirical Study -- Objective An adaptive intelligent client is our application of choice for employing UAI. In this study we focus on predicting the priority of incoming messages based on a learned user model. Control over automatic spam filtering can then be provided by allowing the user to adjust cut-off values that determine when the predicted priority of a message is at the level of legitimate or spam. The objective of this experiment is to demonstrate that user control over appropriate aspects a UAI can improve the usability of the application and user satisfaction.

Empirical Study -- Implementation We have created an intelligent client and developed an intuitive user interface for controlling adaptation. There are two priority cut-off values Suspect cut-off value Spam cut-off value with a priority value lower than the spam cut-off will be placed in the Spam folder. with a priority value equal to or higher than the suspect cut-off will be filed into the Inbox folder. with a priority equal to or higher than the spam cut-off and lower than the suspect cut-off will be put in a Suspect folder.

Empirical Study – Materials and Methods Three variants of the system were tried. Variant N – It employs a UAI based on “no user model”. Variant F – It develops a user model but uses “fixed cut-off values” to determine the priority required for spam and legitimate s. Variant A – It develops a user model and allows the subjects to adjust the spam and suspect cut-off values as desired. This gives the subjects control over the UAI. Each subject used all 3 system variants in one of two possible orders; the first order being N, F and A and the second being N, A, and F. A different subset of s was used for each variant to prevent subjects from memorizing the content of messages. Each subject was surveyed following their trial of the 3 system variants.

Empirical Study – Results and Discussion 92.86% of subjects preferred the UAI variant of the system after the user model was developed, with 82.14% of subjects preferring the adjustable spam and suspect cut- offs over the fixed cut-offs % of subjects felt that the cut-off adjustment increased the accuracy of classifications with more subjects preferring cut-off adjustment on their current clients (78.57%) than user modeling with fixed cut-offs (71.43%). Despite the fact that significantly more misclassifications were made by variant A (with adjustable cut-offs), 67.9% of the subjects (19/28) preferred variant A over F.

Empirical Study – Results and Discussion Of the subjects who liked cut-off adjustment, 95.65% preferred FN over FP meaning they most dislike finding legitimate messages in the Spam folder. Adjustment of the cut-offs allows the users to err on the side of FN classifications even if this reduces the overall performance of the UAI. Of the subjects who responded “do not know” or “disagree” to cut-off adjustment, 80% are less sensitive to FP (legitimate s classified as spam). The fixed default cut-off values worked well for that purpose, the subjects recognized this and preferred it.