The University of Nevada, Reno SYCOPHANT Sycophant A Context Based Generalized User-Modeling Framework for Desktop Applications Anil Shankar Evolutionary Computing Systems Laboratory (ECSL) PhD Defense May 13th, 2008

The University of Nevada, Reno SYCOPHANT Current Desktop Applications 22

The University of Nevada, Reno SYCOPHANT Overview ➢ Why predict user preferred application actions ➢ Sycophant ➢ Adaptive user interfaces research ➢ Learning to predict user preferred actions ➢ User studies ➢ Contributions to user modeling 21

The University of Nevada, Reno SYCOPHANT User Preferences Change ➢ Should your calendar remind you of an appointment if you are not in your office? ➢ Do you stop your music when you leave your room? ➢ Do you pause your music player when your phone rings? ➢ Preferences ➢ context dependent ➢ user dependent 20

The University of Nevada, Reno SYCOPHANT Personalization & Context Personalization ➢ An interface personalizes itself to an individual user if it adapts its actions depending on the context of use Context ➢ “Any information that can be used to characterize the situation of an entity” ➢ “An entity is a person, place or object that is considered relevant to the interaction between a user and an application, including the user and the application themselves” Anind K. Dey, Gregory D. Abowd, and Daniel Salber,

The University of Nevada, Reno SYCOPHANT Context Sources ➢ Internal clock, keyboard, mouse ➢ Advances in ➢ Machine Learning, Speech and Natural Language Processing, Computer Vision ➢ Motion? ➢ Speech? 18

The University of Nevada, Reno SYCOPHANT Main Hypothesis User-Context: Any user-related contextual information in the vicinity of a desktop computer ➢ Internal user-context: keyboard activity, states of different user processes, mouse activity ➢ External user-context: motion and speech in the user’s environment Central claim: External user-context enables applications to better predict user preferred actions thereby leading to better personalization of applications to individual users. Central claim: External user-context enables applications to better predict user preferred actions thereby leading to better personalization of applications to individual users. 17

The University of Nevada, Reno SYCOPHANT Thesis Questions 1. Feasibility ➢ Can Sycophant accurately predict a user-preferred application action? 2.Relevance of User-context ➢ Does user-context help a machine learner to better predict action preferences? 3.Robustness ➢ Which machine learning algorithm is best suited to personalize applications? 4.Generalizability a)Across users b)Across application c)Across duration of use 1. Feasibility ➢ Can Sycophant accurately predict a user-preferred application action? 2.Relevance of User-context ➢ Does user-context help a machine learner to better predict action preferences? 3.Robustness ➢ Which machine learning algorithm is best suited to personalize applications? 4.Generalizability a)Across users b)Across application c)Across duration of use 16

The University of Nevada, Reno SYCOPHANT Related Work Adaptive User Interfaces (AUI) ➢ Attention management – Bailey [2004, 2006,2 006] ➢ select the most opportune moment to interrupt a user ➢ best interruptions produce less annoyance and frustration ➢ monitor a user's primary task and generate interruption during task boundary points ➢ Interruption management – Iqbal [2006, 2007] ➢ manage notification cues generated by applications ➢ analysed multi-tasking user behavior ➢ an average of 10 minutes on switches caused by alerts ➢ use visual cues to serve as reminders 15

The University of Nevada, Reno SYCOPHANT Related Work Adaptive User Interfaces ➢ Task Tracer – Herlocker [2005, 2006] ➢ help multitasking users locate, discover, and reuse applications ➢ activity data from files, folders, s, web-pages ➢ use window information (title, document, pathname) and to predict a current task ➢ Lumière – Horvitz [1998, 2003] ➢ use probability measures (Bayesian networks) ➢ monitor user's background (gaze-tracking, active applications), queries, actions ➢ predict the interruption cost 14

The University of Nevada, Reno SYCOPHANT Related Work Adaptive User Interfaces ➢ Subtle – Fogarty [2004, 2006, 2007] ➢ statistical models to predict highly non-interruptible vs. interruptible ➢ Wizard of Oz. study with office workers to collect audio and video information along with self-reports of interruptibility ➢ Compared statistical models with human observers to predict interruptibility ➢ Subtle – mainly for notebook users ➢ Data collected from notebook opening/closing, audio analyses, mouse-clicks, and WiFi sensing ➢ Sensor-based statistical models can predict a user's interruptibility 13

The University of Nevada, Reno SYCOPHANT Related Work XCS Applications ➢ XCS' performance on Wisconsin Breast Cancer dataset (bench-marking) demonstrated both performance and pattern-discovery viewpoints (Wilson 1996) ➢ Monks bench-marking data sets: 1- concept generation, 2-complex relationships, 3-noise and misclassification ➢ XCS was again successful (Saxon and Barry 1999) ➢ XCS' predictive accuracy on Forest Cover Type dataset (real-world data) comparable with neural-networks, SVMs, and a decision-tree (Bagnall and Crowley 2003) ➢ One of Sycophant's predictive data miners 12

The University of Nevada, Reno SYCOPHANT Integrating AUI research in Sycophant ➢ User-context and user feedback to minimize disruptive interruption effects ➢ My framework gathers both internal and external user-related information in-contrast to previous work that focussed primarily on tasks ➢ Sycophant not only predicts when but also how to generate an interruption (application action-type) 11

The University of Nevada, Reno SYCOPHANT Sycophant ➢ Sensors collect internal and external user-context ➢ Machine Learning algorithms to build user models ➢ User-feedback for predicting interface actions 10

The University of Nevada, Reno SYCOPHANT Sycophant’s Four-layer Architecture 9

The University of Nevada, Reno SYCOPHANT Sycophant's Layers: Sensors Setting up a motion-sensor ➔ motionSensor = Sensor('motion', motionLogFile) ➔ motionSensor.start() ➔ motionSensor.shutdown() In ICSEA ’07: Proceedings of the International Conference on Software Engineering Advances (ICSEA 2007) 8

The University of Nevada, Reno SYCOPHANT Sycophant's Layers: Context Setting up user-context feature extractors ➔ lastNMinutes = 5 # duration of history check for a sensor ➔ motionFeatureExtractor = FeatureExtractor (motionLogFile) ➔ checkAnyNMinutes = motionFeatureExtractor.checkAny (lastNMinutes) ➔ checkAllNMinutes = motionFeatureExtractor.checkAlllastNMinutes) ➔ getCountAllNMinutes = motionFeatureExtractor.getCount (lastNMinutes) User-context features Count: checks how many times a sensor was active in the last five minutes All5: checks if a sensor was active in all the 15 seconds intervals in the last five minutes Any5:checks if a sensor was active in all the 15 seconds intervals in the last five minutes All1: Similar to All5 Any1:Similar to Any1 7

The University of Nevada, Reno SYCOPHANT Creating user-context data ➔ motionFeatures = checkAny-1, checkAll-1, checkAny-5, checkAll-5, getCount-5 ➔ speechFeatures = checkAny-1, checkAll-1, checkAny-5, checkAll-5, getCount-5 ➔ motionUserContextData = UserContextExtractor (motionFeatures) ➔ speechUserContextData = UserContextExtractor (speechFeatures) Sycophant's Layers: Context 6

The University of Nevada, Reno SYCOPHANT Sycophant's Layers: Learning Services Predicting user-preferred alarm types ➔ dataAvailable = checkForUserContextData ( ) ➔ userContextData = getUserContextData (dataAvai lable) ➔ userModel = buildUserModel(learner,userContextData ) ➔ predictedAlarm = predictAlarm ( userModel ) ➔ alarmGenerated = generateAlarm ( ) ➔ userFeedback = FeedbackRequest ( ) ➔ writeCurrentUserContext ( userContextData, predictedAlarm, userFeedback ) 5

The University of Nevada, Reno SYCOPHANT Sycophant's Layers: Application Context-enabling Google Calendar ➔ calendarMonitor = GCalMonitor ( calendarFile) ➔ calendarMonitor.generateAlarms ( ) 4

The University of Nevada, Reno SYCOPHANT Predicting User Preferences ➢ Machine Learning: A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E” ➢ -Tom Mitchell, Machine Learning ➢ Techniques: Decision Tree (J48), Learning Classifier System (XCS) 3

The University of Nevada, Reno SYCOPHANT A Learning Example NoneYesLightShortBlondeKatie NoneNoHeavyAverageBrownJohn NoneNoHeavyTallBrownPete SunburnNoHeavyAverageRedEmily SunburnNoAverageShortBlondeAnnie NoneYesAverageTallBrownAlex noneYesAverageTallBlondeDana SunburnNoLightAverageBlondeSarah ResultLotionWeightHeightHairID Source: Context-Learning Systems (Dr.Sushil J. Louis) 2

The University of Nevada, Reno SYCOPHANT XCS Overview 1

The University of Nevada, Reno SYCOPHANT XCS ➢ Genetics Based Machine Learning (GBML) technique ➢ Working ➢ Sample a user-context data exemplar ➢ Create matching-set and action-set ➢ Choose an action, receive feedback, update fitness 0

The University of Nevada, Reno SYCOPHANT Study 1: Pilot Study ➢ 3 participants ➢ Data collected over a span of six to eight weeks ➢ Average of 340 exemplars from each participant Two Alarm Problems Classes ➢ 2Class: Whether or not to generate an alarm ➢ 4Class: Predict one of the following four alarm types: ➢ Visual, Voice, Both Visual and Voice, and None

The University of Nevada, Reno SYCOPHANT Study 1: Results ➢ XCS and J48 performed better than ZeroR (base-rate) ➢ XCS matched J48's performance and outperformed the decision-tree(underlined values) static alarm preferences Thesis Questions ✔ Feasibility ✗ Relevance of user- context ✗ Robustness ✗ Generalizability ✗ participants ✗ applications ✗ term of use Thesis Questions ✔ Feasibility ✗ Relevance of user- context ✗ Robustness ✗ Generalizability ✗ participants ✗ applications ✗ term of use ➢ In Proceedings of the 2005 Indian International Conference on Artificial Intelligence, December , Poona, India, ➢ Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, IRI , November 8-10,

The University of Nevada, Reno SYCOPHANT Study 2: Google Calendar ➢ Objective: predict alarm type preferences for Google Calendar ➢ 10 participants ➢ 4 separate sessions each lasting 45 minutes ➢ Read short/long articles ➢ Answer questions after reading the articles ➢ Hints (alarms/reminders) generated for participants while reading ➢ A participant provides feedback specifying her preferred hint-type -3

The University of Nevada, Reno SYCOPHANT Study 2: Experimental Design *Alarm Types: 0 =None; 1 = Visual; 2 = Voice; 3 = Both -4

The University of Nevada, Reno SYCOPHANT Study 2: Results 2Class Problem ➢ External user-context is important to predict alarm type preferences ➢ XCS outperforms the decision-tree learner (J48) -5

The University of Nevada, Reno SYCOPHANT Study 2: Results 4Class Problem ➢ External user-context is important to predict alarm type preferences ➢ XCS outperforms the decision-tree learner (J48) ➢ In IUI ’07: Proceedings of the 12th international conference on Intelligent user interfaces ➢ In GECCO ’07: Proceedings of the 9th annual conference on Genetic and evolutionary computation -6

The University of Nevada, Reno SYCOPHANT Study 2: Google Calendar Personalization -7

The University of Nevada, Reno SYCOPHANT Study 2: Google Calendar Summary ➢ Consistent with pilot-study results ➢ External user-context important for predicting alarm type preferences ➢ XCS is more robust than J48 Thesis Questions ✔ Feasibility ✔ Relevance of user- context ✔ Robustness ✔ Generalizability ✔ participants ✗ applications ✗ term of use Thesis Questions ✔ Feasibility ✔ Relevance of user- context ✔ Robustness ✔ Generalizability ✔ participants ✗ applications ✗ term of use -8

The University of Nevada, Reno SYCOPHANT Study 3: Winamp ➢ 10 participants ➢ 4 separate sessions each lasting 45 min. ➢ Read short/long articles while listening to music using Winamp, a media player ➢ Hints generated for the participants while reading ➢ Hints were related to article, a map reading task or the study itself. ➢ Answer questions after reading the articles ➢ A participant provides feedback specifying her preferred media player preference type -9

The University of Nevada, Reno SYCOPHANT Study 3: Design -10

The University of Nevada, Reno SYCOPHANT Study 3: Results ➢ Most users ➢ J48's performance is worse than the Zero-R ➢ XCS outperforms both Zero-R and J48 ➢ External user-context again important for predicting Winamp action preferences ➢ Submitted to Transactions on Evolutionary Computing -11

The University of Nevada, Reno SYCOPHANT Study 3: Winamp Personalization -12

The University of Nevada, Reno SYCOPHANT Study 3: Winamp Summary ➢ Consistent with results from previous two studies ➢ Again, ➢ External user-context important for predicting alarm type preferences ➢ XCS is more robust than J48 ➢ Personalization to individual study participants Thesis Questions ✔ Feasibility ✔ Relevance of user- context ✔ Robustness ✔ Generalizability ✔ participants ✔ applications ✗ term of use Thesis Questions ✔ Feasibility ✔ Relevance of user- context ✔ Robustness ✔ Generalizability ✔ participants ✔ applications ✗ term of use -13

The University of Nevada, Reno SYCOPHANT Study 4: Long-term Google Calendar ➢ Three participants used Google Calendar for 2-4 weeks ➢ 3-7 appointments per day ➢ Collected an average of 75 exemplars ➢ Further substantiate Study-1 (pilot study) which was again a long-term study -14

The University of Nevada, Reno SYCOPHANT Study 4: Results ➢ 2Class ➢ J48 outperforms ZeroR for p1 and p2 ➢ XCS outperforms ZeroR for p2 ➢ Removing user-context degrades J48's performance but not XCS' ➢ XCS outperforms J48 for p2 and p3 ➢ 4Class ➢ user-context important for both XCS and J48 ➢ XCS outperforms J48 -15

The University of Nevada, Reno SYCOPHANT Study 4: Personalization -16

The University of Nevada, Reno SYCOPHANT Study 4: Summary ➢ Results consistent with previous studies ➢ Personalization to individual study participants ➢ Additional data from participants to investigate lower predictive accuracies when compared to earlier studies Thesis Questions ✔ Feasibility ✔ Relevance of user- context ✔ Robustness ✔ Generalizability ✔ participants ✔ applications ✔ term of use Thesis Questions ✔ Feasibility ✔ Relevance of user- context ✔ Robustness ✔ Generalizability ✔ participants ✔ applications ✔ term of use -17

The University of Nevada, Reno SYCOPHANT Main Contributions ➔ Leveraging user-related information (user-context) to address the lack of application personalization ➔ Sycophant's design: modular, flexible, framework with open-source User-context API (C-API) ➔ Study1: 88 percent predictive accuracy on alarm type preferences for three participants Questions Answered ✔ Feasibility ✔ Relevance of user-context ✔ Robustness ✔ Generalizability ✔ participants ✔ applications ✔ term of use Questions Answered ✔ Feasibility ✔ Relevance of user-context ✔ Robustness ✔ Generalizability ✔ participants ✔ applications ✔ term of use Central claim External user-context enables applications to predict user preferred actions thereby leading to better personalization of applications to individual users. Central claim External user-context enables applications to predict user preferred actions thereby leading to better personalization of applications to individual users. -18

The University of Nevada, Reno SYCOPHANT Main Contributions ➔ Study 2: 90 percent predictive accuracy on Google Calendar alarm types for ten participants ➔ Study 3: 80 percent predictive accuracy on Winamp actions for ten participants ➔ Study 4: 65 percent predictive accuracy on Google Calendar's long-term use for three participants ➔ XCS' feasibility for personalizing desktop applications Central claim External user-context enables applications to predict user preferred actions thereby leading to better personalization of applications to individual users. Central claim External user-context enables applications to predict user preferred actions thereby leading to better personalization of applications to individual users. Questions Answered ✔ Feasibility ✔ Relevance of user-context ✔ Robustness ✔ Generalizability ✔ participants ✔ applications ✔ term of use Questions Answered ✔ Feasibility ✔ Relevance of user-context ✔ Robustness ✔ Generalizability ✔ participants ✔ applications ✔ term of use -19

The University of Nevada, Reno SYCOPHANT Outline of Contributions 1.User-context' feasibility to predict application action preferences 2.User-context increases application-action prediction accuracy for different machine learners 3.User studies highlight the generalizability of my approach 4.Sycophant supports multiple desktop applications 5.Personalizing applications over both short-term and long-term application use -20

The University of Nevada, Reno SYCOPHANT Future Work ➢ Conduct additional long-term studies with more participants and user groups ➢ Refine and test C-API on Linux/Mac platforms ➢ Reinforcement learning approaches for application action personalization ➢ Incorporate C-API as a layer above the operating system and provide services for creation of new context sensors and features for application developers ➢ Incorporate advances in community electronics to create new context sensors ➢ Improve accessibility for under-represented user populations -21

The University of Nevada, Reno SYCOPHANT Acknowledgments ➢ Committee members: ➢ Dr. Sushil J. Louis, Dr. Sergiu M. Dascalu, Dr. Ramona Houmanfar, Dr.Monica M. Nicolescu, Dr.Anna K. Panorska ➢ Evolutionary Computing Systems Lab (ECSL) members ➢ Friends and Family ➢ Grants: ➢ ONR grant: N ➢ NSF grant: No

The University of Nevada, Reno SYCOPHANT Thank You! Questions? Sycophant : ➢ Software ➢ Publications

The University of Nevada, Reno SYCOPHANT Study 2: Google Calendar All Participants

The University of Nevada, Reno SYCOPHANT Study 3: Winamp All Participants

The University of Nevada, Reno SYCOPHANT Study 4: Google Calendar All Participants

The University of Nevada, Reno SYCOPHANT Decision Tree Tree structure Nodes, leaf For classifying a case, you traverse the tree until you reach a leaf Example decision-rule: If “Speech-Activity in the last 5 minutes” is greater than 2, Then pause the song Else If “Motion-Activity in the last 5 minutes” is less than or equal to 5, Then play the song Else If “Motion-Activity in the last 5 minute is greater than 6” then play the song Else If “Motion-Activity in the last 5 minutes” is less than or equal to 6, then pause the song LEAF DECISION NODE Speech_getCount5 pause >2 play <=2 motion_getCount5 >5<=5 motion_getCount5 >6<=6 playpause