SUNIL GAHLAWAT LU GAN MIYA SYLVESTER YIRAN WANG Usability and Utility of TopicLens a Visualization System for the Exploration of Topic Models

Topic Models Topic models provide a simple way to analyze large volumes of unlabeled text.

Topic Models A "topic" consists of a cluster of words that frequently occur together.

Topic Models An example of Topic Modeling : Twitter tweets search for "Hollywood"

Why develop these kinds of Models? Organizing and finding patterns in data has become important in the sciences, humanities, industry, and culture. Topic models can be built for images, social networks, music, purchase histories, computer code, genetic data, click-through-data...

TopicLens Interface

Goals Get a good understanding whether the user editing function in TopicLens is useful or necessary. Identify and classify any usability defects. Study whether the design elements and aesthetics enhance the Usablity of the program. Evaluate User satisfaction/User experience in terms of ease of learning, retention, efficiency, and reliability.

A Detailed Summary of Usability Study Primary focus on User Testing  Initial Heuristic Analysis  Short Survey and Interview with Participants 6~10 Participants  Two sub-groups to evaluate different usage behavior  3~5 Students, 3~5 Non-students  Recruited from grad student housing, university center, etc. Recording: Audio and Screen Video Capture Time: 30~40 minutes Location: Donald Bren Hall

Part1: First Impression on Visual Presentation  We will briefly explain about TopicLens to the participant  Participants will be asked to think aloud as they experiment with the interface.  Editing Mode: OFF A Detailed Summary of Usability Study

Part2: Details on User Editing Functionality  We will give participants a few scenarios to explore all the functions available on TopicLens (merging/filtering/hiding) and answer any questions the participant may have about these functions.  Bill is Professor at a research university who interested in software/hardware and security. Bill will be using the merging function of TopicLens to visually organize his topics.  Hazel is a movie buff and she wants to see topics related to all movies. She’ll be filtering function to find the topics related to movies. Hazel has filtered her movie topics but there are still too many for her to sort through. She will use the hiding function to better suit her needs.  Participants will be asked to think aloud as they complete the tasks.  Editing Mode: ON A Detailed Summary of Usability Study

Part3: Survey and Interview  Survey:  Participants will be asked to complete a short survey (adapted from System Usability Scale) on the functionality of TopicLens. The following are examples participants would rate on a scale of 1(strongly disagree) to 5(strongly agree): I thought the system was easy to use. I found the various functions in this system were well integrated. I found the system unnecessarily complex. A Detailed Summary of Usability Study

 Interview:  We will also ask Participants open-ended questions that focus on editing: Was it easy to understand the purpose of merging/hiding/filtering the topics or not? After the editing, does the visual layout of the topics make more or less sense to you? Right now, once you merge two topics, the nodes associated are either related to one or both topics. Do you prefer this, or do you prefer to see topics that are only related to both topics you merged? (For Participants who like the editing function, we will ask detailed questions about the AND/OR Boolean for two or more topics.) Would you use the editing function or not? Would you recommend the editing function to a friend or not? A Detailed Summary of Usability Study

Plans to analyze usability test data Qualitative "think aloud" data  We will play back the audio recording of each participant during their period of experimenting the system and transcribe those that we find most useful.  We will discuss any emergent patterns of user intentions, frustrations, and opinions that were expressed by Participants as a group. Survey results  We will use Excel to perform basic analysis of the rating scales  We will also compare and contrast the ratings and their open-ended comments. Open-ended interview question  We will transcribe the audio-taped open comments of each participant.  We will use content analysis on the open comments.

Challenges Environment configuration Finding participants Striving for consistency  Instructions  Interview questions  Testing situation “Hard” participants Transforming data into findings and practical recommendations

Timeline WEEKPlan Peopl e Week 3~4Identify topics for questions and user sub-population profiles. Group Week 4Create an activity proposalGroup Week 4~5 Word the questions appropriately Group Distribute the questions to customer for reviewYiran Find the SUS for the questionnaireYiran Week 5 Identify and recruit participants for our studyGroup Determine the roles for our Usability TestGroup Prepare interview materials including documentation for confidentiality, consent formsMiya Acquire location in Donald Bren HallMark Acquire incentives (starbucks, coupons, etc.)Sunil Week 6Conduct pilot test, make necessary changes to questions and procedure based on pilotGroup Week 7~8 Set up location with all materials necessary for activityMark Call and confirm with participants and begin Usability StudiesMiya Week 8~10Analyze the dataGroup Week 11Submit the final reportGroup

Drawbacks of TopicLens Observed Font size is not adjustable The background colors of program boxes sometimes make the label unreadable Range slider design could be better Titles of topics overlap when dragged together Reset/return button missing