Intelligent User Interface Research at Texas A&M University: Designing Adaptive Systems to Support Information Triage Frank Shipman Associate Director, Center for the Study of Digital Libraries Professor, Department of Computer Science Texas A&M University

TAMU IUI Group Post-doc: J. Michael Moore Ph.D. students: Soonil Bae, Dan Corlette, James Creel, DoHyoung Kim, Konstantinos Meintanis, Chris Udompanyanan, Anna Zacchi M.S. students: Christopher King, Austin Riddle Alumni (Ph.D.): Luis Francisco-Revilla, Univ. of Texas Haowei Hsieh, Univ. of Iowa In collaboration with Cathy Marshall

Current Projects Design Exploration –Getting more end-user participation in software design –Extending to computer-assisted models of collaborative design Personal & Group Collection Management –Interpreting and visualizing records of user activity –Combining content analysis, metadata analysis and personal interpretation to provide system assistance Information Triage –Combining human-authored and system-generated visualizations –Spatial expression and its value in activity management Newest Projects –Effects of opinion expression on social networks –Workspaces for the freeform analysis of time-series and other scientific data

Current Projects Design Exploration –Getting more end-user participation in software design –Extending to computer-assisted models of collaborative design Personal & Group Collection Management –Interpreting and visualizing records of user activity –Combining content analysis, metadata analysis and personal interpretation to provide system assistance Information Triage –Combining human-authored and system-generated visualizations –Spatial expression and its value in activity management Newest Projects –Effects of opinion expression on social networks –Workspaces for the freeform analysis of time-series and other scientific data

Design Exploration Problem –Users have knowledge needed for the design of effective software but have a difficult time communicating this knowledge Prior solutions –Participatory design, ethnography – rich information but cannot include many users –Surveys and questionnaires – can reach many users but provides limited information Goal: Find an in-between option for software designers to collect input

Design Exploration Provide opportunity for users to express their understanding through combination of GUI/graphical design and textual argumentation Support software designers use of annotated partial designs through search and navigation options End Users Software Designers

Study of End User Expression 75 students provided input on design of software for locating housing in College Station, TX Three conditions: text only, graphical design only, and annotated graphical designs Outcomes –design groups much more engaged in activity –combined group provided more text than text- only group –more detailed information on concepts and processes in design groups than in text group

Example of Annotated Design

Design Exploration Analyzer Supports software designer in accessing end user expression Indexing of designs based on –Text analysis (dictionary, term vector comparison) –Spatial analysis (widget list and composite recognition) Supports navigation and search by terms used and by clustered designs and design components

Design Exploration Analyzer

Design Exploration Directions Study of DE Analyzer –Advanced computer science students with CHI and SE experience used DE Analyzer Moving to On-Line Process –Browser-based DE Builder puts annotated designs in on-line database –Blurring the End-User / Software Designer distinction in the process to support collaborative design processes

Current Projects Design Exploration –Getting more end-user participation in software design –Extending to computer-assisted models of collaborative design Personal & Group Collection Management –Interpreting and visualizing records of user activity –Combining content analysis, metadata analysis and personal interpretation to provide system assistance Information Triage –Combining human-authored and system-generated visualizations –Spatial expression and its value in activity management Newest Projects –Effects of opinion expression on social networks –Workspaces for the freeform analysis of time-series and other scientific data

Information Triage The practice of quickly determining the merit and disposition of relevant content in a collection of data and documents Common aspects –Selection from information repository via querying and browsing interfaces –Extensive, hyper-extensive and intensive reading and viewing in reading interfaces –Collection and interpretation of resources in organization interface –Mode switching / attention shifting

Information Work Variety of information tasks –Short-term: Facts and references What is the escape velocity? –Long-term: Analysis and synthesis How to design a space craft? For longer-term information activities the work really begins after potentially relevant materials are located.

Document Triage We want to look at situations where people are reading more than one document at once Document triage places different demands on attention than single- document reading activities Continuum of types of reading: working in overview (metadata), reading at various levels of depth (skimming), reading intensively

Spatial Hypertext and Document Triage Spatial hypertext – where inter-document relationships are expressed via visual and spatial cues rather than links. Earlier study compared use of two variations of the VIKI spatial hypertext system with paper [Marshall, Shipman 1997] Results showed that –people use the affordances of the medium provided –those working with paper read more –those working with VIKI organized more

Visual Knowledge Builder (VKB) VKB is a second generation spatial hypertext –greater support for collaborative and long-term tasks –navigable history –explicit (as well as implicit) links VKB provides: –A hierarchy of two-dimensional workspaces called collections for placing information –Easy manipulation of visual properties of information –Information objects pointing to external content –Attribute/value pairs for attaching metadata –Integrated search for Google and NSDL

Study of VKB Use for Selecting and Organizing Materials Study designed to understand how spatial hypertext would change work practices when accessing a digital library. Decided to look at document triage –deciding what to keep –expressing an initial view of relationships

Task: 16 subjects placed in role of a reference librarian, selecting and organizing information on ethnomathematics for a teacher Setting: top 20 search results from NSDL & top 20 search results from Google 16 subjects were divided into two groups of 8: * Initial search done by us Subjects given as much time as they deemed necessary (after training for VKB users) Study Setup VKB (VKB/IE)Control (IE/Editor) Search *VKBIE ReadingIE OrganizationVKBEditor

Some of the Results Document Triage Recap: –metadata visualization caused users to avoid changing visual features of objects –people interleave the reading of multiple documents –people switch between reading and organizing –transitions generate potential for breakdown Question: Can a dedicated reading surface make a difference in how people engage with content during triage?

A second look at the earlier data… Subject ID Total time1:04:080:54:140:21:590:22:481:33:281:20:091:03:481:01:43 8 Number of transitions Overview in VKB/Content displayed in IE, and transitions between the two Average number of transitions between applications: 88 VKBIE Total time (seconds) 18,8747,596 71%29% Average time looking at window (seconds) 4720 Summary Average time on the task (minutes): 58 % total time (in app) more than 2/3 of the time is spent organizing references time spent reading unfamiliar material is very brief window management is extremely time-consuming!

Document Triage—Starting Point Given reduced representations of multiple relevant documents (e.g. a list of search results or headers), people don’t spend much time reading (or even skimming) –Lots of time is spent managing screen space and windows (opening, closing, reshaping, etc.) – might people be trying to minimize that? When people are overwhelmed in this way, there’s a tendency to work from metadata instead of content, manipulating and organizing it –Think about how we handle our (especially spam, but others as well) –Think about how we decide to follow a link from a list of search results (perhaps using poor or deceptive metadata) We’d like to give people a chance to read more, focus their attention, and spend less time managing windows: what happens if we give readers a dedicated reading surface like a tablet computer?

Study Configurations Display Configuration Input DevicesAssignment of Activity Laptop and tabletop LCD display Extended desktop controlled via keyboard and mouse User controls which windows are on which display Laptop and projected display Extended desktop controlled via keyboard and mouse User controls which windows are on which display Tablet computer and projected display Projected display controlled via keyboard and mouse, tablet computer controlled via pen Software assigns document overview to projected display and IE to tablet

A First Look at the New Data Configuration Prior StudyCurrent Study Desktop PC Laptop & LCD Screen Laptop & Projected Display Tablet PC & Projected Display # of Displays1222 Avg. Total Time3,3093,5543,6424,234 Avg. Time (VKB)2,3592,4532,6273,005 Avg. Time (IE)9501,1021,0151,229 Avg. # of Transitions (shifts of focus)

Time Spent in IE (glancing, skimming, reading) One DisplayTwo Display tendency: in the 2 display condition, there’s a greater number of brief encounters; might represent more glances, more checking more revisits?

Display-Oriented Results Multiple displays facilitated many more transitions between reading and organizing. Subjects with laptop and extra screen felt most comfortable of the multiple display configurations Preference for multiple displays at the same focal length Subjects found size of projected display and pen interactions annoying

User Actions Anticipate Document Assessment After ethnomathematics task, subjects were asked to identify: –5 documents they found most valuable –5 documents they found least valuable Correlated actions (p <.01) (from most to least correlated) –Number of object moves –Scroll offset –Number of scrolls –Number of border color changes –Number of object resizes –Total number of scroll groups –Number of scrolling direction changes –Number of background color changes –Time spent in document –Number of border width changes –Number of object deletions –Number of document accesses –Length of document in characters

Multi-Application Interest Models for Document Triage Central problem in document triage is limited time –VKB enables rapid expression of human assessment using visual cues –Goal: have system aid in selecting documents by providing visual cues that will aid in the selection of further documents Why Multi-Application Interest Models? –Users tend not to provide explicit feedback –A variety of implicit information has been used, including reading time, scrolling and mouse events, and annotations Problem: Individuals vary greatly due to idiosyncratic work practices –Information across applications may combine to be a better predictor for a larger population.

Process for Providing Support 1.Recognize user interest in and interpretation of documents 2.Generate a representation of user interests 3.Identify documents that match these interests 4.Provide visual cues to indicate the potential value of documents

Acquiring User Interest Model Explicit Methods –users tend not to provide explicit feedback Implicit Methods –Reading time has been used in many cases –Scrolling and mouse events have been shown somewhat predictive –Annotations have been used to identify passages of interest Problem: Individuals vary greatly and have idiosyncratic work practices

Models based on Reading and Interpretation Document triage combines multiple forms of reading and interpretation Infrastructure for applications to construct and share interest models Location/Overview Application Organizing Application Reading Application User Interest Estimation Engine Reading Application Reading Application Interest Profile Manager Interest Profile

Interest Models Based on data from an earlier study, we developed four interest models –Three were mathematically derived Reading-Activity Model Organizing-Activity Model Combined Model –One hand-tuned model included human assessment based on observations of user activity and interviews with users.

Quick Comparison of Models How much of difference in original data was modeled? –Reading-activity model 47.7% –Organizing-activity model63.6% –Combined model70.8% How well would models do for new data?

Evaluation of Models 16 Subjects with same: –Task (collecting information on ethnomathmatics for teacher) and –Setting (20 NSDL and 20 Google results) Different display configuration –Using a single display in this case where used two displays before Different rating of documents –Subjects rated all documents on a 5-point Likert scale (with 1 meaning “not useful” and 5 meaning “very useful”)

Predictive Power of Models Models were conservative due to data from original study. Used aggregated user activity and user evaluations to evaluate models Model Avg. Residue Std. Dev. Reading-activity model Organizing-activity model Combined model Hand-tuned model

Size of Errors

Current Activities Active Support for Information Triage –Representation of user interests –Recognition and visualization of content matching interests VKB 3 –Separation of human and system visual expression –Automatic interpretation and visualization of user activity to support collaboration

Contact Information Frank Shipman Download VKB 2 from: