ASSOCIATIVE BROWSING Evaluating 1 Jin Y. Kim / W. Bruce Croft / David Smith by Simulation

* What do you remember about your documents? 2 Registration James Use search if you recall keywords!

* What if keyword search is not enough? 3 Registration Associative browsing to the rescue!

* Probabilistic User Modeling Query generation model Term selection from a target document [Kim&Croft09] State transition model Use browsing when result looks marginally relevant Link selection model Click on browsing suggestions based on perceived relevance 4

* Simulating Interaction using Probabilistic User Model 5 Initial Query : James Registration Marginally Relevant (11 < Rank D < 50 ) Not Relevant (Rank D > 50 ) Reformulated Query : Two Dollar Registration Search Click On a Result : 1. Two Dollar Regist… End Target Doc at Top 10 Target Doc at Top 10 Target Doc :

* A User Model for Link Selection User’s browsing behavior [Smucker&Allan06] Fan-out 1~3: the number of clicks per ranked list BFS vs. DFS : the order in which documents are visited

* A User Model for Link Selection User’s level of knowledge Random : randomly click on a ranked list Informed : more likely to click on more relevant item Oracle : always click on the most relevant item Relevance estimated using the position of target item

* Evaluation Results Simulated interaction was generated using CS collection 63,260 known-item finding sessions in total The Value of Browsing Browsing was used in 15% of all sessions Browsing saved 42% of sessions when used Comparison with User Study Results Roughly matches in terms of overall usage and success ratio Evaluation Type TotalBrowsing used Successful Simulation63,2609,410 (14.8%)3,957 (42.0%) User Study29042 (14.5%)15 (35.7%)

* Evaluation Results Success Ratio of Browsing More Exploration

* Summary 10 Associative Browsing ModelEvaluation by Simulation Any Questions? Jin Y. Kim / W. Bruce Croft / David Smith Simulated evaluation showed very similar statistics to user study in when and how successfully associative browsing is used Simulated evaluation reveals a subtle interaction between the level of knowledge and the degree of exploration

* Simulation of Know-item Finding using Memory Model Build the model of user’s memory Model how the memory degrades over time Generate search and browsing behavior on the model Query-term selection from the memory model Use information scent to guide browsing choices [Pirolli, Fu, Chi] Update the memory model during the interaction New terms and associations are learned 11 t1t1 t1t1 t2t2 t2t2 t3t3 t3t3 t4t4 t4t4 t5t5 t5t5 t3t3 t3t3

O PTIONAL S LIDES 12

* Evaluation Results Lengths of Successful Sessions

* Summary of Previous Evaluation User study by DocTrack Game [Kim&Croft11] Collect public documents in UMass CS department Build a web interface by which participants can find documents Department people were asked to join and compete Limitations Fixed collection, with a small set of target tasks Hard to evaluate with varying system parameters Simulated Evaluation as a Solution Build a model of user behavior Generate simulated interaction logs 14 If search accuracy improves by X%, how will it affect user behavior? How would its effectiveness vary for diverse groups of users?

* Building the Associative Browsing Model Concept Extraction 3. Link Extraction 4. Link Refinement 1. Document Collection Term Similarity Temporal Similarity Co-occurrence

* DocTrack Game 16

* Community Efforts based on the Datasets 17