1. Jaime Teevan MIT, CSAIL The Re:Search Engine

2. “Pick a card, any card.”

3. Case 1Case 2Case 3Case 4Case 5Case 6

4. Your Card is GONE!

5. People Forget a Lot

6. Change Blindness http://www.usd.edu/psyc301/ChangeBlindness.htm

7. Change Blindness http://www.usd.edu/psyc301/ChangeBlindness.htm

8. Re:Search Engine ?

9. Merge Old and New Results Old New Merged

10. We still need magic!

11. Overview ♠Memorability study ♠Recognition study ♠Assumptions ♠Implementation issues ♠Evaluation issues ♠Choose your own adventure

12. Memorability Study ♠Participants issued self-selected query ♠After an hour, asked to fill out a survey ♠129 people remembered something

13. Data Analysis Probability of being remembered ♥Anything? # of words? # of fields? ♥Features ♣Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc. ♣Query features: query type, query length, # of search in session, elapsed time, etc. ♠Remembered rank v. real rank ♥Map remembered rank to real rank

14. “Memorability”

15. Remembered Results Ranked High

16. Recognition Study ♠Same set-up as Memorability Study ♠Follow-up survey: Results the same? ♥Case 1: Old results ♥Case 2: New results ♥Case 3: Clicked to top ♥Case 4: Intelligent merging ♠92 people have completed both steps 16% 74% 65% 17%

17. Assumptions ♠Re-search v. search ♠Memorable v. relevant ♠Results change v. stay the same ♠Hide change v. show change ♠Forget v. remember as forgettable ♠Merge v. identify old or new Why?How to test?What if I’m wrong?

18. Implementation Issues ♠Page of cached result may disappear ♠Multiple result pages ♠Identifying repeat queries ♥User identified ♥Search sessions are not repeat queries ♥Exact query may be forgotten

19. Evaluation Issues ♠Various goals to test ♥Does a merged list look like the original? ♥Does merging make re-finding easier? ♥Is search improved overall? ♠Lab study ♥How to set up re-finding task? ♥Timing differences significant enough? ♠Longitudinal study – What to measure? ♠What are good baselines?

20. Choose Your Own Adventure ♠Re-search v. search ♠Memorable v. relevant ♠Results change v. stay the same ♠Hide change v. show change ♠Forget v. remember as forgettable ♠Merge v. identify old or new ♠Implementation issues ♠Evaluation issues

21. Choose Your Own Adventure ♠Re-search v. searchRe-search v. search ♠Memorable v. relevantMemorable v. relevant ♠Results change v. stay the sameResults change v. stay the same ♠Hide change v. show changeHide change v. show change ♠Forget v. remember as forgettableForget v. remember as forgettable ♠Merge v. identify old or newMerge v. identify old or new ♠Implementation issuesImplementation issues ♠Evaluation issuesEvaluation issues (Done)Done

22. Hide Change v. Show Change ♠Why I think change should be hidden ♥Example: dynamic menus ♠How to prove ♥New results better, called the same or worse ♥Baseline for testing – 2 lists, change explicit ♠What if we should show change? ♥Memorability suggests changes to highlight ♥Other applications where want to hide change (Done)Done

23. Memorable v. Relevant ♠Why I think memorability is important ♥Relevance at a future date is what matters ♥Necessary to hide changehide change ♠How to prove ♥Baseline for lab study with target first ♠What if relevance is what’s important? ♥Mapping between memorable and relevant ♥Useful related work on implicit feedback (Done)Done

24. Re-search v. Search ♠Why I think people repeat searches ♥Information seeking literature ♥Re-finding consistently reported as a problem ♠How to prove ♥Study shows prefer to follow known paths ♥Search log analysis ♠What if people just want to search? ♥Memorable results ranked first ♥Other domains where list consistency matters (Done)Done

25. Merge v. Identify Old and New ♠Why I think results should be merged ♥Information need not necessarily one or other ♥People don’t like to do extra work ♠How to prove ♥Search log analysis ♥Look at what people do in longitudinal study ♥Lab study – timing becomes an issue ♠What if people want to identify query type? ♥Other applications where merging is useful (Done)Done

26. Results Change v. Stay the Same ♠Why I think results change ♥How search engines work ♥Personalization and dynamic content ♠How to prove ♥Track query results ♠What if results don’t change? ♥Probably will in future applications ♥Existing applications where lists change (Done)Done

27. Forget v. Remember as Forgettable ♠Why I think people forget ♥Visual analogy ♠How to prove ♥Lab study – Do people find new information? ♥Longitudinal study – Ever click on new result? ♠What if remember as forgettable? ♥Build better model of memorability ♥Highlight important changes (Done)Done

28. Implementation Issues ♠Page of cached result may disappear ♠Multiple result pages ♠Identifying repeat queries ♥User identified ♥Search sessions are not repeat queries ♥Exact query may be forgotten (Done)Done

29. Evaluation Issues ♠Various goals to test ♥Does a merged list look like the original? ♥Does merging make re-finding easier? ♥Is search improved overall? ♠Lab study ♥How to set up re-finding task? ♥Timing differences significant enough? ♠Longitudinal study – What to measure? ♠What are good baselines? (Done)Done

30. Jaime Teevan teevan@mit.edu

31. Teleporting Orienteering Strategies for Finding

32. Why Do People Orienteer? ♠Easier than saying what you want ♠You know where you are ♠You know what you find ♠The tools don’t work

33. All must be the same to re-find the information! Structural Consistency Important New name

34. Absolute Consistency Unnecessary New name Focus on search result lists

35. Query Changes ♠Most changes are simple ♥Capitalization ♥Phrasing ♥Word ordering ♥Word form ♥New queries shorter ♠What about longer time horizons? ♠Recognition v. recall

36. Result List Changes ♠Tracked 10 queries on Google for a year+ ♠1.18 of top 10 disappear each week ♠Rate of change likely to increase, e.g.: ♥Raw personalization ♥Relevance feedback ♠People forget their queries ♥28% of queries forgotten within an hour

37. Example: “neon signs”