1. Amazing
The
Re:Search Engine
Jaime Teevan
MIT, CSAIL

2. “Pick a card, any card.”

3. Case 1 Case 2 Case 3 Case 4 Case 5 Case 6
Abracadabra!
Case 1 Case 2 Case 3 Case 4 Case 5 Case 6

4. Your Card is GONE!

5. People Forget a Lot

6. Change Blindness

7. Change Blindness

8. Re:Search Engine ?

9. Merge Old and New Results
Merged
New
Form of personalization

10. We still need magic!

11. Overview Memorability study Recognition study Assumptions
Implementation issues
Evaluation issues
Choose your own adventure

12. Memorability Study Participants issued self-selected query
After an hour, asked to fill out a survey
129 people remembered something

13. Data Analysis Probability of being remembered
Anything? # of words? # of fields?
Features
Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc.
Query features: query type, query length, # of search in session, elapsed time, etc.
Remembered rank v. real rank
Map remembered rank to real rank

14. “Memorability”
Last clicked 4% more likely to be remembered

15. Remembered Results Ranked High

16. Recognition Study Same set-up as Memorability Study
Follow-up survey: Results the same?
Case 1: New results
Case 2: Random 4 same
Case 3: Clicked to top
Case 4: Same results
Case 5: Intelligent merging
165 people completed both steps
19%
38%
41%
66%
81%

17. Assumptions Re-search v. search Memorable v. relevant
Results change v. stay the same
Hide change v. show change
Forget v. remember as forgettable
Merge v. identify old or new
Why?
How to test?
What if I’m wrong?

18. Implementation Issues
Page of cached result may disappear
Multiple result pages
Identifying repeat queries
Exact query may be forgotten
User identified
Search sessions are not repeat queries

19. Evaluation Issues Various goals to test Lab study
Does a merged list look like the original?
Does merging make re-finding easier?
Is search improved overall?
Lab study
How to set up re-finding task?
Timing differences significant enough?
Longitudinal study – What to measure?
What are good baselines?
What are good baselines? Old results, new results, both clearly demarcated,
Timing an issue for lab study
Effect of task on churn or memorability

20. Choose Your Own Adventure
Re-search v. search
Memorable v. relevant
Results change v. stay the same
Hide change v. show change
Forget v. remember as forgettable
Merge v. identify old or new
Implementation issues
Evaluation issues

21. Choose Your Own Adventure
Re-search v. search
Memorable v. relevant
Results change v. stay the same
Hide change v. show change
Forget v. remember as forgettable
Merge v. identify old or new
Implementation issues
Evaluation issues
(Done)

22. Hide Change v. Show Change
Why I think change should be hidden
Example: dynamic menus
How to prove
New results better, called the same or worse
Baseline for testing – 2 lists, change explicit
What if we should show change?
Memorability suggests changes to highlight
Other applications where want to hide change
35 believed to have changed -- 54% same or worse (14% worse)
Lab study – timing an issue
(Done)

23. Memorable v. Relevant Why I think memorability is important
Relevance at a future date is what matters
Necessary to hide change
How to prove
Baseline for lab study with target first
What if relevance is what’s important?
Mapping between memorable and relevant
Useful related work on implicit feedback
Lab study – timing an issue
Do people know what will be relevant in the future when they first see it?
(Done)

24. Re-search v. Search Why I think people repeat searches How to prove
Information seeking literature
Re-finding consistently reported as a problem
How to prove
Study shows prefer to follow known paths
Search log analysis
What if people just want to search?
Memorable results ranked first
Other domains where list consistency matters
Memorable v. relevant – if search, relevant is all that matters
(Done)

25. Merge v. Identify Old and New
Why I think results should be merged
Information need not necessarily one or other
People don’t like to do extra work
How to prove
Search log analysis
Look at what people do in longitudinal study
Lab study – timing becomes an issue
What if people want to identify query type?
Other applications where merging is useful
Approach taken by Google and A9
More passive ways to get an idea of whether they want old or new results – query completion
(Done)

26. Results Change v. Stay the Same
Why I think results change
How search engines work
Personalization and dynamic content
How to prove
Track query results
What if results don’t change?
Probably will in future applications
Existing applications where lists change
(Done)

27. Forget v. Remember as Forgettable
Why I think people forget
Visual analogy
How to prove
Lab study – Do people find new information?
Longitudinal study – Ever click on new result?
What if remember as forgettable?
Build better model of memorability
Highlight important changes
(Done)

28. Implementation Issues
Page of cached result may disappear
Multiple result pages
Identifying repeat queries
User identified
Search sessions are not repeat queries
Exact query may be forgotten
(Done)

29. Evaluation Issues Various goals to test Lab study
Does a merged list look like the original?
Does merging make re-finding easier?
Is search improved overall?
Lab study
How to set up re-finding task?
Timing differences significant enough?
Longitudinal study – What to measure?
What are good baselines?
What are good baselines? Old results, new results, both clearly demarcated,
Timing an issue for lab study
Effect of task on churn or memorability
(Done)

30. Jaime Teevan teevan@mit.edu
Thank you!
Jaime Teevan

31. Strategies for Finding
Teleporting
Orienteering

32. Why Do People Orienteer?
The tools don’t work
Easier than saying what you want
You know where you are
You know what you find

33. Structural Consistency Important
All must be the same to re-find the information!
New name

34. Absolute Consistency Unnecessary
New name
Focus on search result lists

35. Query Changes Most changes are simple What about longer time horizons?
Capitalization
Phrasing
Word ordering
Word form
New queries shorter
What about longer time horizons?
Recognition v. recall

36. Result List Changes Tracked 10 queries on Google for a year+
1.18 of top 10 disappear each week
Rate of change likely to increase, e.g.:
Raw personalization
Relevance feedback
People forget their queries
28% of queries forgotten within an hour

37. Example: “neon signs”