Jaime Teevan MIT, CSAIL The Re:Search Engine

“Pick a card, any card.”

Case 1Case 2Case 3Case 4Case 5Case 6

Your Card is GONE!

People Forget a Lot

Change Blindness

Change Blindness

Re:Search Engine ?

Merge Old and New Results Old New Merged

We still need magic!

Overview ♠Memorability study ♠Recognition study ♠Assumptions ♠Implementation issues ♠Evaluation issues ♠Choose your own adventure

Memorability Study ♠Participants issued self-selected query ♠After an hour, asked to fill out a survey ♠129 people remembered something

Data Analysis Probability of being remembered ♥Anything? # of words? # of fields? ♥Features ♣Result features: clicked, not clicked, last clicked, rank, dwell time, frequency of visit, etc. ♣Query features: query type, query length, # of search in session, elapsed time, etc. ♠Remembered rank v. real rank ♥Map remembered rank to real rank

“Memorability”

Remembered Results Ranked High

Recognition Study ♠Same set-up as Memorability Study ♠Follow-up survey: Results the same? ♥Case 1: Old results ♥Case 2: New results ♥Case 3: Clicked to top ♥Case 4: Intelligent merging ♠92 people have completed both steps 16% 74% 65% 17%

Assumptions ♠Re-search v. search ♠Memorable v. relevant ♠Results change v. stay the same ♠Hide change v. show change ♠Forget v. remember as forgettable ♠Merge v. identify old or new Why?How to test?What if I’m wrong?

Implementation Issues ♠Page of cached result may disappear ♠Multiple result pages ♠Identifying repeat queries ♥User identified ♥Search sessions are not repeat queries ♥Exact query may be forgotten

Evaluation Issues ♠Various goals to test ♥Does a merged list look like the original? ♥Does merging make re-finding easier? ♥Is search improved overall? ♠Lab study ♥How to set up re-finding task? ♥Timing differences significant enough? ♠Longitudinal study – What to measure? ♠What are good baselines?

Choose Your Own Adventure ♠Re-search v. search ♠Memorable v. relevant ♠Results change v. stay the same ♠Hide change v. show change ♠Forget v. remember as forgettable ♠Merge v. identify old or new ♠Implementation issues ♠Evaluation issues

Choose Your Own Adventure ♠Re-search v. searchRe-search v. search ♠Memorable v. relevantMemorable v. relevant ♠Results change v. stay the sameResults change v. stay the same ♠Hide change v. show changeHide change v. show change ♠Forget v. remember as forgettableForget v. remember as forgettable ♠Merge v. identify old or newMerge v. identify old or new ♠Implementation issuesImplementation issues ♠Evaluation issuesEvaluation issues (Done)Done

Hide Change v. Show Change ♠Why I think change should be hidden ♥Example: dynamic menus ♠How to prove ♥New results better, called the same or worse ♥Baseline for testing – 2 lists, change explicit ♠What if we should show change? ♥Memorability suggests changes to highlight ♥Other applications where want to hide change (Done)Done

Memorable v. Relevant ♠Why I think memorability is important ♥Relevance at a future date is what matters ♥Necessary to hide changehide change ♠How to prove ♥Baseline for lab study with target first ♠What if relevance is what’s important? ♥Mapping between memorable and relevant ♥Useful related work on implicit feedback (Done)Done

Re-search v. Search ♠Why I think people repeat searches ♥Information seeking literature ♥Re-finding consistently reported as a problem ♠How to prove ♥Study shows prefer to follow known paths ♥Search log analysis ♠What if people just want to search? ♥Memorable results ranked first ♥Other domains where list consistency matters (Done)Done

Merge v. Identify Old and New ♠Why I think results should be merged ♥Information need not necessarily one or other ♥People don’t like to do extra work ♠How to prove ♥Search log analysis ♥Look at what people do in longitudinal study ♥Lab study – timing becomes an issue ♠What if people want to identify query type? ♥Other applications where merging is useful (Done)Done

Results Change v. Stay the Same ♠Why I think results change ♥How search engines work ♥Personalization and dynamic content ♠How to prove ♥Track query results ♠What if results don’t change? ♥Probably will in future applications ♥Existing applications where lists change (Done)Done

Forget v. Remember as Forgettable ♠Why I think people forget ♥Visual analogy ♠How to prove ♥Lab study – Do people find new information? ♥Longitudinal study – Ever click on new result? ♠What if remember as forgettable? ♥Build better model of memorability ♥Highlight important changes (Done)Done

Implementation Issues ♠Page of cached result may disappear ♠Multiple result pages ♠Identifying repeat queries ♥User identified ♥Search sessions are not repeat queries ♥Exact query may be forgotten (Done)Done

Evaluation Issues ♠Various goals to test ♥Does a merged list look like the original? ♥Does merging make re-finding easier? ♥Is search improved overall? ♠Lab study ♥How to set up re-finding task? ♥Timing differences significant enough? ♠Longitudinal study – What to measure? ♠What are good baselines? (Done)Done

Jaime Teevan

Teleporting Orienteering Strategies for Finding

Why Do People Orienteer? ♠Easier than saying what you want ♠You know where you are ♠You know what you find ♠The tools don’t work

All must be the same to re-find the information! Structural Consistency Important New name

Absolute Consistency Unnecessary New name Focus on search result lists

Query Changes ♠Most changes are simple ♥Capitalization ♥Phrasing ♥Word ordering ♥Word form ♥New queries shorter ♠What about longer time horizons? ♠Recognition v. recall

Result List Changes ♠Tracked 10 queries on Google for a year+ ♠1.18 of top 10 disappear each week ♠Rate of change likely to increase, e.g.: ♥Raw personalization ♥Relevance feedback ♠People forget their queries ♥28% of queries forgotten within an hour

Example: “neon signs”