The Perfect Search Engine Is Not Enough Jaime Teevan, MIT with Christine Alvarado, Mark Ackerman and David Karger

Let Me Interview You! Web: Email: Files: What’s the last Web page you visited? How did you get there? Have you looked for anything on the Web? Email: What’s the last email you read? What did you do with it? Have you gone back to an email you’ve read before? Files: What’s the last file you looked at? How did you get to it? Have you looked for a file?

Overview: Understanding Search Directed Introduction Related work Methodology What we learned How? Why? Who? So what? Prefer to search in steps Because it’s easier Step size varies by person

Haystack: Personal Information Storage Email Web pages Haystack Lots of separate info, Haystack stores in central repository. Easy to separate info from its form, easy to connect related info. A lot of people in our group are working on getting information into the Haystack. Talk will be about a study we did to understand how information should be gotten out of the Haystack. Perhaps worth noting: What we found is not what we were expecting. Does not just reinforce the direction we were going with Haystack but has modified the direction. Files Calendar Contacts

Directed Search in Haystack What was that paper I read last week about Information Retrieval? Haystack

Directed Search in Haystack Ah yes! Thank you. Haystack

…Or Elsewhere Ah yes! Thank you. “Perfect Search Engine”

Related Work Directed search Observational studies [Malone83] Lab studies [Capra03, Maglio97] Log analysis [Broder02, Spink01] Observational studies [Malone83] Information Seeking Marchionini, O’Day and Jeffries, Bates, Belkin, … Evolving information need

Modified Diary Study Subjects: 15 CS graduate students Ten interviews each (2/day x 5 days) Two question types Last email/file/Web page looked at Last email/file/Web page looked for Supplemented with direct observation and an hour-long semi-structured interview

Overview: Understanding Directed Search Introduction Related work Methodology What we learned How? Why? Who? So what?

Directed Search Today Target: Connie Monroe’s office number  Type into a search engine: “Connie Monroe, office number”

What We Observed Interviewer: Have you looked for anything on the Web today? Jim: I had to look for the office number of the Harvard professor. I: So how did you go about doing that? J: I went to the homepage of the Math department at Harvard

What We Observed I: So you went to the Math department, and then what did you do over there? J: It had a place where you can find people and I went to that page and they had a dropdown list of visiting faculty, and so I went to that link and I looked for her name and there it was.

What We Observed J: I knew that she had a very small Web page saying, “I’m here at Harvard. Here’s my contact information.”

Strategies Looking for Information Teleporting Orienteering

Why Do People Orienteer? The tools don’t work Easier than saying what you want You know where you are You know what you find

Easier Than Saying What You Want Describing the target is hard Can’t Prefer not to Habit “Whichever way I remember first.” Search for source E.g., Your last email search

Stay in known space Backtracking You Know Where You Are URL manipulation Bookmarks History Backtracking Following an information scent Never end up at a dead end

You Know What You Find Context gives understanding of answer “I was looking for a specific file. But even when I saw its name, I wouldn’t have known that that was the file I wanted until I saw all of the other names in the same directory…” Understanding negative results “I basically clicked on every single button until I was convinced… I don’t think that it exists…”

Individual Strategies Search strategies varied by individual People who pile information take small steps People who file information take big steps Where was the last email you found? Inbox? Elsewhere?

File or Pile Email Filer Piler

How Individuals Search For Files Filers Big steps Irony: Those people who had their information well classified don’t take advantage of their classification to get to their information While people who had their information all in a jumble wanted to sort through that jumble. Similar on the Web. Pilers were more likely to navigate to a specific site and do site search, an orienteering behavior While filers used a general purpose search engine like Google more often. Pilers Small steps

Applying What We Learned  Support orienteering Advantages to orienteering Easier than saying what you want You know where you are You know what you find Individual differences in step size Meta-info, source, flag sources with info URL manipulation, paths apparent, all steps Answer context, trusted sources, exhaustive Allow for different step sizes

Structural Consistency Important All must be the same to re-find the information!

Preserve What User Remembers Supports orienteering for re-finding Allows access to new information

More to Learn from the Data Differences in finding v. re-finding How organization relates to search Importance of type (email, files and Web) Looked at v. looked for  Keep in mind population

Questions? Teevan, J., Alvarado, C., Ackerman, M. S. and Karger, D. R. (2004). The Perfect Search Engine is Not Enough: A Study of Orienteering Behavior in Directed Search. To appear in Proceedings of CHI 2004. (Linked from http://www.teevan.org) Questions.

Relating How and What People only keyword search 39% of the time Specific General Document Other 47 19 41 Keyword 34 23 17 People only keyword search 39% of the time What people look for related to how they look Now we understand how people search and what they search for Even looking at a subset of the orienteering behavior, we see people orienteer a lot For example, we see people orienteer to specific documents all the time Not surprising – think how you would get to, for example, a specific email or a specific file We see it’s a little more likely that a person would teleport when looking for general information But while you might expect that trend to continue for specific information, instead we see that people orienteer a lot, too! Weird. Why? Surprise: Orienteer to specific information

Relating How and Corpus Email Files Web Other 59 42 19 Keyword 06 10 64 Email and files: Almost never keyword searched Easy to associate information with document Web: Used keyword search much more often Additional information I don’t have time to talk about … Available if there are questions. We saw that how people searched relates to what they were searching for This table shows how the corpus people were searching in related to how they searched Email and files almost always orienteered Most email searches were for specific information Most file searches were for documents Also, easy to associate information with the document in these cases, because they knew the information On the Web we saw more teleporting Although keyword search is easy on Web, so people may still be orienteering a lot, it’s just getting classified as orienteering

Relating What and Corpus Email Files Web Specific 39 7 33 General 10 30 Document 08 35 14 Additional information I don’t have time to talk about … Available if there are questions. Email searches were primarily for specific information File searches were primarily for documents Web searches were more evenly distributed