1. An Interdisciplinary Perspective on IR Susan Dumais Microsoft Research SIGIR 2009 Salton Award Lecture

2. Thanks! Salton Award Committee Salton Award Committee Many great colleagues Many great colleagues 1979-1997, Bell Labs/Bellcore 1979-1997, Bell Labs/Bellcore 1997-present, Microsoft Research 1997-present, Microsoft Research Many other collaborators … Many other collaborators … Tremendous honor Tremendous honor Salton number = 2 or 3 Salton number = 2 or 3 Michael Lesk: SMART @ Harvard (early 1960’s) Michael Lesk: SMART @ Harvard (early 1960’s) CHI 1995 Panel: “Searching & browsing: Can we find a synergy”? CHI 1995 Panel: “Searching & browsing: Can we find a synergy”? SIGIR 2009

3. Overview Personal reflections Personal reflections My research is interdisciplinary, at the intersection of IR and HCI My research is interdisciplinary, at the intersection of IR and HCI User-centric vs. system-centric User-centric vs. system-centric Empirical vs. theoretical Empirical vs. theoretical Evaluation via many methods Evaluation via many methods Test collections, field work, prototypes, deployment experiences, lab studies, etc. Test collections, field work, prototypes, deployment experiences, lab studies, etc. My background My background Common themes Common themes Understanding user, domain, and task contexts Understanding user, domain, and task contexts Future challenges Future challenges Dynamics, data and more Dynamics, data and more SIGIR 2009

4. Background Mathematics and Psychology Mathematics and Psychology HCI group at Bell Labs, 1979 HCI group at Bell Labs, 1979 Introduction to IR, 1980-82 Introduction to IR, 1980-82 The problem(s) … The problem(s) … Human factors in database access Human factors in database access Describing categories of objects for menu retrieval Describing categories of objects for menu retrieval Verbal disagreement/Statistical semantics/Vocabulary problem Verbal disagreement/Statistical semantics/Vocabulary problem Some solutions & applications … Some solutions & applications … Rich aliasing / Adaptive indexing / Latent semantic indexing Rich aliasing / Adaptive indexing / Latent semantic indexing Closing the loop back to psychology … Closing the loop back to psychology … A solution to Plato’s problem [Psychological Review, 1997] A solution to Plato’s problem [Psychological Review, 1997] SIGIR 2009

5. From Verbal Disagreement to LSI Observed: Mismatch between the way that people want to retrieve information from a computer and the way that systems designers describe that information Observed: Mismatch between the way that people want to retrieve information from a computer and the way that systems designers describe that information The trouble with UNIX The trouble with UNIX Command names, menu and category descriptors, keywords Command names, menu and category descriptors, keywords Studied: How people describe objects and operations Studied: How people describe objects and operations Text editing operations, systems functionality, common objects, recipes, classified ads, etc. Text editing operations, systems functionality, common objects, recipes, classified ads, etc. Demo: Demo: Data: Data: SIGIR 2009 On a piece of paper write the name you would give to a program that tells users about interesting activities occurring in Boston this weekend. (Try to think of a name that will be as obvious as possible; one that other people would think of.) SIGIR 2009

6. Findings: Findings: Tremendous diversity in the name that people use to describe the same objects or actions (aka, “the long tail”) Tremendous diversity in the name that people use to describe the same objects or actions (aka, “the long tail”) Single keyword: 0.07 – 0.18 “repeat rate” Single keyword: 0.07 – 0.18 “repeat rate” Single normative keyword: 0.16 - 0.36 Single normative keyword: 0.16 - 0.36 Three aliases: 0.38 – 0.67 Three aliases: 0.38 – 0.67 Infinite aliasing: Infinite aliasing: Interestingly, we have referred to this problem as: verbal disagreement, vocabulary mismatch, statistical semantics Interestingly, we have referred to this problem as: verbal disagreement, vocabulary mismatch, statistical semantics From Verbal Disagreement to LSI SIGIR 2009

7. CHI 1982 Paper … 0 th CHI Conference CHI 1982 Paper … 0 th CHI Conference From Verbal Disagreement to LSI SIGIR 2009 Photos from B. Shneiderman

8. From Verbal Disagreement to LSI Some solutions: … with a lot of help from our friends Some solutions: … with a lot of help from our friends Rich aliasing [Gomez et al. 1990] Rich aliasing [Gomez et al. 1990] Allow alternative words for the same item Allow alternative words for the same item “Natural” in the world of full-text indexing, but less so for keyword indexing or command naming “Natural” in the world of full-text indexing, but less so for keyword indexing or command naming Adaptive indexing [Furnas 1985] Adaptive indexing [Furnas 1985] Associate (failed) user queries to destination objects Associate (failed) user queries to destination objects Add these queries as new entries in term-document matrix Add these queries as new entries in term-document matrix Quickly reduces failure rate for common requests/tasks Quickly reduces failure rate for common requests/tasks Latent Semantic Indexing [Dumais et al. 1988; Deerwester et al. 1990] Latent Semantic Indexing [Dumais et al. 1988; Deerwester et al. 1990] Model relationships among words, using dimension reduction Model relationships among words, using dimension reduction Especially useful when query and documents are short Especially useful when query and documents are short Baker, Borko/Bernick, Ossario (1962-1966); Kohl (SIGIR 1978, p.1) Baker, Borko/Bernick, Ossario (1962-1966); Kohl (SIGIR 1978, p.1) SIGIR 2009

9. From Verbal Disagreement to LSI Many applications and algorithms of LSI Many applications and algorithms of LSI Bell Labs directory of services, expert finding, reviewer assignment, handwritten notes, data evidence analysis, measurement of knowledge, literature-based discovery, IR & IF test collections Bell Labs directory of services, expert finding, reviewer assignment, handwritten notes, data evidence analysis, measurement of knowledge, literature-based discovery, IR & IF test collections Rich aliasing and Adaptive indexing in Web era Rich aliasing and Adaptive indexing in Web era Full text indexing (rich aliases from authors) Full text indexing (rich aliases from authors) Anchor text or Tags (rich aliases from other users) Anchor text or Tags (rich aliases from other users) Historical query-click data (adaptive indexing, with implicit measures) Historical query-click data (adaptive indexing, with implicit measures) SIGIR 2009

10. Common Themes The last 10-20 years … amazing time to be involved in IR The last 10-20 years … amazing time to be involved in IR TREC and related evaluations TREC and related evaluations TREC-1 in 1992 TREC-1 in 1992 Search is everywhere – desktop, enterprise, Web Search is everywhere – desktop, enterprise, Web Web search Web search Big advances in scale, diversity of content and users, quality of results (for some tasks), etc. Big advances in scale, diversity of content and users, quality of results (for some tasks), etc. SIGIR community has a lot to be proud of SIGIR community has a lot to be proud of But … many search tasks are still quite hard But … many search tasks are still quite hard Need to represent and leverage richer contextual information about users, domains, and task environments in which search occurs Need to represent and leverage richer contextual information about users, domains, and task environments in which search occurs SIGIR 2009

11. Web Search at 15 Number of pages indexed Number of pages indexed 7/94 Lycos – 54,000 pages 7/94 Lycos – 54,000 pages 95 – 10^6 millions 95 – 10^6 millions 97 – 10^7 97 – 10^7 98 – 10^8 98 – 10^8 01 – 10^9 billions 01 – 10^9 billions 05 – 10^10 … 05 – 10^10 … Types of content Types of content Web pages, newsgroups Web pages, newsgroups Images, videos, maps Images, videos, maps News, blogs, spaces News, blogs, spaces Shopping, local, desktop Shopping, local, desktop Books, papers, many formats Books, papers, many formats Health, finance, travel … Health, finance, travel … What’s available How it’s accessed SIGIR 2009

12. The search box The search box Spelling suggestions Spelling suggestions Query suggestions Query suggestions Auto complete Auto complete Inline answers Inline answers Richer snippets Richer snippets But, we can do better But, we can do better Support for Searchers SIGIR 2009 … by understanding context Search in the future will look nothing like today's simple search engine interfaces, [Susan Dumais] said, adding, "If in 10 years we are still using a rectangular box and a list of results, I should be fired.“ [Mar 7, 2007, NYTimes, John Markoff]

13. SIGIR 2009 Search Today User Context Task/Use Context Query Words Ranked List Search and Context Document Context Query Words Ranked List

14. SIGIR 2009 Modeling Users Short vs. long term Individual vs. group Implicit vs. explicit Using User Models Stuff I’ve Seen (re-finding) Personalized Search News Junkie (novelty) User Behavior in Ranking Domain Expertise at Web-scale Evaluation Many methods, scales Individual components and their combinations Systems/Prototypes New capabilities and experiences Algorithms and prototypes Deploy, evaluate and iterate Redundancy Temporal Dynamics Inter-Relationships among Documents Categorization and Metadata Reuters, spam, landmarks, web categories … Domain-specific features, time Interfaces and Interaction Stuff I’ve Seen, Phlat, Timelines, SWISH Tight coupling of browsing and search

15. User Modeling Modeling searcher’s interests and activities over time Iterative and interactive nature of search Within and across sessions Example applications Re-finding (e.g., Stuff I’ve Seen, Web) [Dumais et al. 2003] Personalization (e.g., PSearch) [Teevan et al. 2005] Novelty (e.g., News Junkie) [Gabrilovich et al. 2004] Domain expertise at Web-scale [White & Dumais 2009] User behavior for Web ranking [Agichtein et al. 2006] Evaluation via explicit judgments, questionnaires, client-side instrumentation, and large-scale search logs, lab and field studies, etc. SIGIR 2009

16. Re-Finding on the Desktop Stuff I’ve Seen (SIS) [Dumais et al. 2003] : Unified access to many types of info (e.g., files, email, calendar, contacts, web pages, rss, im) Index of content and metadata (e.g., time, author, title, size, usage) Rich UI possibilities, because it’s your stuff and client application Demo: Analysis: Deployed different versions Query syntax, Result previews Ranking defaults (time, best-match) Questionnaires, Free-form feedback, Log data, Lab experiments SIGIR 2009 Stuff I’ve Seen

17. Re-Finding on the Desktop Research Results: Research Results: Short queries Short queries Few advanced operators in initial query (<10%) Few advanced operators in initial query (<10%) Many advanced operators via specification in UI (~50%) - filter; sort Many advanced operators via specification in UI (~50%) - filter; sort Date by far the most common sort attribute (vs. best-match) Date by far the most common sort attribute (vs. best-match) Importance of time, people, episodes in human memory Importance of time, people, episodes in human memory Few searches for “best match”; many other criteria Few searches for “best match”; many other criteria Need for “abstractions” – date, people, kind Need for “abstractions” – date, people, kind Rich client-side interface Rich client-side interface Support fast iteration/refinement Support fast iteration/refinement Fast filter-sort-scroll vs. next-next-next Fast filter-sort-scroll vs. next-next-next Interesting reviews from SIGIR Interesting reviews from SIGIR Practice: XP and Vista desktop search Practice: XP and Vista desktop search SIGIR 2009

18. Re-Finding on the Web 50-80% page visits are re-visits 50-80% page visits are re-visits 30-50% of queries are re-finding queries 30-50% of queries are re-finding queries Data from Teevan et al., SIGIR 2007 SIGIR 2009 Total = 43% Big opportunity to support re-finding on Web Big opportunity to support re-finding on Web Models to combine Web rank w/ personal history of interaction Models to combine Web rank w/ personal history of interaction Interfaces to support finding and re-finding Interfaces to support finding and re-finding

19. Personalization Today: People get the same results, independent of current session, previous search history, etc. Today: People get the same results, independent of current session, previous search history, etc. PSearch [Teevan et al. 2005] : Uses rich client-side model of a user to personalize search results PSearch [Teevan et al. 2005] : Uses rich client-side model of a user to personalize search results SIGIR 2009 sigir 2009 sigir 2009 User profile: * Content * Interaction history

20. PSearch “Demo” Query: SIGIR Query: SIGIR SIGIR 2009

21.  Building a User Profile  Type of information  Content: Past queries, web pages, desktop  Behavior: Visited pages, explicit feedback  Time frame: Short term, long term  Who: Individual, group  Where the profile resides:  Local: Richer profile, improved privacy [but, increasingly rich public data]  Server: Richer communities, portability  Using the User Profile  Ranking  Query support  Result presentation PSearch SIGIR 2009 Personalization

22. Ranking algorithm [Teevan et al. 2007] Ranking algorithm [Teevan et al. 2007] Linear combination of scores from: content match, history of interaction, Web ranks Linear combination of scores from: content match, history of interaction, Web ranks When to personalize? [Teevan et al. 2008, in press] When to personalize? [Teevan et al. 2008, in press] Personalization works well for some queries, but not others Personalization works well for some queries, but not others Models for predicting when to personalize using features the query and query-user Models for predicting when to personalize using features the query and query-user Evaluating personalized search [Fox 2005; Teevan et al. in press] Evaluating personalized search [Fox 2005; Teevan et al. in press] What’s relevant for you What’s relevant for you Explicit judgments (offline and in situ) Explicit judgments (offline and in situ) Implicit “judgments” from behavioral interaction Implicit “judgments” from behavioral interaction Linking explicit and implicit (e.g., Curious Browser) Linking explicit and implicit (e.g., Curious Browser) Personalization SIGIR 2009 Curious Browser Study (~4k) * 45% w/ just click * 75% w/ click + dwell + session

23. Personalization and Privacy PSearch PSearch Local profile, local computation Local profile, local computation Nothing sent to the server except the original query Nothing sent to the server except the original query Need profile and web content in same place to rank Need profile and web content in same place to rank When information is stored in the cloud When information is stored in the cloud Send query and user profile, or store user profile in cloud Send query and user profile, or store user profile in cloud Transparency Transparency Control Control Other approaches we are exploring Other approaches we are exploring Matching an individual to a group Matching an individual to a group Light weight profiles (e.g., queries in a session) Light weight profiles (e.g., queries in a session) Public or semi-public profiles (e.g., Tweets, Facebook status, blogs) Public or semi-public profiles (e.g., Tweets, Facebook status, blogs) SIGIR 2009

24. End of Serendipity? Does great search and personalization mean the end of serendipity? Does great search and personalization mean the end of serendipity? No, actually better potential for serendipity No, actually better potential for serendipity Relevance vs. interestingness Relevance vs. interestingness Personalization finds more relevant results Personalization finds more relevant results Personalization finds more interesting results Personalization finds more interesting results Many not relevant results are interesting Many not relevant results are interesting Need to be “ready” for serendipity Need to be “ready” for serendipity SIGIR 2009

25. Categorization and Metadata Algorithms and applications Algorithms and applications Reuters, Web - fast SVM algorithm [Dumais et al. 1998, 2000] Reuters, Web - fast SVM algorithm [Dumais et al. 1998, 2000] Junk email [Sahami et al. 1998] Junk email [Sahami et al. 1998] Domain-specific feature engineering Domain-specific feature engineering Constantly changing content (both ham and spam) Constantly changing content (both ham and spam) Using metadata for ranking [Bennett et al.] Using metadata for ranking [Bennett et al.] Using metadata in UX Using metadata in UX Tight coupling search & browse – e.g., SIS, Phlat [Dumais et al. 2003] Tight coupling search & browse – e.g., SIS, Phlat [Dumais et al. 2003] Faceted-metadata in many verticals -> Web? [Teevan et al. 2008] Faceted-metadata in many verticals -> Web? [Teevan et al. 2008] Information theoretic models of search/navigation [Downey et al. 2008] Information theoretic models of search/navigation [Downey et al. 2008] Leveraging relations among documents Leveraging relations among documents SIGIR 2009

26. Many queries contain implicit metadata Many queries contain implicit metadata thomas edison image portrait thomas edison image portrait latest lasik techniques, canada latest lasik techniques, canada good nursing programs in baltimore good nursing programs in baltimore cheap digital camera cheap digital camera overview of active directory domains overview of active directory domains … Limited support for users to articulate this Limited support for users to articulate this Metadata on the Web UMAP, June 24 2009 SIGIR 2009

27. Future Challenges Dynamic information environments [Adar et al., Elsas et al.] Dynamic information environments [Adar et al., Elsas et al.] Content changes (e.g., news, blogs, lifelogs … much more general) Content changes (e.g., news, blogs, lifelogs … much more general) People re-visit, re-query, re-find People re-visit, re-query, re-find IR opportunities … crawling, doc and user representation, ranking, etc. IR opportunities … crawling, doc and user representation, ranking, etc. Interesting historically and socially Interesting historically and socially Data/Evaluation Data/Evaluation Data as valuable resource Data as valuable resource Large-scale log data Large-scale log data Operational systems and a “Living Laboratory” Operational systems and a “Living Laboratory” IR opportunities … representations, ranking, etc. IR opportunities … representations, ranking, etc. Thinking outside the traditional IR boxes Thinking outside the traditional IR boxes Better understanding of users and application domains Better understanding of users and application domains Collaborations across disciplinary boundaries Collaborations across disciplinary boundaries SIGIR 2009

28. Information Dynamics 1996 Microsoft Research Homepage 2009 SIGIR 2009

29. Information Dynamics My Homepage 1998 2008 SIGIR 2009 1998 2008

30. Information Dynamics 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 Content Changes Today’s Browse and Search Experiences But, ignores … User Visitation/ReVisitation 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 SIGIR 2009

31. Dynamics and Search Improved crawl policy (common use) Improved crawl policy (common use) Improved ranking using temporal IR models [Elsas and Dumais] Improved ranking using temporal IR models [Elsas and Dumais] Queries have different temporal patterns Queries have different temporal patterns Pages have different rates of change Pages have different rates of change Document priors (using temporal vs. link structure) Document priors (using temporal vs. link structure) Terms have different longevity Terms have different longevity Some are always on the page; some transient Some are always on the page; some transient Show change in snippets Show change in snippets More general browser support [Teevan et al. 2009] More general browser support [Teevan et al. 2009] SIGIR 2009 Time Sept Oct Nov Dec Welcome – Please join us in Boston | SIGIR’09 The SIGIR 2009 conference opens in just over a week in Boston, Massachusetts, at the Sheraton Boston Hotel and Northeastern University. The conference is chock full of exciting... New content: Please join your colleagues by starting the conference with a free continental breakfast in the Sheraton Hotel, Back Bay A&B, from 7:00am to 8:20am on Monday July 20. sigir2009.org SIGIR 2009

32. Data and Evaluation Data as a critical resource Data as a critical resource Shared IR data resources typically consist of Shared IR data resources typically consist of Static collection of documents and queries Static collection of documents and queries Judgments of Q-Doc in isolation Judgments of Q-Doc in isolation Judgments with limited context (just the current query) Judgments with limited context (just the current query) Judges (who are usually not the searcher) Judges (who are usually not the searcher) … and these resources often shape the questions we ask … and these resources often shape the questions we ask Search is an inherently interactive and iterative process, so user interaction data, is an especially important resource for the IR community Search is an inherently interactive and iterative process, so user interaction data, is an especially important resource for the IR community Large-scale log data Large-scale log data Operational system as an experimental platform Operational system as an experimental platform SIGIR 2009

33. Data and Evaluation Large-scale log data Large-scale log data Understanding how user interact with existing systems Understanding how user interact with existing systems What they are trying to do; Where they are failing; etc. What they are trying to do; Where they are failing; etc. Implications for: models, and interactive systems Implications for: models, and interactive systems Lemur Query Log Toolbar – developing a community resource ! Lemur Query Log Toolbar – developing a community resource ! Operational systems as an experimental platform Operational systems as an experimental platform Can also conduct controlled experiments in situ Can also conduct controlled experiments in situ Interleave results from different methods [Radlinski & Joachims 2005] Interleave results from different methods [Radlinski & Joachims 2005] A/B testing -- Data vs. the “hippo” [Kohavi 2008] A/B testing -- Data vs. the “hippo” [Kohavi 2008] Important in: linking offline and interactive results, understanding effect sizes, relations among results (and other page components), etc. Important in: linking offline and interactive results, understanding effect sizes, relations among results (and other page components), etc. Can we build such a “Living Laboratory”? Can we build such a “Living Laboratory”? Replicability in the face of changing content, users, queries Replicability in the face of changing content, users, queries SIGIR 2009

34. Opportunities Continued improvements in representation and ranking Continued improvements in representation and ranking Think outside the traditional IR boxes !!! Think outside the traditional IR boxes !!! Develop a better understanding of users, and their tasks Develop a better understanding of users, and their tasks Design and evaluate interactive systems to support this Design and evaluate interactive systems to support this Importance of Importance of New data resources New data resources Interdisciplinary perspective Interdisciplinary perspective SIGIR 2009

35. Thanks (again)! Bell Labs MSR, CLUES (Context, Learning and User Experience In Search) SIGIR 2009