1. Using Metadata to Improve Search User Interfaces Marti Hearst UC Berkeley FLINT Workshop, August 2001

2. My Claims Web Search is OK Gets people to the right starting points Web SITE search is NOT ok The best way to improve site search is NOT to make new fancy algorithms Instead …

3. The best wasy to improve search: Improve the User Interface

4. Recent Study by Vividence Research Spring 2001, 69 web sites The most common problems: 53% had poorly organized search results 32% had poor information architecture 32% had slow performance 27% had cluttered home pages

5. Organizing Search Results There is a lot of prior work on this Cha-Cha (Chen et al. 1999) Scatter-Gather clustering (Cutting et al. 93, Hearst et al. 1996) Becoming more prevalent in web search too. Teoma Visimo Northern Light

6. Putting Results into Categories

7. Usually doesn’t work well when drilling down, however

11. Web Search Results Grouping Drill down one category Cannot mix and match categories Probably isn’t all that useful But …what about site search?

12. A Taxonomy of WebSites Catalog Sites Web-based Information Systems Web-Presence Sites Service- Oriented Sites low high Complexity of Applications Complexity of Data From Mecca et al., WebDB’99

13. An Important Trend Web sites generated from databases Implications: Web sites can adapt to user actions Web sites can be instrumented

14. Navigation on the Web Web search engines are good at getting people to the right site. But … what happens when the user reaches the site? Follow Links … or … Search

15. Following Hyperlinks Works great when it is clear where to go next Frustrating when the desired directions are undetectable or unavailable Site Search Is not getting good reviews

16. An Analogy text search hypertext

17. Analogy Hypertext: A fixed number of choices of where to go next; A glance at the map tells you where you are; But may not go where you want to go. To get from Topeka to Santa Fe, may have to go through Frostbite Falls Site Search: Can go anywhere; But may get stuck, disoriented, in a crevasse!

18. Goal: An All-Tertrain Vehicle The best of both techniques A vehicle that magically lays down track to suggest choices of where you want to go next based on what you’ve done so far and what you are trying to do The tracks follow the lay of the land and go everywhere, but cross over the crevasses The tracks allow you to back up easily

19. New interfaces are mixing and matching thesaurus-style metadata Time/DateTopicRoleGeoRegion  The question: how to do this effectively?

20. Goals for Metadata Usage Well-integrated with search Provides useful hints of where to go next Tailored to task as it develops Personalized Dynamic

21. The FLAMENCO Project FLexible Access using MEtadata in Novel Combinations Main goal: Perform systematic studies to determine how metadata should be incorporated into search Answer questions such as: Given a set of user goals and a set of information: How many metadata combinations to show? What level of detail to show? How best to preview and postview choices?

22. Evaluation Methodology Regression Test Select a set of tasks Use these throughout the evaluation Start with a baseline system Evaluate using the test tasks Add a feature Evaluation again Compare to baseline Only retain those changes that improve results

23. Recipe Example

24. soar.berkeley.edu/recipes

27. www.epicurious.com

31. Epicurious Metadata Usage Advantages Creates combinations of metadata on the fly Different metadata choices show the same information in different ways Previews show how many recipes will result Easy to back up Supports several task types ``Help me find a summer pasta,'' (ingredient type with event type), ``How can I use an avocado in a salad?'' (ingredient type with dish type), ``How can I bake sea-bass'' (preparation type and ingredient type)

32. Recipe Information Architecture Information design Recipes have five types of metadata categories Cuisine, Preparation, Ingredients, Dish, Occasion Each category has one level of subcategories

33. Recipe Information Architecture Navigation design Home page: show top level of all categories Other pages: A link on an attribute ANDS that attribute to the current query; results are shown according to a category that is not yet part of the query A change-view link does not change the query, but does change which category’s metadata organizes the results

34. Metadata usage in Epicurious PrepareCuisineIngredientDish Recipe

35. Metadata usage in Epicurious PrepareCuisineIngredientDish PrepareCuisineDish I Select

36. Metadata usage in Epicurious PrepareCuisineIngredientDish I > Group by PrepareCuisineDish

37. Metadata usage in Epicurious PrepareCuisineIngredientDish PrepareCuisineDish I > Group by

38. Metadata usage in Epicurious PrepareCuisineIngredientDish PrepareCuisineDish I > Group by PrepareCuisine I Select

39. Metadata Usage in Epicurious Can choose category types in any order But categories never more than one level deep And can never use more than one instance of a category Even though items may be assigned more than one of each category type Items (recipes) are dead-ends Don’t link to “more like this” Not fully integrated with search

40. Epicurious Basic Search Lacks integration with metadata

42. The FLAMENCO PROJECT Usability Studies to determine how to improve website search Marti Hearst Rashmi Sinha Ame Elliott Jen English Kirsten Swearington Ping Yee http://bailando.sims.berkeley.edu/flamenco.html

43. The FLAMENCO Project Usability studies to determine how to improve site search using metadata Researchers: Marti Hearst, Rashmi Sinha Ame Elliott, Jennifer English, Kirstin Swearington, Ping Yee So far: Epicurious usability study Image search interface framework and study design

44. Epicurious Usability Study 9 participants so far Independent Variables: 1) Epicurious Interface (Basic vs. Enhanced vs. Browse) 2) Task type (known-item search vs. browsing for inspiration) 3) Degree of constraint of query 4) Number of results required (1 vs. many) Dependent Variables: 1) Time to find satisfactory recipe(s) 2) Navigation path (backtracking, starting over, revising queries) 3) Satisfaction with results of search 4) Satisfaction with individual system features (e.g. breadcrumbs, query previews, refine by hyperlinks) 5) Likelihood of using each interface in the future.

45. Epicurious Usability Study Participants were asked to: Do 3 pre-specified searches in advance In the lab: Specify a cooking scenario of interest to them Search for 3 recipes for this recipe Search for each recipe using each of the interfaces Complete several structured tasks Along the way, answer questions about Getting closer or farther away from goal Satisfaction with search results Satisfaction with the interace

46. Usability Study: Preliminary Results, Preference Data

47. Usability Study Preliminary Results: Feature Preference

48. Usability Study Preliminary Results: Quantitative

49. Usability Study Preliminary Results: Constraint-based Preferences

50. Usability Study Results: Summary People liked the browsing-style metadata-based search and found it helpful People sometimes preferred the search-style metadata search when the task was more constrained But zero results are frustrating This can be alleviated with query previews People disprefer the standard simple search

51. Application to Image Search

52. Image Search: What is the task? Illustrate my slides? “Find a crevasse” Keyword match works pretty well Find inspiration for an architectural design? Needs richer search support

53. Faceted Metadata Planalto Palace Parti Communiste Francais Pantheon Oscar Neimeyer Oscar Neimeyer Jacques-Gabriel Soufflot 20 th Century 20 th Century 17 th & 18 th C. Brasilia Paris Paris Stone Curvilinear Stone Image: Architect: Period: Location: Concept:

54. Planalto Palace Parti Communiste Francais Pantheon Oscar Neimeyer Oscar Neimeyer Jaques-Gabriel Soufflot 20 th Century 20 th Century 17 th & 18 th C. Brasilia Paris Paris Stone Curvilinear Stone Image: Architect: Period: Location: Concept: Faceted Metadata

55. SPIRO Query Form

56. SPIRO query on Subject: church

60. Pilot Study Architecture task: Emphasize images over text Use hypertext-style interface as a reasonable baseline for comparison Find out how much choice is too much Find out whether explicit metadata is better than implicit more-like-this

61. Evaluation Methodology Solicit feedback from architects to determine if faceted metadata is helpful and how to present it Informal evaluation of paper prototype Informal study of a crude live version 1 hour one-on-one with 9 architects /grad students, 2 tasks (audio recorded) and a survey

62. Results of a pilot study with Archictects: Metadata is Helpful Very positive feedback about the general approach All 9 participants named the metadata in the search results area as their favorite aspect of Flamenco Metadata was successful at giving hints about where to go next Perceived as useful “These are places I can go from here.”

63. Results: More Metadata Please Participants asked for more metadata Although there were complaints about the contents of the metadata, users still wanted more Longer lists of options (more hints) Users wanted more control to make very specific searches Half the participants requested the ability to control order of results with metadata Juxtapose visible images 2 different ways: Overview (one image from each project) vs. like together ( all images of a project next to each other) Different than ranking for text retrieval (precision, recall), but ordering does matter

64. Results: Complaints The UI was not successful at clarifying searching within results vs. starting a new search Only 2 of the 9 participants understood the distinction without discussion – but they want to do both The 1/3 of the participants who couldn’t find a treasure hunt image felt that Flamenco was slow Corroborates findings that perceived system speed is about finding what you want (Spool ‘00)

65. Summary Web site search needs improvement Users want more organized results Our approach: integrate navigation with search Metadata is being mixed and matched in interesting ways, but there are no guidelines on what works We are investigating how to design websites containing large sets of items Architectural images Biomedical text Preliminary results indicate that metadata organization is useful in some situations Depends on the type of search need

66. Summary Our goals Systematically determine what works, with the following emphases: Task-centric Integrate metadata with search Dynamic previews Easily retrace steps Develop recommendations that reflect both the task structure and the richness of the information structure In future: integrate with more sophisticated displays

67. http://bailando.sims.berkeley.edu

68. “Parametric” Search From an XML glossary "A search request submitted to a search or database engine delivered with consideration for the metadata of the underlying dataset.” www.sla.org/chapter/ctor/courier/v37/v37n1.pdf A survey of sites using parametric search: http://www.amp.com/search/default.asp (see product family search) http://ebiz.zilog.com/ http://www.sears.com (Dieselpoint) http://dieselpoint.com/flashlink.htm (for Dieselpoint 2.0 demo) http://www.findmro.com (Requisite's BugsEye) http://www.cypress.com (Saqqara's one step) http://infineon-tech.sacosnet.de/search/index.htm http://www.idt.com/tools/parametric.html http://www.ti.com/sc/docs/psheets/parms/uarts.htm#parms http://www.gensemi.com/search/productsearch.htm http://www.usa.samsungsemi.com/search/ http://www.gearfinder.com http://www.mysimon.com/category/index.jhtml?c=babydiaperingbathing

69. “Parametric” Search Sites Goal is to focus on product group for comparison shopping. Common Procedure Begin with a list of product "families" or groups. User selects a category, and is prompted to 1) select a sub-category from a list of hyperlinks or 2) select search parameters using a form If the number of results is too big, the system may prompt the user to refine the search further. When an acceptable number of results is returned, the user sees a list of products which can be: 1) sorted by various criteria 2) selected for display in a comparison table 3) viewed individually with more detail.

70. “Parametric” Search Observations: Only one facet (appropriate for products?) No query previews Breadcrumbs rare Many allow sorting by attribute to facilitate comparison “Others like this” simply moves up the hierarchy