Measuring Information Architecture Marti Hearst UC Berkeley

A View of Information Architecture (Newman et al. 00) Information design –structure, categories of information Navigation design –interaction with information structure Graphic design –visual presentation of information and navigation (color, typography, etc.) Information Architecture

A Simple Taxonomy Catalog Sites Web-based Information Systems Web- Presence Sites Service- Oriented Sites low high Complexity of Applications Complexity of Data From Mecca et al., WebDB’99

An Important IA Trend Generating web pages from databases Implications: –Web sites can adapt to user actions –Web sites can be instrumented –“An essential feature of a design environment is to give authors the possibility of evaluating the current network against the final adaptive system.” »Petrelli, Baggio, & Pezzulo, Adaptive Hypertext Design Environments: Putting Principles into Practice, AH 2000

Can we measure? Research by Ivory, Sinha, & Hearst –Simple measures of surface properties can fairly accurately predict website ratings. –Clearly we are successfully quantifying something about good vs. bad design. –However, additional work is needed to show causality, create insight –We aren’t capturing everything.

What to Measure? Jessi: don’t blindly count clicks Ronny Kohavi, Blue Martini, agrees: –“Using hits and page views to judge site success is like evaluating a musical performance by its volume.” Forrester Research Report, 1999 –Suggests conversion rate as an important metric –The key is to build measurement of what you care about directly into the design of the system.

Why Measure? To learn what works … and what doesn’t Nick: measurement lags innovation –So? Not everyone is a virtuoso. Let’s work to help those who do things other than design web sites.

Why measure? Test hypotheses. My current interest: –How can an information architecture be designed to successfully convey scent during navigation?

Metadata usage in Epicurious PrepareCuisineIngredientDish Recipe

Metadata usage in Epicurious PrepareCuisineIngredientDish PrepareCuisineDish I Select

Metadata usage in Epicurious PrepareCuisineIngredientDish I > Group by PrepareCuisineDish

Metadata usage in Epicurious PrepareCuisineIngredientDish PrepareCuisineDish I > Group by

Metadata usage in Epicurious PrepareCuisineIngredientDish PrepareCuisineDish I > Group by PrepareCuisine I Select

Recipe Information Architecture Information design –Recipes have five types of metadata categories Cuisine, Preparation, Ingredients, Dish, Occasion Each category has one level of subcategories

Recipe Information Architecture Navigation design –Home page: show top level of all categories –Other pages: A link on an attribute ANDS that attribute to the current query; results are shown according to a category that is not yet part of the query A change-view link does not change the query, but does change which category’s metadata organizes the results

Metadata Usage in Epicurious Can choose category types in any order But categories never more than one level deep And can never use more than one instance of a category –Even though items may be assigned more than one of each category type Items (recipes) are dead-ends –Don’t link to “more like this” Not fully integrated with search

Epicurious Metadata Usage Problem: lacks integration with search

Scent for grocery shopping Homerun.com

Scent for grocery shopping Homerun.com

Scent for grocery shopping Homerun.com

Scent for grocery shopping peapod.com

Scent for grocery shopping peapod.com

Scent for grocery shopping peapod.com

Scent for grocery shopping webvan.com

Scent for grocery shopping webvan.com

Questions I want to answer How many facets are allowable? Should facets be mixed and matched? How much is too much? Should hierarchies be progressively revealed, tabbed, some combination? How should free-text search be integrated?

How to measure this A framework that can be systematically varied will give us some answers Measuring the results of tens (of thousands) of users completing their tasks should also help

Closing Thought Putting science into HCI isn’t easy Let’s work hard to make it work