Empirically Validated Web Page Design Metrics Melody Y. Ivory, Rashmi R. Sinha, Marti A. Hearst UC Berkeley CHI 2001

Empirically Validated Web Page Design Metrics 2 The Usability Gap 196M new Web sites in the next 5 years [Nielsen99] ~20,000 user interface professionals [Nielson99]

Empirically Validated Web Page Design Metrics 3 The Usability Gap Most sites have inadequate usability [Forrester, Spool, Hurst] (users can’t find what they want % of the time) 196M new Web sites in the next 5 years [Nielsen99] A shortage of user interface professionals [Nielson99]

Empirically Validated Web Page Design Metrics 4 The Usability Problem NON-professionals need to create websites Guidelines are helpful, but – There are MANY usability guidelines Survey of 21 web guidelines found little overlap [Ratner et al. 96] Why? – One idea: because they are not empirically validated – Sometimes imprecise – Sometimes conflict

Empirically Validated Web Page Design Metrics 5 Possible Solutions: Tools to Help Non-Professional Designers Examples: – A “grammar checker” to assess guideline conformance Imperfect Only suggestions – not dogma – Automatic comparison to highly usable pages/sites – Automatic template suggestions How to create these?

Empirically Validated Web Page Design Metrics 6 Current Design Analysis Tools Some tools report on easy-to- measure attributes – Compare measures to thresholds Stein (Rating Game), Theng & Marsden, Thimbley (Gentler) Not empirically validated – Guideline conformance CAST (Bobby), Scholtz & Laskowski (WebSAT), Lift Online – Perceptually based heuristics Faraday (Design Advisor) Small subset of features [Brajnik00] – Simplistic – Not empirically validated

Empirically Validated Web Page Design Metrics 7 The WebTANGO Approach Models of good design by looking at existing designs – Empirical foundation for easy-to-measure attributes – Focus on information-centric sites First work to take a large set of sites and analyze them

Empirically Validated Web Page Design Metrics 8 The Investigation Using quantitative measures to predict Web site ratings – Followup investigation [HFW00] Given – 1898 pages from 400+ sites – 11 quantitative measures to assess various Web page aspects Questions – Which features distinguish well-designed web pages? – Can metrics predict ratings? – Are there differences for categories of pages?

Empirically Validated Web Page Design Metrics 9 Webby Awards sites initially – 27 topical categories We studied sites from information-centric categories – Finance, education, community, living, health, services 100 judges – International Academy of Digital Arts & Sciences – 3 rounds of judging

Empirically Validated Web Page Design Metrics 10 Webby Awards criteria – Content – Structure & navigation – Visual design – Functionality – Interactivity – Overall experience Scale: 1-10 (highest) Nearly normally distributed What are judgments about?

Empirically Validated Web Page Design Metrics 11 Webby Awards 2000 Content criterion is best predictor Visual design criterion is worst predictor User study of 57 sites – Ratings reflect usability

Empirically Validated Web Page Design Metrics 12 Quantitative Measures: Aspects Impacting Usability Identified 42 attributes from the literature Roughly characterized: – Page Composition words, links, images, … – Page Formatting fonts, lists, colors, … – Overall Page Characteristics information & layout quality, download speed, …

Empirically Validated Web Page Design Metrics 13 Quantitative Measures: Word Count

Empirically Validated Web Page Design Metrics 14 Quantitative Measures: Body Text %

Empirically Validated Web Page Design Metrics 15 Quantitative Measures: Emphasized Body Text %

Empirically Validated Web Page Design Metrics 16 Quantitative Measures: Text Positioning Count

Empirically Validated Web Page Design Metrics 17 Quantitative Measures: Text Cluster Count

Empirically Validated Web Page Design Metrics 18 Quantitative Measures: Link Count

Empirically Validated Web Page Design Metrics 19 Quantitative Measures: Page Size (Bytes)

Empirically Validated Web Page Design Metrics 20 Quantitative Measures: Graphic %

Empirically Validated Web Page Design Metrics 21 Quantitative Measures: Graphic Count

Empirically Validated Web Page Design Metrics 22 Quantitative Measures: Color Count

Empirically Validated Web Page Design Metrics 23 Quantitative Measures: Font Count

Empirically Validated Web Page Design Metrics 24 Characterizing Measures: A View of Web Site Structure [Newman et al. DIS00] Information design – structure, categories of information Navigation design – interaction with information structure Graphic design – visual presentation Courtesy of Mark Newman

Characterizing Measures: Web Site Structure Assessed

Empirically Validated Web Page Design Metrics 26 Study Method The Webby factor – Principle components analysis of the 6 criteria Accounted for 91% of the variance Two comparisons – Model 1: Highly rated sites (top 33%) vs. the rest Using the overall Webby score – Model 2: Highly rated sites vs. bottom 33% Using the Webby factor

Empirically Validated Web Page Design Metrics 27 Findings We can accurately classify web pages – Linear discriminant analysis – Model 1: For highly-rated sites vs. rest 67% correct when not considering content categories 73% correct when taking content categories into account – Model 2: For highly-rated sites vs. bottom 65% correct when not considering content categories 80% correct when taking content categories into account

Empirically Validated Web Page Design Metrics 28 Findings Top vs. bottom – Webby factor Linear discriminant analysis Better for categories

Empirically Validated Web Page Design Metrics 29 Deeper Analysis Which metrics matter? – Linear regression analysis Backward elimination until adjusted R² reduced – All metrics played a role Compared small, medium, and large pages – Across the board (preliminary profiles) Good pages had significantly smaller graphics percentage Good pages had less emphasized body text Good pages had more colors (on text)

Empirically Validated Web Page Design Metrics 30 Small pages (66 words on average) Good small pages have (according to beta coefficients) – Slightly more content – Smaller page sizes – Fewer graphics – More font variations Suggests that these pages – Have faster download times corroborated by a download time metric – Use different fonts for headings vs. the rest of the text Examples – Services (Home Pages) Top Bottom

Empirically Validated Web Page Design Metrics 31 Medium pages (230 words on average) Good medium pages – Emphasize less of the body text – Appear to organize text into clusters (e.g., lists and shaded table areas) – Use colors to distinguish headings from body text Suggests that these pages – Are easier to scan

Empirically Validated Web Page Design Metrics 32 Large pages (827 words on average) Good large pages have – More headings – More links – Are larger but have fewer graphics Probably attributable to style sheets Suggests that good large pages – Are easier to scan and facilitate information seeking

Empirically Validated Web Page Design Metrics 33 Why does this work? Content is most important predictor BUT there’s predictive power in other aspects – Visual and navigation design – Verifies preliminary investigation [HFW00] Possibly: Good design is good design all over Note: we are NOT suggesting we can characterize: – Aesthetics or subjective preferences

Empirically Validated Web Page Design Metrics 34 How might we use this? Web Site Design Prediction Similarities Differences Suggestions Analysis Tool Profiles Comparable Designs Favorite Designs

Empirically Validated Web Page Design Metrics 35 Future work Distinguish according to page role – Home page vs. content vs. index … Better metrics – More aspects of info, navigation, and graphic design Site level as well as page level Category-based profiles – Use clustering to create profiles of good and poor sites – These can be used to suggest alternative designs

Empirically Validated Web Page Design Metrics 36 In Summary Automated tools should help close the Web Usability Gap We have a foundation for a new methodology – Empirical, bottom up We can empirically distinguish good pages – Empirical validation of design guidelines – Can build profiles of good vs. poor sites Eventually build tools to help users assess designs

Empirically Validated Web Page Design Metrics 37 More Information

Empirically Validated Web Page Design Metrics 38 Extra Slides Quantitative Metrics Statistical Measures Category Differences

Empirically Validated Web Page Design Metrics 39 Quantitative Metrics: Text Measures Text Content – Total visible words; also used for Reading Complexity over all text (I,O) – Words that are body text (I) – Words that are display text (i.e., headers) (I) – Gunning Fog Index (O) ratios of words, sentences and words with 3+ syllables) computed over prose – Number of words in the page’s title (I) – Number of prose words (for Reading Complexity) (O) – Number of big prose words (for Reading Complexity) (O) – Number of sentences for prose words (for both Reading Complexity measures) (O)

Empirically Validated Web Page Design Metrics 40 Quantitative Metrics: Text Measures Text Formatting – Body text words that are emphasized (e.g., bold, capitalized or near !'s) (G,O) – Changes in text position from flush left (G,O) How are aligned text blocks perceived? exampleexample – Text areas highlighted with color, bordered regions, rules or lists (G,O) How are colored clusters perceived? Example1 Example2Example1Example2 – Gunning Fog Index (ratios of words, sentences and words with 3+ syllables); computed over all text (O) – Number of big words (for Reading Complexity over all text) (O)

Empirically Validated Web Page Design Metrics 41 Quantitative Metrics: Text Measures Text Formatting – Minimum font size (G,O) – Maximum font size (G,O) – Average font size (G,O)

Empirically Validated Web Page Design Metrics 42 Quantitative Metrics: Link Measures Link Content – Words that are link text (I,N) – Average number of words in link text (N) – Total links (N) – Duplicate links (N) – Internal links (N,O)

Empirically Validated Web Page Design Metrics 43 Quantitative Measures: Link Measures Link Formatting – Total text links (N,I) – Links that are images (N,G,O) – Link areas highlighted with color, bordered regions, rules or lists (G,O)

Empirically Validated Web Page Design Metrics 44 Quantitative Measures: Image Measures Image Content – Total images (G) – Animated images (G,O) – Images that are links (G,N,O) – Duplicate images (G) – Average image height (G,O) – Average image width (G,O)

Empirically Validated Web Page Design Metrics 45 Quantitative Metrics: Color Measures Purpose – Quantify color usage (how many, for what) – Assess color quality (Murch)

Empirically Validated Web Page Design Metrics 46 Quantitative Metrics: Color Measures Color Usage – Total colors (G) – Number of body text colors (G,O) – Number of display text (headers) colors (G,O) – Number of link colors (N,G,O) – Number of default link colors (N,G,O) – Average number of times a color is used (G,O) What’s the right way to do this? – Number of good color combinations (Murch) (G,O) – Number of bad color combinations (Murch) (G,O) – Number of neutral color combinations (Murch) (G,O)

Empirically Validated Web Page Design Metrics 47 Quantitative Metrics: Page Formatting Measures Measures – Total fonts (face, size, bold, italic combinations) (G,O) – Height of page in pixels (600 pixel window) (G,O) – Width of page in pixels (800 pixel window) (G,O) – Total page pixels (G,O) – Total graphic pixels (G,O) – Number of horizontal scrolls required (G,O) – Number of vertical scrolls required (G,O)

Empirically Validated Web Page Design Metrics 48 Quantitative Metrics: Download Time Measure Estimating Download Time – Predictive model based on 56.6K modem (41.2K connection speed).957 correlation with measured time Explains 91% of the variance

Empirically Validated Web Page Design Metrics 49 Quantitative Metrics: Download Time Measure Model Measures – Total bytes for html (page & stylesheets) (O) – Total bytes for images (O) – Total bytes for scripts (O) – Total bytes for objects (applets, objects, layers) (O) – HTML files (.htm &.css) (O) – Image files (O) – Object files (applets, layer, sound, etc.) (O) – Script files (O) – Total number of scripts, applets, objects, etc. (O) – Number of tables (O)

Empirically Validated Web Page Design Metrics 50 Quantitative Metrics: Other Measures Measures – Level of non-home pages within the site (N,O) – Number of ads (O) – Number of accessibility syntax errors on page (Bobby) (O) – Number of HTML syntax errors on page (HTML Lint) (O) – Number of text fields, radio boxes, buttons and other form objects (O) – Number of words in meta tags (keyword & description) (I,N) – Functional type (e.g., home, index, content, reference, or other) (O)

Empirically Validated Web Page Design Metrics 51 Quantitative Metrics: Other Measures Measures – Similarity between link text words and content on the destination page (I,N) – Number of spelling errors (O)

Empirically Validated Web Page Design Metrics 52 Quantitative Metrics: Site Measures Measures – Consistency of page contents across pages (O) – Consistency of page formatting across pages Other Measures?

Statistical Measures

Empirically Validated Web Page Design Metrics 54 Statistical Measures

Empirically Validated Web Page Design Metrics 55 Category Differences Significant predictor metrics