1. Universal Access: More People. More Situations Content or Graphics Content or Graphics? An Empirical Analysis of Criteria for Award-Winning Websites Rashmi Sinha, Marti Hearst & Melody Ivory University of California, Berkeley Maya Draisin International Academy of Digital Arts and Sciences For more information: http://sims.berkeley.edu/~sinha/WebbyAwards

2. Universal Access: More People. More Situations Content or Graphics Question: How can we identify characteristics of good websites on a large scale? Question: How can we turn these characteristics into empirically validated guidelines?

3. Universal Access: More People. More Situations Content or Graphics Conduct Usability Studies: Hard to do on a large scale Find a corpus of websites already identified as good! Use the WebbyAwards database

4. Universal Access: More People. More Situations Content or Graphics Details about WebbyAwards 2000 Qualities of highly rated websites Relative importance of Content & Graphics Empirically validated design guidelines Future Plans: WebbyAwards 2001, user studies etc. Talk Plan

5. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Criteria for submission to the WebbyAwards Anyone who has a current, live website Should be accessible to the general public Should be predominantly in English No limit to the number of entries that each person can make

6. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Site Category Sites must fit into at least one of 27 categories. For example: Arts Activism Fashion Health News Radio Sports Music News Personal Websites Travel Weird Sites can be listed in multiple categories

7. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web 3 Stage Judging Process Review Stage: From 3000 to 400 sites –3 judges rate each site on 6 criteria, and cast a vote if it will go to the next stage Nominating Stage: From 400 to 135 sites –3 judges rate each site on 6 criteria, and cast a vote if it will go to the next stage

8. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web People’s Voice: 135 to 27 sites –Anyone can vote for their favorite site among the ones in the final stage Final Stage: From 135 to 27 sites Judges casts vote for favorite site

9. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Criteria for judging 6 criteria –Overall Site Experience –Five specific Criteria Content Structure and Navigation Visual Design Interactivity Functionality Scale: 1-10 (highest)

10. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web ContentContent: is the information provided on the site. Good content is engaging, relevant, appropriate for the audience-you can tell it's been developed for the Web because it's clear and concise and it works in the medium …

11. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Structure & NavigationStructure & Navigation: is the organization of information and navigation. Sites with good structure and navigation are consistent and effective. They allow you to form a mental model of the information provided …

12. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Visual DesignVisual Design: is the appearance of the site. Good visual design is high quality, appropriate, and relevant for the audience and the message it is supporting …

13. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web InteractivityInteractivity: is the way a site allows a user to do something. Good interactivity is more than sound effects, and a Flash animation. It allows the user to give and receive. Its input/output in searches, chat rooms, ecommerce etc.…

14. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web FunctionalityFunctionality: is the use of technology on the site. Good functionality means the site loads quickly, has live links, and any new technology used is functional and relevant for the intended audience …

15. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Overall ExperienceOverall Experience: The overall experience encompasses content, structure and navigation, visual design, functionality, and interactivity, but it also encompasses the intangibles that make one stay or leave…

16. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Webby Judges –Internet professionals who work with and on the internet: new media journalists, editors, web developers, and other Internet professionals –have clearly demonstrable familiarity with the category which they review

17. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Quick Review of Judging Process Criteria for judging Criteria for judging: FunctionalityFunctionality InteractivityInteractivity Overall ExperienceOverall Experience ContentContent NavigationNavigation Visual DesignVisual Design Nominating StageNominating Stage: 400 to 135 sites, each sites judged by 3 judges Review StageReview Stage : 3000 to 400 sites, each site judged by 3 judges

18. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Are we sampling from the whole range of websites (good to bad) in the Review Stage?

19. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Review Stage: The whole range of sites, good to bad Overall Rating Mean = 6.01 SD = 1.59

20. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Classification Accuracy for Sites = 91% Can votes be predicted by specific criteria? Statistical Technique: Discriminant analysis Question: Can we predict the votes from the 5 specific criteria? Can overall rating be predicted by specific criteria? Statistical Technique: Regression analysis Question: What % variance is explained by 5 criteria Percentage variance explained = 89%

21. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web What criteria contribute most to overall rating?

22. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Nominating Stage Analysis 6 criteria –Content, Structure & Navigation, Visual Design, Functionality & Interactivity –Overall experience 400 sites 3 judges rated each site

23. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Nominating Stage: Top sites for each category Mean = 7.6 SD = 1.66 Overall Rating

24. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web What criteria contribute to overall rating at Nominating Stage? 77% variance explained in overall rating

25. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Unique Contribution of Content and Visual Design People’s Voice Ratings also indicate that people vote for sites with better content rather than better visual design

26. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Summary of Findings The specific ratings do explain overall experience. The best predictor of overall score is content. The second best predictor is interactivity. The worst predictor is visual design

27. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Are there differences between categories? Focus on a few Categories Art, Commerce & Radio Arts Activism Fashion Health News Sports Music News Personal Websites Travel

28. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Art

29. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Commerce Sites

30. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Radio Sites

31. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Conclusions The importance of criteria varies by category. Content is by far the best predictor of overall site experience. Interactivity comes next. Visual Design does not have as much predictive power except in specific categories Can we develop design guidelines by identifying characteristics of good web pages?

32. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Study 2 An empirical bottom-up approach to developing design guidelines –Challenge: How to go use Webby criteria to inform web page design? –Answer: Identify quantitative measures that characterize pages

33. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Quantitative Measures –Page Composition words, links, images, … –Page Formatting fonts, lists, colors, … –Overall Characteristics information & layout quality

34. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Quantitative page measures Word Count Body Text % Emphasized Body Text % Text Cluster Count Link Count Page Size Graphic % Color Count Font Count

35. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Quantitative Measures: Word Count

36. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Study Design Low Rated Sites Bottom 33% Webby Ratings Highly Rated Sites Top 33% Quantitative Page Metrics Word Count, Body Text %, Text Cluster Count, Link Count etc. Model Accuracy Across Categories 67% 63% Within Categories 83% 76%

37. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Classification Accuracy Comparing Top vs. bottom Accuracy higher for within categories

38. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web What page metrics predict site quality All metrics played a role –However their role differed for various categories of pages (small, medium & large) Summary –Across all pages in the sample Good pages had significantly smaller graphics percentage Good pages had less emphasized body text Good pages had more colors (on text)

39. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Role of Metrics for Medium Pages (230 words on average) Good medium pages –Emphasize less of the body text –Appear to organize text into clusters (e.g., lists and shaded table areas) –Use colors to distinguish headings from body text Suggests that these pages –Are easier to scan

40. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web No Text ClusteringNo Selective Highlighting Low Rated Page

41. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Selective Highlighting High Rated Page Text Clustering

42. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Why does this approach work? Superficial page metrics reflect deeper aspects of information architecture, interactivity etc.

43. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web Future work Do similar analysis for Webby2001 Add more metrics –Aspects of info, navigation, and graphic design Category-based profiles –Use clustering to identify different kinds of good and poor sites –These can be used to suggest alternative designs Do user studies to verify guidelines

44. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web In Summary Developing an understanding of characteristics of good pages and also generating empirically validated design guidelines Laying the foundation for a new methodology –Empirical, bottom up Can predict if a page is good or not with some accuracy

45. Universal Access: More People. More Situations Content or Graphics7 th Conference on Human Factors and the Web More information: http://sims.berkeley.edu/~sinha/WebbyAwards