Evaluation of an Information Service: Qualitative Assessment of the Internet Archive and the Wayback Machine SIMS Final Project Presentation May 13, 2002 Pallavi Aravind Vanessa Arce Peter Roessler

Outline Introduction to The Internet Archive Introduction to Our Study Research Design and Artifacts Research Findings Recommendations Conclusions

The Internet Archive is building a digital library of Internet sites. The Wayback machine makes it possible to “surf the web as it was” by accessing web pages stored in the Internet archive. this “Internet equivalent of the Library of Congress has been capturing and archiving every public Web page since 1996” The Internet Archive

Introduction to Our Study We attempt to identify and define ways in which the Archive is serving mankind Motivation: – its use for documenting the provenance of the Internet, as “a historical record of cyberspace… [and] as part of an innovative search tool that lets users call up ‘out-of-print’ Web pages.” – Gilliland-Swetland said, “it is important to understand the societal roles of archives because it is in the fulfillment of these roles that archivists provide the necessary skills and knowledge to contribute to the [current] paradigm.”

Research Design and Artifacts PhaseApproach Artifact Initial Assessment Meet with Brewster Kahle and assess objectives of project. Business Proposal Industry Research Evaluate use, design, and value of archives. Research References Heuristic Inspection Heuristic evaluation and inspection.Heuristic Recommendations User Testing Conduct usability tests with task-oriented operations and questionnaire. Pre and Post Questionnaire, Task Sheet, Results Phone Interviews Formal and anonymous phone interviews with active users of the Archive. Interview Questions, Coded Results Final Analysis Analyze data, compile and evaluate results, prepare report. Final Report

Research Design and Artifacts Initial Assessment Industry Research Heuristic Inspection User Testing Phone Interview Final Analysis

Research Artifact: Heuristics Plan – Educated Information Gathering – Evaluate Interface and Usability – Basis for Design of User Tasks and Questionnaire Methodology – Independent Evaluator Walkthrough – Comparative Heuristic Analysis

Heuristic Themes Research Artifact: Heuristics Interface Design & Architecture Search Mechanism Search Results User Help Heuristics

Research Artifact: User Testing Plan – User Impressions and Interactions – User Learning Curve Methodology – Performance Driven Tasks – Clear and Direct Design

User Test Themes Unique Service Interface Design Concerns Information Hierarchy Search Mechanism Limited/Meaningful Search Mechanism Inconsistencies Presentation of Search Results Depth of Information to Search Results IA does not have/provide desired information Research Artifact: User Test User Tests

Research Artifact: Phone Interviews Plan – Typical Archive User Insight – Verify User Definitions and Characteristics – Gather Opinions of Wayback Machine Methodology – Administered to Anonymous IA Users – Voluntary Participation – Actual Use Scenarios and Feedback

Phone Interview Themes User Profile First Exposure Uses Satisfaction and Recommendations Value and Vision Research Artifact: Phone Interviews Phone Interviews

High Level Themes Vision of Wayback vs. User Practicality Look and Feel Search & Results Technical Capability Communication User Threshold, Expectations and Use Scenarios Final Analysis

Look & Feel Intuitive Search Interface Invaluable Resource What? Why?

Search & Results

Methods to access the archive. Provision of advanced search How is the information stored? How is the information retrieved? Where is the boundary between archive and real web site? Frequency of Archival Request a crawl Policy on IP Communication FAQ Help! Internet Archive Archival Retrieval Internal Storage

Conclusions Aesthetic and Minimalist Index Page Design Ability to Make Personalized Collections Provision of a Directory Service URL Recommender System Advanced Faceted Search – Time and Event Based Faster Server Response Time Standard Frequency of Archival Ubiquitous Documentation

Thank You & Audience Questions Tremendous thanks to: Our Project Advisor, Professor Peter Lyman; Internet Archive Founder and Director, Brewster Kahle; all SIMS classmates who participated in our research, and Internet Archive users everywhere. Project Website located at: Q & A ?