VELTI Evaluation Methodology Paris 4th Meeting 5-6 Dec, 2002 Created by VeltiNet

Table of Contents Prototype Evaluation at SCIE02 event Evaluation Methodology of Web Spidering, Information Extraction and User Interface

SCIE02 General On a total population of 10 evaluators participating Web Spidering On a total of 9 answers IE Tool On a total of 8 answers Responsiveness: 62.50% Accuracy: 87.50%, 75.00% User Interface On a total of 7 answers Simplicity: 57.14% (I1) Responsiveness: 57.14% (I2) Comprehensibility: 71.3% (I3) Consistency: 57.14% (I4), 42.86% (I5), 42.86% (I6) Workflow: 57.14% (I7), 71.3% (I8), 57.14% (I9) Forgetfulness: 57.14% (I10), 42.86% (I11) Localisation: 57.14% (I12) Help: 71.3% (I13), 57.14% (I14)

Evaluation Methodology Scope Behavioral Tests Test Cases Acceptance Criteria

Scope Address the quality of the artifacts produced Address product quality by the attributes outlined in ISO 9126 model Functionality, Efficiency, Reliability, and Usability Database consisted of 10 web pages

Behavioral Tests Simplicity. Were some parts unnecessarily complex? Responsiveness. Did the user perceive the system as responsive? If some operation takes long, was the user given a visual indication that the system is progressing, albeit slowly, in its response? Comprehensibility. Was the user at any time in doubt as to the meaning of some part of the interface? Consistency. Did the same action lead to the same results everywhere? Workflow. Did the sequence of the actions the user was forced to go through correspond to the user’s intuitive idea of performing the task? Did each action offer a clear feedback? Was the next action to do evident enough? Was the user’s task performed with the minimum number of steps? Forgetfulness. If the user happened to do something wrong, was the way to fix it easy? How much time did it took to return to the user’s intended path? Was it possible to restore the system to its past state? Localisation. Did the interface conform to customary conventions for the user’s language? Help. Was it easy to obtain help?

Test Cases Web Spidering Information Extraction User Interface

Web Spidering Scenario 1: Description: Use the Web Spidering Tool and select one of the existing URLs. While the Web Spidering Tool is processing the selected URL, open the URL with a browser and search it manually in order to find 2 pages presenting laptop offers Scenario 2: Description: Use the Web Spidering Tool and select a new URL of your choice. While the Web Spidering Tool is processing the selected URL, open the URL with a browser and search it manually in order to find 2 pages presenting laptop offers

Information Extraction Scenario 1: Description: Ask the system to extract information from two pages you choose from the ones provided by the system, according to the language you prefer. The pages provided belong to two types: List A contains single product description pages List B contains multiple product descriptions pages Scenario 2: Description: Ask the system to extract information from one of the pages collected by the Web Spidering Tool, according to the language you prefer

User Interface Scenario 1: Description: Search for laptops suitable for graphic designers. The user is supposed to discover unaided the proper preferences for this search Scenario 2: Description: The user is given some vague requirements, not necessarily expressed in the same language used in the interface. This will test how well the interface adapts to the domain Scenario 3: Description: The user is to find the most powerful laptop, as he/she understands it, regardless of price Scenario 4: Description: The user is to find the absolutely cheapest laptop, regardless of features

Acceptance Criteria Not a single formal method or technique The standardization process is ongoing by the ISO project 'Evaluation and Metrics' The ISO/IEC 9126 proposes four rating levels: Excellent, Good, Fair and Poor. The first three are considered as satisfactory; the last is considered as unsatisfactory Test output indicating above 25% efficiency can be considered as an adequate level of conformance to requirements

Q & A