1. VELTI Evaluation Methodology
Paris 4th Meeting Dec, Created by VeltiNet

2. Table of Contents
Prototype Evaluation at SCIE02 event
Evaluation Methodology of Web Spidering, Information Extraction and User Interface

3. SCIE02 General On a total population of 10 evaluators participating
Web Spidering
On a total of 9 answers
IE Tool
On a total of 8 answers
Responsiveness: 62.50%
Accuracy: 87.50%, 75.00%
User Interface
On a total of 7 answers
Simplicity: 57.14% (I1)
Responsiveness: 57.14% (I2)
Comprehensibility: 71.3% (I3)
Consistency: 57.14% (I4), 42.86% (I5), 42.86% (I6)
Workflow: 57.14% (I7), 71.3% (I8), 57.14% (I9)
Forgetfulness: 57.14% (I10), 42.86% (I11)
Localisation: 57.14% (I12)
Help: 71.3% (I13), 57.14% (I14)

4. Evaluation Methodology
Scope
Behavioral Tests
Test Cases
Acceptance Criteria

5. Scope Address the quality of the artifacts produced
Address product quality by the attributes outlined in ISO 9126 model
Functionality, Efficiency, Reliability, and Usability
Database consisted of 10 web pages

6. Behavioral Tests Simplicity. Were some parts unnecessarily complex?
Responsiveness. Did the user perceive the system as responsive? If some operation takes long, was the user given a visual indication that the system is progressing, albeit slowly, in its response?
Comprehensibility. Was the user at any time in doubt as to the meaning of some part of the interface?
Consistency. Did the same action lead to the same results everywhere?
Workflow. Did the sequence of the actions the user was forced to go through correspond to the user’s intuitive idea of performing the task? Did each action offer a clear feedback? Was the next action to do evident enough? Was the user’s task performed with the minimum number of steps?
Forgetfulness. If the user happened to do something wrong, was the way to fix it easy? How much time did it took to return to the user’s intended path? Was it possible to restore the system to its past state?
Localisation. Did the interface conform to customary conventions for the user’s language?
Help. Was it easy to obtain help?

7. Test Cases
Web Spidering
Information Extraction
User Interface

8. Web Spidering Scenario 1:
Description: Use the Web Spidering Tool and select one of the existing URLs. While the Web Spidering Tool is processing the selected URL, open the URL with a browser and search it manually in order to find 2 pages presenting laptop offers
Scenario 2:
Description: Use the Web Spidering Tool and select a new URL of your choice. While the Web Spidering Tool is processing the selected URL, open the URL with a browser and search it manually in order to find 2 pages presenting laptop offers

9. Information Extraction
Scenario 1:
Description: Ask the system to extract information from two pages you choose from the ones provided by the system, according to the language you prefer. The pages provided belong to two types:
List A contains single product description pages
List B contains multiple product descriptions pages
Scenario 2:
Description: Ask the system to extract information from one of the pages collected by the Web Spidering Tool, according to the language you prefer

10. User Interface Scenario 1:
Description: Search for laptops suitable for graphic designers. The user is supposed to discover unaided the proper preferences for this search
Scenario 2:
Description: The user is given some vague requirements, not necessarily expressed in the same language used in the interface. This will test how well the interface adapts to the domain
Scenario 3:
Description: The user is to find the most powerful laptop, as he/she understands it, regardless of price
Scenario 4:
Description: The user is to find the absolutely cheapest laptop, regardless of features

11. Acceptance Criteria Not a single formal method or technique
The standardization process is ongoing by the ISO project 'Evaluation and Metrics'
The ISO/IEC 9126 proposes four rating levels: Excellent, Good, Fair and Poor. The first three are considered as satisfactory; the last is considered as unsatisfactory
Test output indicating above 25% efficiency can be considered as an adequate level of conformance to requirements

12. Q & A