1.  Copyright 2008 Digital Enterprise Research Institute. All rights reserved. Digital Enterprise Research Institute www.deri.ie Evaluation of Software Systems Günther Gediga, Kai-Christoph Hamborg, Ivo Düntsch presented by Simon Scerri

2. Digital Enterprise Research Institute www.deri.ie Overview Software Evaluation Evaluation Goals & Approaches Evaluation Criteria Evaluation Techniques  Predictive Evaluation Techniques  Descriptive Evaluation Techniques Evaluation Models Conclusion 2 of 34

3. Digital Enterprise Research Institute www.deri.ie Evaluation Evaluation is a task, which results in one or more reported outcomes Evaluation is an aid for planning, and therefore the outcome is an evaluation of different possible actions Evaluation is goal-oriented. The primary goal is to check results of actions or interventions, in order to improve the quality of the actions or to choose the best action alternative Evaluation is dependent on the current knowledge of science and the methodological standards

4. Digital Enterprise Research Institute www.deri.ie Software Evaluation Software can be evaluated WRT different aspects: functionality, reliability, usability, efficiency, maintainability, portability In this survey, focus is on usability from an ergonomic point of view Software Evaluation should not only take place at the end of development, but serve as a tool for information gathering within iterative design Development of usable software can only be done by a systematic consideration of usability aspects within the life- cycle model Ergonomics: the science of designing the job, equipment, and workplace to fit the worker

5. Digital Enterprise Research Institute www.deri.ie Evaluation Goals & Approaches “Why is it bad?” – to determine the weaknesses of a software such that the result generates suggestions for further development “Which one is better?” - to compare alternative software systems, or different prototypes/versions of a system “How good is it?” - to determine the degree of desired qualities of a finished system Formative Evaluation (early stage) Summative Evaluation (late stage)

6. Digital Enterprise Research Institute www.deri.ie Formative Evaluation Goal: Improvement of software and design supporting aspects. Main part of software evaluation. Important role in iterative system development. In every development cycle, formative evaluation results in  Quantitative data – for the description of the progress of usability goal realisation. Consists of Objective data, i.e. directly observable data such as user behaviour during the use of the interface/application system. Typically in the form of numerical data for statistical analysis, e.g. ratings.  Qualitative data - to detect the usability problems of the system. Consists of Subjective data, i.e. opinions, expressed by users with respect to usability of the interface/application system. Typically non-numerical data, e.g. problems, suggestions.

7. Digital Enterprise Research Institute www.deri.ie Summative Evaluation Global aspects of software development No constructive information for changing the design of the system in a direct manner Performed when the development of the system is almost or entirely accomplished (can apply to prototypes to control effect of design changes in comparison to a preceding version)

8. Digital Enterprise Research Institute www.deri.ie Evaluation Criteria The measurable part of attributes of design or evaluation. Question: What will be measured? The concept of usability is often used to determine the criteria: “ Usability of a product is the extent to which the product can achieve specific goals with effectiveness, efficiency, and satisfaction in a specific context of use.” Characterising usability by “effectiveness”, “efficiency” and “satisfaction” is a first step towards reaching criteria Problem: Evaluation standards do not state how to achieve criteria that fit in the needs of a specific software evaluation task.

9. Digital Enterprise Research Institute www.deri.ie Evaluation Techniques (General) Predictive evaluation techniques:  Expertise-based  Make recommendations for future software development and the prevention of usability errors  Do not need a built system, as empirical methods are replaced by a theory/contribution of experts  User involvement is not as dominant as in descriptive techniques  User participation can be more prominent, as users can play a more active role and influence the development process Descriptive evaluation techniques  Used to describe the status and the actual problems of the software in an objective, reliable and valid way Don’t worry! I’m an expert

10. Digital Enterprise Research Institute www.deri.ie Evaluation Techniques Predictive Evaluation Techniques Descriptive Evaluation Techniques  Behaviour-based – Observational techniques – Thinking-aloud protocols – Video confrontation  Opinion-based – Oral Interview – Written Questionnaires  Usability Testing

11. Digital Enterprise Research Institute www.deri.ie Predictive Evaluation Techniques Usability Walkthroughs  Step by step instructions to check the system under development. Evaluators note problems with task/UI in each step. After data collection, problems are discussed and improvements of the system proposed.  Example of standard walkthrough - Cognitive Walkthrough, evaluates UIs by analysing mental processes required by user. Scope is the ease of learning and learning by exploration. Inspection by Experts and Heuristic reviews  Used for the generation of problem lists, form the basis for system optimisation  Used at an early stage of the life cycle using paper prototypes  Heuristics are general principles which guide evaluators through inspection process Group discussion  As a first step of the evaluation - brainstorming to generate evaluation criteria  Frequently combined with walkthroughs. Don’t worry! I’m an expert

12. Digital Enterprise Research Institute www.deri.ie Evaluation Techniques Predictive Evaluation Techniques Descriptive Evaluation Techniques  Behaviour-based – Observational techniques – Thinking-aloud protocols – Video confrontation  Opinion-based – Oral Interview – Written Questionnaires  Usability Testing

13. Digital Enterprise Research Institute www.deri.ie Descriptive Evaluation Techniques (overview) Behaviour-based evaluation techniques  Record user behaviour while working with a system which “produces” some kind of data  Rely on some form of Observation to detect usability problems  Can only be applied in the later stages of system development, as a prototype is needed Opinion-based evaluation methods  To elicit the user’s (subjective) Opinions, e.g. interviews, surveys and questionnaires. Usability Testing  Stems from classical experimental design studies  Usability Testing is understood to be a combination of Behaviour + Opinion-based measures with some Experimental Control

14. Digital Enterprise Research Institute www.deri.ie Evaluation Techniques Predictive Evaluation Techniques Descriptive Evaluation Techniques  Behaviour-based – Observational techniques – Thinking-aloud protocols – Video confrontation  Opinion-based – Oral Interview – Written Questionnaires  Usability Testing

15. Digital Enterprise Research Institute www.deri.ie Behaviour-based evaluation Observational techniques  Can produce quantitative as well as qualitative data  Avoids subjectivity by standardising procedures and documentation  Direct observation is not possible as the observer cannot deal with amount of information which has to processed.  Indirect techniques (e.g. video recording) are commonly used Thinking-aloud protocols  Provides observer with information about cognitions and emotions of users as they perform tasks  In pre-event and post-event thinking-aloud procedures, users think aloud before a task has started, or after a task has been completed Video confrontation  Based on observational + post-event thinking-aloud technique  Observer picks out interesting recordings and interviews user about them to examine relationships between user errors and emotional reactions

16. Digital Enterprise Research Institute www.deri.ie Evaluation Techniques Predictive Evaluation Techniques Descriptive Evaluation Techniques  Behaviour-based – Observational techniques – Thinking-aloud protocols – Video confrontation  Opinion-based – Oral Interview – Written Questionnaires  Usability Testing

17. Digital Enterprise Research Institute www.deri.ie Opinion based evaluation methods Oral + Written interview techniques to gather user’s opinion about system, which cannot be gathered by observation Used in specification, design, re-engineering phases or for system comparison Questionnaires are better for standardisation and accuracy Standard Questionnaires  QUIS: Questionnaire for User Interface Satisfaction – Measure of overall satisfaction – Evaluates UI aspects based on opinions – Included categories: “User Reactions”, “Screen”, “Terminology & System Info”, “Learning”, “System Capabilities”  SUMI: The Software Usability Measurement Inventory – Summative instrument to measure user’s perception of software usability – Included categories: “Global Efficiency”, “Effect”, “Helpfulness”, “Control”, “Learnability”  IsoMetrics Two versions, based on the seven design principles of ISO 9241-110 – IsoMetrics (short) supports summative evaluation of software systems – IsoMetrics (long) for formative evaluation  Many others…

18. Digital Enterprise Research Institute www.deri.ie Evaluation Techniques Predictive Evaluation Techniques Descriptive Evaluation Techniques  Behaviour-based – Observational techniques – Thinking-aloud protocols – Video confrontation  Opinion-based – Oral Interview – Written Questionnaires  Usability Testing

19. Digital Enterprise Research Institute www.deri.ie Usability Testing Systematic, Rigid, Experimentally-based gathering of information about a product/prototype Combines Behaviour and Opinion-based measures with some amount of experimental control as chosen by an expert. Problem of defining measurables, or dependent variables. Chosen pragmatically according to evaluation goals Usability Testing requires a large amount of expertise to  Set up experimental design  Choose the suitable tasks for comparison  Select the users and the number of users  Define the measurables properly Best-suited for usability engineers Not a suitable technique for untrained evaluators.

20. Digital Enterprise Research Institute www.deri.ie Evaluation Technique Comparison Predictive techniques concentrate on severe problems only, descriptive techniques address more specific (and cosmetic) aspects Predictive evaluation techniques can be applied earlier in the life-cycle than Usability Testing procedures Usability Testing results in more information than predictive techniques Usability Testing needs more effort and equipment than predictive techniques Behavioural evaluation procedures applied in Usability Testing are best suited for summative evaluation Usability Testing & Expert-based techniques do not support participatory design. Collaborative Walkthroughs or Group discussion techniques enable user to be active part in the system development

21. Digital Enterprise Research Institute www.deri.ie Evaluation Models Software Evaluation Models determine the frame of the evaluation which consists of  Choosing techniques appropriate for the life-cycle  Setting the focus with respect to the object under study and the measurement criteria Three classes of evaluation models  Method-driven models  Criteria-driven models  Usability Engineering

22. Digital Enterprise Research Institute www.deri.ie Method-driven models Arrangement of evaluation techniques, amended by the regulation of preparatory and subsequent tasks An example: EVADIS II  Well tested for office automatisation software.  Model combines interviews, task analysis, and expert judgement: 1.Installation and exploration 2.Construction of test tasks 3. Selection of ergonomic test items based on user characteristics 4. Evaluation (by an expert) 5. Interpretation of results in a test report  1-3 (in parallel) result in testing tasks and ranked ergonomic evaluation criteria, based on ISO 9241 (Part 10)  In 4, test tasks are evaluated by an expert who answers the questions formulated in the ergonomic items  Recorded answers form the basis for the test report

23. Digital Enterprise Research Institute www.deri.ie Start with assumptions about structure of the design process in which criteria are defined Give advice how to derive measurables from the criteria Standards such as ISO 9241 or ISO/IEC 9126 can constitute a basis for a criteria driven evaluation model. An example: the Evaluation Process Model’s (basis for many software quality assurance procedures) 3 phases: 1. Requirement Definition: Requirements derived from the application context 2. Preparation: Operationalisation of the criteria into metrics and composite measurables. 3. Evaluation: Performed on the basis of the derived requirements Criteria-driven models

24. Digital Enterprise Research Institute www.deri.ie Usability Engineering Requires Software Engineering Model allowing for feedback loops Systematic Integration of methods and techniques for the building of usable software in the System Development Process User-Centred Design based on the assumption of an iterative system design with user participation User is not a producer of usability data, but is directly and actively involved in the design Models of Usability Engineering subdivided in 3 phases (closely connected to ISO/DIS 13407 Part 37) : 1. Analysis/Specification - Develop usability specifications consisting of user-oriented design goals 2. Iterative development – Continuous evaluation + prototype revision. Cycle repeated until goals of user-oriented design fulfilled 3. System Installation – Assure system acceptance. Organise system support and maintenance. Feedback collection for new releases

25. Digital Enterprise Research Institute www.deri.ie Conclusions… Most software evaluation techniques aim at the operationalisation of “Usability” of the software Many evaluation techniques, Few standard evaluation models Selecting evaluation techniques  Need to investigate where techniques will fit into development cycle  Rule of thumb “early  Predictive & late  Descriptive” should be replaced by a more differentiated rule system Usability is a combination of at least 2 different kinds of constructs  Ease of use  Enjoyment of use Evaluation  is NOT a process to be performed only at the End of Development  Should form Part of the Life-Cycle  Should involve User Participation Actively, not just passively Operationalisation - process of turning abstract concepts into observable and measurable quantities required for data collection Focus of newer evaluation studies. Milestone: SUN and SAP proclaimed that even office software should be enjoyable

26. Digital Enterprise Research Institute www.deri.ie Noteworthy References The survey paper itself, as a survey of software system evaluation criteria, techniques, and models  Gediga, G. et. al (2001). Evaluation of Software Systems International Standards:  ISO 9241 Ergonomics of Human System Interaction – Part 11: Guidance on usability – Part 110 (previously 10): Dialogue principles  ISO/DIS 13407 Human centred design processes for interactive systems Nielsen  Nielsen, J. (1992). The usability life cycle  Nielsen, J. (1993b). Usability Engineering Online resources:  Comparison/generator of Standard Web-Based Questionnaires for User Interface Usability Evaluation http://hcibib.org/perlman/question.htmlhttp://hcibib.org/perlman/question.html