1 ©Serco Reproduction permitted provided the source is acknowledged Formalising usability as part of ISO standards Nigel Bevan Serco Usability Services UK

2 ©Serco Reproduction permitted provided the source is acknowledged Topics  What is usability?  Definitions of usability? ISO and ISO  Specifying usability as quality in use: ISO , the Common Industry Format and ISO  Usability of everyday products: ISO  User interface usability: ISO 9241  User centred design process: ISO  Usability capability: ISO TR 18529

3 ©Serco Reproduction permitted provided the source is acknowledged What is usability?  what do you think usability is......?

4 ©Serco Reproduction permitted provided the source is acknowledged IBM  “IBM's focus on ease of use has led to technology breakthroughs and advances in design research”  “For developers and manufacturers, the advantages of creating usable products far outweigh the costs”  “Every dollar invested in ease of use returns $10 to $100”

5 ©Serco Reproduction permitted provided the source is acknowledged Microsoft  “The Usability Group is an integral part of the product design process at Microsoft”  Over 120 usability engineers  Over 25 usability labs

6 ©Serco Reproduction permitted provided the source is acknowledged Why is usability important?  Focus on user and organisational needs  reduce development times  less training and support and documentation is required  Improve productivity  simpler interface, fewer user errors  Improve the competitive edge  increasing expectations for ease of use  increasing usability of competitive products  high profile of usability in advertising  Improve the quality of life  less stress, users are more satisfied  lower staff turnover  Health and safety legislation  European Directive on Display Screen Equipment

7 ©Serco Reproduction permitted provided the source is acknowledged Case study: Hewlett Packard User-centred design methods were applied to redesign software used for identifying network problems: Old New time to finish task9.4 min 4.1 min problems identified16% 78% average length of call30 min 10 min size of manual25 pages 4 pages people needing the manual53% 3% user satisfaction rating In addition to the benefits to customers, HP recovered their costs in 18 months

8 ©Serco Reproduction permitted provided the source is acknowledged ISO/IEC 9126 Software Product Quality Model functionalityreliability usability efficiency maintainabili ty portability accuracy suitability interoperability security understandability learnability operability analysability changeability stability testability maturity fault tolerance recoverability availability time behaviour resource utilisation adaptability installability co-existence replaceability

9 ©Serco Reproduction permitted provided the source is acknowledged ISO/IEC Definitions  Functionality  The capability of the software product to provide functions which meet stated and implied needs when the software is used under specified conditions.  Reliability  The capability of the software product to maintain a specified level of performance when used under specified conditions  Usability  The capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions.  Efficiency  The capability of the software product to provide appropriate performance, relative to the amount of resources used, under stated conditions.  Maintainability  The capability of the software product to be modified. Modifications may include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specifications.  Portability  The capability of the software product to be transferred from one environment to another.

10 ©Serco Reproduction permitted provided the source is acknowledged Usability: The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. Effectiveness: The accuracy and completeness with which users achieve specified goals. Efficiency: The resources expended in relation to the accuracy and completeness with which users achieve goals. Satisfaction: The comfort and acceptability of use. ISO Guidance on Usability

11 ©Serco Reproduction permitted provided the source is acknowledged ISO/IEC Software Product Quality Model quality in use functionalityreliability usability efficiency maintainabilityportability accuracy suitability interoperability security understandability learnability operability attractiveness analysability changeability stability testability maturity fault tolerance recoverability availability time behaviour resource utilisation adaptability installability co-existence replaceability

12 ©Serco Reproduction permitted provided the source is acknowledged Quality in Use ISO/IEC (2001) Software Engineering - Product quality - Part 1: Quality model quality in use the capability of the software product to enable specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in specified contexts of use  Quality in use is the user’s view of the quality of a system containing software, and is measured in terms of the result of using the software, rather than properties of the software itself  Quality in use measures the combined effect of ease of use, functionality, efficiency and reliability.

13 ©Serco Reproduction permitted provided the source is acknowledged Approaches to achieving usability process quality product quality quality in use usability in context development process product effect of the product user centred process interface and interaction ISO ISO ISO ISO/IEC ISO/IEC ISO organisational capability life cycle processes usability capability ISOTR ISO 9241 parts 10, ISO/IEC /3

14 ©Serco Reproduction permitted provided the source is acknowledged Approaches to achieving usability 1Design for quality in use  ISO , ISO : effectiveness, efficiency, (safety), satisfaction  ISO Usability of everyday products 2Design the user interface  ISO 9241 parts 12 to 17 Ergonomic principles 3Use user-centred methods in development  ISO Human centred design processes 4Establish organisational processes  ISO TR Human-centred lifecycle process descriptions

15 ©Serco Reproduction permitted provided the source is acknowledged requirements User requirements Functional requirements Detailed specification inspection evaluation product Deliver Prototype Code What can be specified and measured? Test user function Test user goal Internal usability metrics ISO/IEC requirements External usability metrics ISO/IEC Quality in use metrics ISO/IEC

16 ©Serco Reproduction permitted provided the source is acknowledged 1. Design for quality in use Usability/quality in use measures:  Effectiveness and efficiency  MUSiC Performance Measurement Method  Satisfaction  SUMI questionnaire  Reporting usability  Common industry format for usability test results  includes intended context of use and actual context of evaluation

17 ©Serco Reproduction permitted provided the source is acknowledged ISO/IEC CD 25030: Software quality requirements and evaluation – Quality requirements  Requirements can be categorised as:  functional requirements  quality requirements  other non-functional requirements  Software quality requirements are categorised using the ISO/IEC software quality model into requirements for:  quality in use (the effectiveness, productivity, safety and satisfaction with the software applied in its context of use)  external quality (the behaviour of the finished software product)  internal quality (static properties during development, such as adherence to coding standards, and meeting size and complexity thresholds)

18 ©Serco Reproduction permitted provided the source is acknowledged Requirements life cycle model Definition Measured quality Stated requirements Formalisation Evaluation Measurable requirements Internal quality External quality Quality in use Quality requirements Stake- holder needs Measured quality Developed system Functional requirements Other non-functional requirements

19 ©Serco Reproduction permitted provided the source is acknowledged Quality requirements specification  Define quality requirements  Identify stakeholders   end users, developers, producers, trainers, maintainers, disposers, acquirer, supplier organisations and regulatory bodies  Elicit requirements from stakeholders  Quality in use, External and Internal Quality  Analyse the set of requirements  Resolve problems  Confirm requirements  Record requirements  Formalise quality requirements  Specify target values for measures  Demonstrate traceability  Maintain requirements

20 ©Serco Reproduction permitted provided the source is acknowledged Elicit requirements from stakeholders  Include the needs, wants, desires, expectations and perceived constraints  Identify requirements  Quality in use  External  Internal  Identify constraints  Define scenarios of use

21 ©Serco Reproduction permitted provided the source is acknowledged Quality in use Extent to which a product used by specified users meets their needs to achieve specified goals with effectiveness, productivity and satisfaction in specified contexts of use  Effectiveness: accuracy and completeness  Productivity: the resources expended by users in relation to the effectiveness achieved

22 ©Serco Reproduction permitted provided the source is acknowledged Why is quality in use important?  It is about meeting business and user needs  Effectiveness: success in achieving goals  Productivity: staff time  Satisfaction: willingness to use the system

23 ©Serco Reproduction permitted provided the source is acknowledged Quality in use: context of Use  The usability of a product is affected not only by the features of the product itself but also by its Context of Use  Context is the characteristics of:  the users of the product  the tasks they carry out  the technical, organisational and physical environment in which the product is used  the date and time when the product is being used

24 ©Serco Reproduction permitted provided the source is acknowledged Example: A bank ATM Context description:  The users  The user characteristics  The tasks users perform  The technical environment (hardware and software to support system)  The physical environment  The social or organisational environment

25 ©Serco Reproduction permitted provided the source is acknowledged New design features to meet contextual needs  Recess for wheelchair access.  Speech output for visually impaired users.  Customisation features for rapid access.  Finger print for identification.  Visor appears during sunny weather.  Buttons light during darkness.  Alarm button for security alert.

26 ©Serco Reproduction permitted provided the source is acknowledged Efficiency: mean transaction times

27 ©Serco Reproduction permitted provided the source is acknowledged User satisfaction: SUMI questionnaire GlobalEfficiencyAffectHelpfulnessControlLearnability Old System New System

28 ©Serco Reproduction permitted provided the source is acknowledged The danger of not setting quality in use requirements  New software for issuing UK passports  Installed in passport issuing offices  Took operators twice as long  Caused delays of up to 3 months in obtaining a passport  Huge cost of additional clerical staff

29 ©Serco Reproduction permitted provided the source is acknowledged Example of quality in use requirements  User performance  “all data entry clerks will be able to complete the task with at least 95% accuracy in under 10 minutes”  User satisfaction  “the mean score on the SUMI scale will be greater than 50” More information on quality in use requirements

30 ©Serco Reproduction permitted provided the source is acknowledged Common Industry Format usability tests  NIST initiative  National Institute of Standards and Technology  Objective: raise the profile of usability in procurement  Suppliers provide standard test reports to purchasers  Suppliers include:  IBM, Microsoft, HP, Sun, Kodak, Oracle, Compaq  Purchasers include:  Boeing, Northwest Mutual Life, State Farm Insurance, Fidelity  Reports provided in confidence  Could permit comparisons  Agreed as US standard ANSI/NCITS 354, to be submitted to ISO

31 ©Serco Reproduction permitted provided the source is acknowledged Report format - technical  Test objectives  Participants  Total number and user groups  Key characteristics and capabilities expected  How selected: with the key characteristics and capabilities?  Context of evaluation  Task scenarios and task data used  Any performance criteria used  Physical and computer environment

32 ©Serco Reproduction permitted provided the source is acknowledged Report format - results  Effectiveness  % completely and correctly achieving their goals  Separate data where assistance was given  Efficiency  Mean time taken to complete the task  Standard deviation (and standard error)  Satisfaction  Optional: own scales or standard questionnaire

33 ©Serco Reproduction permitted provided the source is acknowledged ISO 20282: Usability of everyday products Part 1: Universal user profile Design should take account of  Strength and Biomechanical Abilities  Handedness  Body Dimensions.  Visual Abilities.  Auditory Abilities  Cognitive Abilities  Language and literacy  Culture  Age  Gender

34 ©Serco Reproduction permitted provided the source is acknowledged ISO WD : Test method  A summative test method for measuring usability based on  ISO (Guidance on usability)  ISO (Quality in use metrics)  Common Industry Format for usability test reports  Identify intended context of use and user groups  Measures  Success rate (ease of operation)  Task time (efficiency)  Satisfaction scale (being developed)

35 ©Serco Reproduction permitted provided the source is acknowledged Approaches to achieving usability process quality product quality quality in use usability in context development process product effect of the product user centred process interface and interaction ISO ISO ISO ISO/IEC ISO/IEC ISO organisational capability life cycle processes usability capability ISOTR ISO 9241 parts 10, ISO/IEC /3

36 ©Serco Reproduction permitted provided the source is acknowledged 2. Design the user interface ISO 9241: Ergonomic requirements for office work with visual display terminals Parts 10 to 17 give guidance on software 10 Dialogue principles 11 Guidance on usability 12 Presentation of information 13 User guidance 14 Menu dialogues 15 Command Language dialogues 16 Direct Manipulation dialogues 17 Form-filling dialogues

37 ©Serco Reproduction permitted provided the source is acknowledged Approaches to achieving usability process quality product quality quality in use usability in context development process product effect of the product user centred process interface and interaction ISO ISO ISO ISO/IEC ISO/IEC ISO organisational capability life cycle processes usability capability ISOTR ISO 9241 parts 10, ISO/IEC /3

38 ©Serco Reproduction permitted provided the source is acknowledged 3. ISO 13407: Human centred design process for interactive systems 1. Plan the human centred process 2. Specify the context of use 4. Produce design solutions 3. Specify user and organisational requirements 5. Evaluate designs against user requirements Meets requirements

39 ©Serco Reproduction permitted provided the source is acknowledged ISO Scope  Computer-based interactive systems  The detailed recommendations are applicable to computer hardware and software  Can apply the same principles to the design of any system for use by humans  A pen  A chair  A meeting room  A conference

40 ©Serco Reproduction permitted provided the source is acknowledged TRUMP: Trial Usability Maturity Process  EU-funded trial application of user-centred design methods developed in previous research projects (INUSE and RESPECT)  Serco: apply the methods  Lloyds Register: Usability Maturity Assessment  Inland Revenue/EDS - IT for 60,000 staff  RAD methodology  Israel Aircraft Industries - aerospace systems  traditional methodology

41 ©Serco Reproduction permitted provided the source is acknowledged Typical approach to usability feasibilityrequirementsdesignimplementrelease Usability group user and task analysis usability test prototyping

42 ©Serco Reproduction permitted provided the source is acknowledged TRUMP approach: “ubiquitous usability” 1. Stake- holder meeting 2. Context of use 3. Scenarios 4. Usability requirements 5. Evaluate existing system 6. Prototyping 7. Style guide 8. Evaluation 9. Usability testing 10. Collect feedback feasibilityrequirementsdesignimplementrelease System lifecycle Plan Process Specify Context of Use Design Solution s Specify Requirements Evaluate against Requirements

43 ©Serco Reproduction permitted provided the source is acknowledged requirements User requirements Functional requirements Detailed specification Function test evaluationproduct Deliver Prototyp e Code Conventional design process How do we incorporate user centred design?

44 ©Serco Reproduction permitted provided the source is acknowledged requirements User requirements Functional requirements Detailed specification evaluationproduct 2. Intended context of use Context used for evaluation 6. Mock up Use in context Prototyp e Code 6. Paper prototype 6. Paper specificatio n 3. Intended scenarios of use Scenarios used for evaluation 1. Plans 4. Quality in use requirements 4. External usability requirements 4. Internal usability requirements 5. Evaluate existing system Function test 7. Style guide Inspection 7. Style guide

45 ©Serco Reproduction permitted provided the source is acknowledged Methods: feasibility 1. Stakeholder meeting A half-day meeting to identify and agree on the role of usability, broadly identifying the intended context of use and usability goals, and how these relate to the business objectives and success criteria for the system. 2. Context of use A half-day workshop to collect and agree detailed information about the intended users, their tasks, and the technical and environmental constraints. 3. Scenarios of use A half day workshop to document examples of how users are expected carry out key tasks in a specified contexts, to provide an input to design and a basis for subsequent usability testing.

46 ©Serco Reproduction permitted provided the source is acknowledged Methods: requirements 4. Evaluate an existing system Evaluate an earlier version or competitor system to identify usability problems and obtain measures of usability as an input to usability requirements. 5. Usability requirements A half-day workshop to establish requirements for effectiveness, efficiency and satisfaction for the user groups and tasks identified in the context of use analysis and in the scenarios.

47 ©Serco Reproduction permitted provided the source is acknowledged Methods: design 6. Paper prototyping Evaluation by users of quick low fidelity prototypes (using paper or other materials) to clarify requirements and enable draft interaction designs and screen designs to be rapidly simulated and tested. 7. Style guide Identify, document and adhere to industry, corporate or project conventions for screen and page design.

48 ©Serco Reproduction permitted provided the source is acknowledged Methods: evaluation 8. Evaluation of machine prototypes Informal usability testing with 3-5 representative users carrying out key tasks to provide rapid feedback on the usability of prototypes. 9. Usability testing Formal usability testing with 8 representatives of a user group carrying out key tasks to identify any remaining usability problems and evaluate whether usability objectives have been achieved. 10. Collect feedback from users Collect information from sources such as usability surveys, help lines and support services to identify any problems that should be fixed in future versions.

49 ©Serco Reproduction permitted provided the source is acknowledged Cost benefits at IAI  Development cost/benefits  A all the methods used (except Style Guide) resulted in savings in development costs of between $5K and $70K for each method, with a total saving of $330K.  The cost of using the methods was only $22K, giving a cost-benefit ration of 1:15.  Sales cost/benefits  Increased sales because customers would be more likely to buy the product were estimated to be $400K  Support cost/benefits  Reduced costs of training and support were estimated at £50K.  Overall cost benefits  The overall costs of the maturity assessments and use of methods was $27K. The total estimated savings and increased sales is $780K, giving a cost-benefit ration of 1:29.

50 ©Serco Reproduction permitted provided the source is acknowledged Trial Application at IAI Bottom Line  Positive Feedback from Participants  A Definite Improvement in the Development Process  Very Cost Effective and Low Cost  Mostly Intuitive however tailoring sometimes required  Expert Guidance needed in few techniques  LAHAV decided to incorporate TRUMP techniques in it’s standard development process

51 ©Serco Reproduction permitted provided the source is acknowledged IAI Conclusions  Low cost simple methods for improving systems usability are here  The methods can be easily integrated into any software development process  The responsibility for quality in use should be transferred to development organization  The Methods and supporting means are available on the Web  No more Excuses

52 ©Serco Reproduction permitted provided the source is acknowledged Approaches to achieving usability process quality product quality quality in use usability in context development process product effect of the product user centred process interface and interaction ISO ISO ISO ISO/IEC ISO/IEC ISO organisational capability life cycle processes usability capability ISOTR ISO 9241 parts 10, ISO/IEC /3

53 ©Serco Reproduction permitted provided the source is acknowledged 4. Establish usability capability ISO TR 18529: Human-centred lifecycle process descriptions HCD.1 Ensure HCD content in system strategy HCD.2 Plan and manage the HCD process HCD.3 Specify the user and organisational requirements HCD.4 Understand and specify the context of use HCD.5 Produce design solutions HCD.6 Evaluate designs against requirements HCD.7 Introduce and operate the system

54 ©Serco Reproduction permitted provided the source is acknowledged Process improvement overview 1. Identify needs for usability process improvement Use the usability maturity model in ISO TR to assess the current capability of each organisation Use the usability maturity model in ISO TR to assess the current capability of each organisation 2. Make the identified improvements to the software development processes Introduce simple user-based methods implementing ISO Introduce simple user-based methods implementing ISO Reassess the usability capability the organisation Assess the extent of the improvement Assess the extent of the improvement 4. Identify the cost-benefits of the improvements Integrate the methods into the documented processes Integrate the methods into the documented processes

55 ©Serco Reproduction permitted provided the source is acknowledged Usability Maturity Model HCD.2 Plan and manage the HCD process  Consult stakeholders.  Identify and plan user involvement.  Select human-centred methods and techniques.  Ensure a human-centred approach within the project team.  Plan human-centred design process.  Manage human-centred process.  Champion human-centred activities.  Provide support for human-centred design

56 ©Serco Reproduction permitted provided the source is acknowledged Usability Maturity Model HCD.3 Specify the user and organisational requirements  Clarify and document the system goals.  Define stakeholders.  Assess risk to stakeholders.  Define the system.  Generate the user and organisational requirements.  Set usability objectives.

57 ©Serco Reproduction permitted provided the source is acknowledged Usability Maturity Model HCD.4 Understand and specify the context of use Identify and document  user’s tasks.  significant user attributes.  organisational environment.  technical environment.  physical environment.

58 ©Serco Reproduction permitted provided the source is acknowledged Usability Maturity Model HCD.5 Produce design solutions  Allocate functions  Produce composite task model  Produce system design  Use existing knowledge to develop design solutions  Specify system  Develop prototypes  Provide user training  Provide user support

59 ©Serco Reproduction permitted provided the source is acknowledged Usability Maturity Model HCD.6 Evaluate designs against requirements  Specify and validate context of evaluation.  Evaluate early prototypes in order to define the requirements for the system.  Evaluate prototypes in order to improve the design.  Evaluate the system in order to check that the system requirements have been met.  Evaluate the system in order to check that the required practice has been followed.  Evaluate the system in use in order to ensure that it continues to meet organisational and user needs.

60 ©Serco Reproduction permitted provided the source is acknowledged l United Kingdom’s tax collection department l Employ over 60,000 staff in more than 600 local offices l Partnership with EDS for software development now running for more than 5 years l JAD: Joint Application Development l RAD: Rapid Application Development

61 ©Serco Reproduction permitted provided the source is acknowledged Usability capability assessment Formal CMM/SPICE style assessment by Lloyds Register  Interviewed 13 stakeholders in twelve 3-hour sessions over one week.  A conventional software process assessment procedure based on SPICE (ISO 15504) was used.  Produced a detailed profile and rich information about where improvements would be beneficial.  At a feedback meeting on the last day, it provided the basis for an agreed set of improvement activities  A second assessment a year later showed significant improvements

62 ©Serco Reproduction permitted provided the source is acknowledged Inland Revenue/EDS Conclusions 4 Significant improvement in all areas of usability maturity 4 IR/EDS now strong in the “core” areas of context, requirements, design and evaluation 4 Evaluation activity rated as outstanding 4 The methods are very cost effective and will be incorporated into the documented development process 4 “A step change in the awareness and practice of human centred design within IR/EDS”.

63 ©Serco Reproduction permitted provided the source is acknowledged IAI LAHAV Background  Expertise in Military Aircraft “Avionics Upgrade Programs”  Customers Worldwide  User needs addressed by a group of Pilots  Trial Application: PC-based Mission Planning system

64 ©Serco Reproduction permitted provided the source is acknowledged Usability capability assessment workshop Process improvement workshop  One-day workshop  Review each process and activity in ISO TR in detail  Score each process as:  Not, Partly, Largely or Fully performed  Note whether the process is managed  a documented part of normal procedures  Establish a target level for each process on the same scales  A second assessment a year later showed significant improvements

65 ©Serco Reproduction permitted provided the source is acknowledged Trial Application at IAI Bottom Line  Positive Feedback from Participants  A Definite Improvement in the Development Process  Very Cost Effective and Low Cost methods  LAHAV decided to incorporate TRUMP techniques in it’s standard development process

66 ©Serco Reproduction permitted provided the source is acknowledged Process improvement conclusions  The usability maturity model was a valuable tool for identifying needs for process improvement  Inland Revenue:summative assessment requiring three person weeks effort  IAI: a simpler formative one-day assessment  A very promising way to make major process improvements

67 ©Serco Reproduction permitted provided the source is acknowledged ISO Conclusions  ISO and ISO TR provide an internationally-agreed description of good practice in user-centred design  The content can be used to specify or evaluate:  How usability should be incorporated into the development process  Whether a project has applied user centred design  Organisational capability in user-centred design  Basis for process improvement  Appropriate content for education and training courses in user centred design  Necessary areas of professional competence: professional accreditation

68 ©Serco Reproduction permitted provided the source is acknowledged