Cost-Justifying Usability Javad Sadeghi CPSC 681 Nov 5, 2007

Outline Motivation Background Discount usability Samples Methodology Worked example

Motivation Dennis Wixon, SIGCHI Bulletin, Volume 27, Number 2, April (1995), p94

View from the other side of the table For each dollar a company invests in developing the usability of a product, the company receives $10-$100 in benefits and wins customer satisfaction and continued business For each dollar spent to fix a problem during product design, $10 are spent to fix the same problem in product development, and $100 or more are spent to fix the same problem after product release Claire Marie Karat, A business case approach to usability cost justification. In, R. Bias and D. Mayhew, Eds. Cost-Justifying Usability, Academic Press, NY, 1994.

Background Cost-benefit of usability (Marilyn Mantei and Toby Teorey) Business case approach (Clare-Marie Karat ) Cost-justifying usability (Bias and Mayhew) Discount usability (Jacob Nielsen) Web-based applications Bias R. G., Mayhew D. J. Cost-Justifying Usability: An Update for the Internet Age, 2nd Edition. Morgan Kaufmann Publishers, 2005

Discount usability “Le mieux est l’ennemi du bien” Focuses on “the good”:  Reduces the costs  Requires less experience and knowledge Three techniques:  Scenarios  Simplified thinking aloud  Heuristic evaluation Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Scenarios Cuts down complexity by eliminating parts of full system Small and cheap to design and implement  Paper mock-ups Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Simplified thinking aloud Thinking aloud studies, videotape subjects, perform detailed protocol analysis Simplified version  Take notes  3~5 test user per test Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Heuristic evaluation Finding usability problems Evaluator inspect alone Current usability guidelines: 1000 rules 10 basic usability principles Easy to learn Experts best Non-experts can find most problems Remaining, using simplified thinking aloud Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Nielsen’s surveys on cost-justifying usability Usability costs  ~ 10% of the budget of each project  Cost does not increase linearly with the budget Usability benefits  135% usability increase after redesign Return on investment (ROI)  It is not linear with the costs  The more users, the bigger ROI Jakob Nielsen's Alertbox,

Samples of cost-justifying usability Australian insurance company  Usability-enhancing project, costing < A$ 100,000 Average of 7.8 errors per form More than one hour to repair the errors of each form  Annually saving as a result > A$ 530,000 Redesigning application forms IBM employees login system  Usability project, financing $ 20,700 Making sign-on attempts faster  Savings in work time during the first day of the new system $ 41,700 Nielsen J., Usability Engineering. Academic Press, 1993.

Methodology Cost-justifying usability is  A conservative approach  Part of project planning Mayhew and Mantei’s framework  A basic framework for cost-justifying usability  Estimating costs  Estimating benefits  Justifying the assumptions made for benefits Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Estimating costs Steps  Personnel hours and equipment costs Needs experience Additional one time setup  Lab setup  Development of survey  Pilot testing  Distribution and collection  Responding  Coding and entering data  Analyzing results  Computer time  Supplies and duplicating costs Usability tasks (e.g. user profile)  Techniques (e.g. user questionnaire) Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Estimating benefits Relevant audience  Internal development organization  Vendor company Relevant categories of benefits Estimation  Best measurement unit  Assuming magnitudes  Increased user productivity  Decreased user errors  Decreased training costs  Savings gained from making changes earlier in design  Decreased user support  Increased sales  Decreased customer support  Savings gained from making changes earlier in design  Reduced cost of providing training Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Example Data entry system for 250 users Two to four primary screens Process 60 of them per day Users work 230 days a year Total cost per hour of $25 The benefit of 1 second speedup in year 250 users * 60 screens * 230 days * 1/3600 hours * $25 = $23,958 / year Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Justifying the assumptions made for benefits Reasonable and likely minimum benefits Not precise, proven and specific Basing the predictions on known facts  Published research  Actual case histories  Previous experience Randolph G. Bias, Deborah J. Mayhew, Cost-Justifying Usability. Academic Press, 1994.

Cost-benefit analysis of heuristic evaluation: a case study Case study: integrating system  For internal telephone company 11 evaluators Find 44 usability problems:  40 core problems  4 from other part of the interface Estimating the costs (evaluation time) Estimating the benefits (increased usability) Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Estimates of the times spent Assessing appropriate ways to use heuristic evaluation, 4 2 hours8 Having outside evaluation expert learn about the domain and scenario8 Finding and scheduling evaluators, 1.8 hours hours per evaluator4 Preparing the briefing3 Preparing scenario for the evaluators2 Briefing, 1 system expert, 1 evaluation expert, hours19.5 Preparing the prototype (software and its hardware platform) for the evaluation5 Actual evaluation, 11 1 hour11 Observing the evaluation sessions, 2 11 hours22 Debriefing, 3 evaluators, 3 developers, 1 evaluation 1 hour7 Writing list of usability problems based on notes from evaluation sessions2 Writing problem descriptions for use in severity-rating questionnaire6 Severity rating, hours5.5 Analyzing severity ratings2 Total105 ► ► ► Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Time expenditure time(i) = i (Eq1)  i: number of evaluators  Not exact for larger i Reduction: reducing 2 observers to 1 Revised: time(i) = i (Eq2) Estimated hourly loaded cost: $100 Cost estimation: $10,500 Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Benefit estimation Estimations of evaluators for improvements Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Benefit estimation (cont.) Mean values: 0.8 days -learning time reduction 18%-expert speedup To be conservative: 0.5 days - learning time reduction 10%- expert speedup (3.3% of total work time) Assuming 2000 users (from 3000) User-days saved 1,000 User-years saved 67 = 13,000 user days Total user days saved = 14,000 Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Convert to monetary terms User-day cost of $100 With actual saving of 50% One-year saving $540,000 Net present value (benefit): $500,000 Benefit-cost ratio: 48 Nielsen J., Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier

Thanks! Questions?