© Simeon Keates 2008 Usability with Project Lecture 19 – 19/11/08 Dr. Simeon Keates

© Simeon Keates 2008 Exercise  Start/continue your user trials  If you want feedback on your group presentation – then prepare slides for Friday NOTE – this is optional, but recommended Page 2

© Simeon Keates 2008 Cost-justifying usability Page 3

© Simeon Keates 2008 The need to cost-justify  Not all companies (and managers) appreciate the benefits of usability  They may cite other factors as more important  Examples: Tight deadlines Functionality already developed Limited money/resources Page 4

© Simeon Keates 2008 Cost-justifying usability  However, these are often “false economies”  Examples: Deadlines: Releasing the “wrong” product on time is as bad (or worse) as releasing the “right” product late Existing functionality: Existing functionality should have nothing to fear from usability, if it is the “right” functionality Limited resources: Putting the “right” resource in at the “right” time can make the overall project more efficient Page 5

© Simeon Keates 2008 Rosson and Carroll “Usability Engineering”  “Cost-benefit analysis of usability activities contributes to more systematic usability engineering …”  “… BUT benefits are difficult to quantify, so estimates will often be overly conservative.” Issues:  Benefits (e.g. customer satisfaction) are harder to quantify and predict accurately  Costs are very concrete and easy to identify  Thus, can be difficult to justify usability… Page 6

© Simeon Keates 2008 Typical sources of usability “costs” (also Rosson and Carroll’s usability approach - 1)  Development of requirements scenarios  Validation/refinements of scenarios with users and customers  Development of basic-level task scenarios  Refinement of design scenarios with development team and customers  Development of information model  Review with team members  Development of paper prototypes  Walk-throughs with users of paper prototypes  … Page 7

© Simeon Keates 2008 Typical sources of usability “costs” (also Rosson and Carroll’s usability approach - 2)  …  Analysis of transcripts/report preparation  Development of interaction model  Review with team members  Development of running prototypes  Formative evaluation  Analysis of transcripts/report preparation  Detailed design and prototype-driven iteration of previous three steps Page 8 All of these steps have “usability” costs

© Simeon Keates 2008 Typical usability benefits  Fewer downstream design changes  Increased sales (and consequently reduced time to profitability)  Reduced need for user training  Enhanced customer productivity  Reduced resources spent on customer support  Increased loyalty in customer base (repeat and referral sales) Page 9 Many of these benefits identified in airport case study from 2 lectures ago…

© Simeon Keates 2008 A more scientific approach  Mantei and Teorey(*) examined the cost/benefit analysis in 1988  We will look at their calculations  But first we need to examine their methodology  NOTES: Costs are based on 1988 prices and techniques Not all costs are required for every project – choose which ones carefully Costs are indicative, not definitive Assumes 32,000 lines of code to be used by 250 employees Page 10 (*) Mantei and Teorey (1988) Cost/benefit analysis for incorporating human factors in the software lifecycle. Communications of the ACM 31(4):

© Simeon Keates 2008 Typical development stages in a prototyping lifecycle  Feasibility study  Requirements definition  Global design  Prototype construction  User evaluation  System implementation  Testing  Update and maintenance Page 11

© Simeon Keates 2008 Typical development stages in a Human Factors software lifecycle  Market analysis  Feasibility study  Requirements definition  Product acceptance analysis  Task analysis  Global design  Prototype construction  User testing and evaluation  System implementation  Product testing  User testing  Update and maintenance  Product survey Page 12

© Simeon Keates 2008 A breakdown of tangible costs Cost typeCost (1988 levels) Salaried employee (SE)Fixed ($40/hr) Hourly employee (HE)Fixed ($15/hr) External contractor (EC)Fixed ($60/hr) Consultant (CN)Variable Equipment and supplies (ES)Variable Software purchase (SP)Variable Page 13

© Simeon Keates 2008 Further assumptions  40 hours per week  4.3 weeks per month  =>172 hours per working month (WM) Examples:  $6880/WM for a salaried employee (at $40/hr) i.e. 172 * 40  $10,320/WM for an external contractor i.e. 172 * 60 Page 14

© Simeon Keates 2008 Costs added by Human Factors stages  1 – Cost of running focus groups  2 – Cost of building product mock-ups  3 – Expense of the initial design of a prototype  4 – Expense of making a prototyping design change  5 – Expense of purchasing the prototyping software  6 – Cost of running user studies  7 – Cost of creating (or renting) a user study environment (laboratory)  8 – Cost of conducting the user survey Page 15

© Simeon Keates 2008 Cost 1 – Focus groups cost breakdown  Time cost of the individuals involved + small equipment cost  Individuals involved: Participants Moderator/facilitator Video-taping/recording personnel Any other observers/data analysts Page 16 ~3 people ~10 people 1 person

© Simeon Keates 2008 Focus groups cost breakdown  Focus groups typically take 3 hours to run + 1 day to set-up and 1 day to dismantle  Minimum of 2 days to analyse the data  Moderator: Time for focus group + analysis  Support staff: Time for set-up + focus group + dismantling  Participants: Time for focus group [+ any preparation time]  A complete focus group analysis (3 consecutive groups) takes ~2 weeks  Used for Market Analysis and Product Acceptance Analysis Page 17

© Simeon Keates 2008 Focus groups cost breakdown Estimated costs for conducting focus groups Type of expenseCategoryAmount ACost of operating 3 internal focus groups 30 participants, 10 per focus group (3 hrs each) SE$3,600 Group moderator (25 hrs)CN$1,500 3 support staff (25 hrs each)HE$1,125 VideotapeES$60 Total$6,285 BCost of contracting3 external focus groups Fee charged by agency for complete study ( 3 focus groups and analysis for 2 weeks) CN$10,000 Page 18

© Simeon Keates 2008 Cost 2 – Estimating product mock-up costs  Building a product mock-up involves constructing a false UI scenario in software and video-taping the scenario  Script has to be written  Someone has to execute the scenario  Videotape needs to be professional  Videotape mock-up is used in Product Acceptance Analysis To focus groups to initiate discussions To target users who are then asked to complete questionnaires about the use of the product and their response to it  Can also be used in marketing Page 19

© Simeon Keates 2008 Estimating product mock-up costs Costs incurred in building a product mock-up of the proposed software system Type of expenseCategoryAmount Preparation of mock-up scenario (40 hrs)SE$1,600 Videotaping sessions (20 hrs)SE$800 Splicing/integration of scenarios (20 hrs)SE$800 Equipment rental for splicing, etc.ES$500 VideotapeES$60 Total$3,760 Page 20

© Simeon Keates 2008 Estimating product mock-up costs  The techniques just described are somewhat out of date  More usually accomplished via early development cycle prototypes e.g. alpha and beta versions Flash movies, etc.  Q - What to do if no software written?  A – Wizard of Oz! Page 21

© Simeon Keates 2008 Wizard of Oz Page 22 Do Action A {Receive response to Action A} Do Action B, etc.

© Simeon Keates 2008 Wizard of Oz (The Wizard unmasked!) Page 23 Do Action A {Receive response to Action A} Do Action B, etc. Create response to Action A Create response to Action B, etc.

© Simeon Keates 2008 Wizard of Oz explained  User interacts with a UI mock-up only  There is no significant software back-end  Remote user (the Wizard) “pretends to be the system”  Wizard creates the response based on expected system performance  Still needs a script and plenty of preparation Imagine if the Wizard does not know a response! Page 24

© Simeon Keates 2008 Cost 3 – Estimating user survey costs  Used in the Product Acceptance stage to assess users’ responses to the product mock-up  Also used in Product Survey (after launch) to: Determine the difficulties users have with the working system Examine the tasks the system is being used for Gather suggestions for changes to system For a user population of 250 employees  Typically half (125) would receive a user survey  A typical survey is 4 pages in length…  … and takes 30 minutes to complete Page 25

© Simeon Keates 2008 Estimating user survey costs Cost breakdown for running a user survey for the software product being tested Type of expenseCategoryAmount Development of questionnaire (40 hrs)SE$1,600 Pilot testing of questionnaire (40 hrs)SE$1,600 Distributing and collecting survey (20 hrs)HE$300 Coding and entering data (20 hrs)HE$300 Analysing the results of the survey (40 hrs)SE$1,600 Cost of time “lost” in filling out survey (40 hrs)SE$1,600 Computer timeES$100 Supplies and duplicating costsES$100 Total$7,200 Page 26

© Simeon Keates 2008 Cost 4 – Estimating initial prototype building costs  Cost breakdown for building a prototype does NOT include the design time Only the building time  Typically 4 weeks  Most prototyping systems involve 2 stages: Stage 1 – specify the connections between the screen displays Stage 2 – design each individual screen layout  Alternatively (and more modern) Stage 1 – specify the states between user interactions Stage 2 – design the individual states and the alterations that take place because of user actions Page 27

© Simeon Keates 2008 Estimating initial prototype building costs  As the UI grows more complex, time required to build the prototype also increases  Cost of building a prototype is: C = S × ( a + b D) Where  C = Cost  S = Number of states  D = Average number of new details per state  a = Constant reflecting the cost of building a single state  b = Constant reflecting the cost of adding a single detail Page 28

© Simeon Keates 2008 Estimating initial prototype building costs Costs incurred in the initial design of a prototype of the proposed software system (the prior development of a global design is assumed) Type of expenseCategoryAmount Specifications of the screen transitions (80 hrs)SE$3,200 Design of the individual screen layouts (80 hrs)SE$3,200 Total$6,400 Page 29

© Simeon Keates 2008 Cost 5 – Estimating costs for design changes to original prototype  Once the prototype is built, user studies will uncover problems with it Difficulties learning and using the system  Design change suggestions will be made …  … incorporated into the design …  … and evaluated again, etc. …  … until the number and severity of problems are “acceptable”  The initial user studies will usually find the most and most major issues  Later changes should be minimal Especially if the prototype is close to the final product design … … which it should be if task analyses, user surveys, etc. have been used Also, design changes should simply be updates of parts of the prototype … … and not a complete re-design Page 30

© Simeon Keates 2008 Estimating costs for design changes to original prototype  The initial user studies will usually find the most and most major issues  Later changes should be minimal  Especially if the prototype is close to the final product design …  … which it should be if task analyses, user surveys, etc. have been used  Also, design changes should simply be updates of parts of the prototype …  … and not a complete re-design Page 31

© Simeon Keates 2008 Estimating costs for design changes to original prototype  Since changes should be restricted to parts of the UI, should only take ~1 day per change  The number of changes expected and amount of time per change depends upon complexity of the interface  Similar equation to cost of building a prototype Page 32

© Simeon Keates 2008 Estimating costs for design changes to original prototype Costs incurred in incorporating a design change to the original prototype of the proposed software system. These costs assume that the design change does not require a complete revision of the screen transitions. Type of expenseCategoryAmount Modification of the screen transitions (4 hrs)SE$160 Re-design of the individual screen layouts (4 hrs)SE$160 Total$320 Page 33

© Simeon Keates 2008 Cost 6 – Estimating the cost of purchasing the prototyping software purchase  Actual purchase cost  Also time spent deciding which one …  … and then learning it  1988 prices - $10,000 for the software ($2,500 to $15,000+)  2008 prices - $600 per user (AXURE) Page 34

© Simeon Keates 2008 Estimating the cost of purchasing the prototyping software purchase Page 35 Costs incurred in incorporating a design change to the original prototype of the proposed software system. These costs assume that the design change does not require a complete revision of the screen transitions. Type of expenseCategoryAmount Time spent reviewing potential packages (1 month)SE$6,080 Purchase cost of packageSP$10,000 Total$16,080 + Time spent learning package

© Simeon Keates 2008 Cost 7 – Estimating costs of performing user studies  Preparation time can be substantial Example: preparing a manual for a completely new UI Manual need not be complete, but it needs to be sufficient and pilot-tested Example: preparing the study protocol Protocol needs to be complete and pilot tested  Costs of individual user studies are often independent of complexity of UI …  … however, the number of user studies increases with complexity  3 types of user study in the Human Factors software lifecycle… Page 36

© Simeon Keates 2008 Estimating costs of performing user studies In Task Analysis:  Users are asked to perform the types of tasks the system should support  Aim is to build a model of how users approach the task In User Testing and Evaluation and in final User Testing:  More conventional user studies  Conducted on prototypes or final system  Final system evaluation is always needed May be significant changes from the prototype behaviour Page 37

© Simeon Keates 2008 Estimating costs of performing user studies Costs incurred in conducting a single user study on 5 participants Type of expenseCategoryAmount Development of user directions [i.e. protocol] (40 hrs)SE$1,600 Pilot testing of directions (20 hrs)SE$800 Re-designing user directions (20 hrs)SE$800 Running experiment (40 hrs)SE$1,600 Analysing results of lab study (40 hrs)SE$1,600 VideotapeES$120 Cost of users in experiment (20 hrs)SE$800 Total$7,320 Page 38

© Simeon Keates 2008 Cost 8 – Estimating costs of construction of a user laboratory  A user lab can be a borrowed / rented or permanent office  Permanent office is recommended where significant usability activity is expected  User lab should be a mock-up of the environment of use for the product  The lab should allow space for an adjoining observation/recording room  Construction of a lab takes ~5 weeks Page 39

© Simeon Keates 2008 Estimating costs of construction of a user laboratory Cost of establishing a permanent human factors lab. These are mid-level costs. Much more sophisticated labs can be built. Type of expenseCategoryAmount Time spent laying out lab design and selecting lab equipment (160 hrs) SE$6,400 Cost of carpenter and electrician (20 hrs)EC$1,200 Cost of cameras, VCRs, one-way mirrorES$10,800 Total$17,600 Page 40

© Simeon Keates 2008 Estimating costs – a summary Lifecycle stageCost itemTotal cost Market analysisFocus group (set of 3)$6.285 Product acceptance analysisFocus group (set of 3)$6.285 Product mock-up$3,760 User survey$7,200 Task analysisUser study$7,320 Lab construction$17,600 Prototype constructionInitial design$6,400 Design change $320)$6,400 UIMS system$16,080 User testing and evaluationUser study $7,320)$29,280 User survey$7,200 User testingUser study$7,320 Product surveyUser survey$7,200 Total$128,330 Page 41

© Simeon Keates 2008 Estimating times – a summary Lifecycle stageCost itemTotal WMDev time Market analysisFocus group (set of 3)1.116 wks Product acceptance analysis Focus group (set of 3)1.116 wks Product mock-up0.472 wks User survey1.164 wks Task analysisUser study wks Lab construction1.055 wks Prototype constructionInitial design0.934 wks Design change WM)1.004 wks UIMS system wks User testing and evaluation User study 1.05 WM) wks User survey1.164 wks User testingUser study wks Product surveyUser survey1.164 wks Total16.45 WM69.1 wks Page 42

© Simeon Keates 2008 Common tangible benefits of Human Factors design  Direct benefits can be calculated by making several valid assumptions about the improvements to the UI, specifically:  (1) A reduction in user learning times  (2) A reduction in user errors  (3) A reduction in the cost of maintaining the system  Remember: for 32,000 line program for 250 employees… Page 43

© Simeon Keates 2008 Benefit 1 – Potential training cost savings  It is estimated that the learning time for new system will be 25% lower  Learning time is typically 2 weeks of classes 10 working days / 80 working hours  Staff turnover is 50 hourly employees and 50 salaried employees per year (Savings / year) = (Turnover) × (Training time save) × (Wage) = 50 × 20 × $15 = $15,000 per year for HEs = 50 × 20 × $40 = $40,000 per year for SEs Page 44

© Simeon Keates 2008 Benefit 2 – Potential error reduction cost savings  User “traps” exist in all Uis i.e. a standard sequence of user responses that always result in an error  Errors may be trivial and easily corrected …  … but are often annoying and time-consuming  These traps are almost always the result of the UI violating the learned behaviour of the user Example: driving an automatic car Page 45

© Simeon Keates 2008 Potential error reduction cost savings  How many errors per year? Assume:  User has chance of falling into a “trap”  Company has 250 employees using software 3 hrs/day and 20 scenarios/hr Errors/year = (# of employees) × p(Error) × (scenarios/hr) × (hrs/year) = 250 employees × × 20 × (21.5 dy/mo × 3 hrs/dy × 12 mo/yr) = 96,750 errors per year Page 46

© Simeon Keates 2008 Potential error reduction cost savings  How much cost per year? Assume:  96,750 errors per year  2 minutes recovery time per error Cost/year = (errors/yr) × (hrs/error corr) × (wage/hr) = 96,750 × (2 min/err) × (1hr ÷ 60mins) × ($15 / hr) = $48,375 per year for hourly employees (HE) = 96,750 × (2 ÷ 60 ) × ($40 / hr) = $129,000 per year for salaried employees (SE) Page 47

© Simeon Keates 2008 Benefit 3 – Potential design change cost savings  It is hard to estimate maintenance savings  However, it is claimed that early changes cost ¼ of later design changes Assume:  1 day (i.e. 8 hours) per change  25 changes Early cost = (hr / change) × (# of changes) × (wage / hr) = 8 × 25 × $40 = $8,000 Page 48

© Simeon Keates 2008 Potential design change cost savings Late cost = (early cost) × 4 = $8000 × 4 = $32,000 Design change savings = (late cost) – (early cost) = $32,000 – $8,000 = $24,000 Page 49

© Simeon Keates 2008 Benefit 4 – System adoption  If the development of the system is planned to meet the needs and wants of the user …  … then the probability of acceptance and use is higher  The cost benefit here is the entire cost of the systems development …  … because it is not “lost” Page 50

© Simeon Keates 2008 Tangible benefits of Human Factors design – Summary Sample estimates of 1 st year savings from the introduction of HF elements in the software design process Type of savings incurredAmount (HE)Amount (SE) Training costs (HE or SE)$15,000$40,000 Error reduction costs (HE or SE)$48,375$129,000 Avoidance of late design changes (SE)$24,000 Total$87,375$193,000 Page 51 Total savings ~$280,375 Total costs ~$128,330

© Simeon Keates 2008 Intangible costs of Human Factors software lifecycle  1 - The selection of non-critical design decisions for user studies  2 - The establishment of too high a level of usability  3 - Falling into the trap of overdesign  4 - Communication problems between Human Factors specialists and software designers Page 52

© Simeon Keates 2008 Selection of non-critical design decisions for user studies  Not all design decisions affect the overall “quality” and “acceptability” of the final UI  How do you determine which ones are worth investigating? Examples: Arial or Helvetica? 11 pt or 12 pt? Icon names?  Most usability experts rely on experience and intuition to decide…  … but they can be wrong! Costs:  Wasted user study time  Discovery of important changes may be put off until later Solution:  Make your user studies flexible enough to allow discovery of new problems Page 53

© Simeon Keates 2008 Establishment of too high a level of usability  Some UI performance targets simply cannot be met  Example: “This system must be learnable within 1 week”  This may appear to be a reasonable target …  … but what if the task itself takes >1 week to learn? Costs:  Design and development time chasing unrealistic performance targets Solution:  Set realistic targets  This can be informed by your task analysis studies  Also, aim for incremental improvements in performance over design cycles Page 54

© Simeon Keates 2008 Falling into the trap of overdesign  Prototyping software (UIMS) can make it very easy to add more features (“bells and whistles”) Examples: borders, colours, icons, images, etc.  How to know when to stop???  How to know what are valuable additions? Costs:  Designer’s time + implementation time Solution:  Strong management control  Occam’s razor Page 55

© Simeon Keates 2008 Communication problems between Human Factors specialists and software designers  There is a knowledge gap between HF specialists and the coders/developers…  Unless there is a shared common language and understanding, it is difficult to communicate effectively Costs:  Time lost in establishing a common language/understanding  Time lost developing the wrong thing (or the thing wrongly)  Time lost designing solutions that simply cannot be implemented Solutions:  HF specialists learns to develop own UI  Better corporate culture of communication Page 56

© Simeon Keates 2008 Intangible benefits from Human Factors software lifecycle  1 - Adoption of features that save time  2 - Avoiding system sabotage problems  3 - Enhancing the ability to solve conceptual problems using the software system Page 57

© Simeon Keates 2008 Adoption of features that save time  Feature adoption by users is reduced when the system is so complex that users give up trying to learn the advanced features i.e. they satisfice  Users will typically adopt the least complex system that offers them the required functionality Even if it is less efficient  The Product Acceptance Analysis should identify unnecessary features  The User Tests should reduce feature complexity Page 58

© Simeon Keates 2008 Avoiding system sabotage problems  Being asked to use a system that is inappropriate, difficult or inadequate can lead to employee frustration  This frustration can lead to employees “taking it out on the system” Example: entering incorrect or incomplete data Example: reporting false system failures  This is referred to as “system sabotage”  Focus groups in Market Analysis and Product Acceptance stages are designed to address this issue They test the receptivity of the target users to the new system  Also, the Task Analysis ensures that the final product matches the needs of the users as closely as possible Page 59

© Simeon Keates 2008 Enhancing the ability to solve conceptual problems using the software system  Increasing the user’s cognitive load to use the system …  … decreases the user’s cognitive capacity for “solving the problem”  So although the problem may be “solved” …  … there may have been a “better” solution  User Testing is designed to remove complexities from the system …  … and thus support users’ creative problem solving Page 60

© Simeon Keates 2008 Recommendations for Human Factors inclusion The HF lifecycle stages and the type of cost reduction that they are most likely to effect Cost reduction itemRelated lifecycle stage Increased system adoptionMarket analysis Product acceptance analysis User testing and evaluation Reduced training costsTask analysis User testing Reduced user errorsTask analysis User testing Transfer of design changes to an earlier stage in the lifecycle Prototype construction User testing (on prototype) Product survey (next re-design) Page 61

© Simeon Keates 2008 Cost of running User Tests vs. size of user population Page 62 Size of user population $ Break-even point Benefits of testing Cost of testing User testing not cost-effective User testing cost-effective

© Simeon Keates 2008 Summary of cost-benefit analysis  We have examined the factors that can increase costs …  … and also the tangible benefits that can be seen  For any large user group, the benefits will usually outstrip the costs …  …often by a significant margin  For smaller user groups, the case is less clear-cut …  … so either decrease costs … e.g. through “discount” usability methods  … or examine the intangible benefits more closely Page 63

© Simeon Keates 2008 Designing for mobile devices Page 64

© Simeon Keates products killed by the mobile phone (according to Wired.com)  1 – The PDA  2 – The camera  3 – The Ultra Mobile PC  4 – The telephone  5 – The MP3 player Page 65

© Simeon Keates products killed by the mobile phone (according to Wired.com readers)  1 – The pager  2 – The wristwatch  3 – The pocket calculator  4 – The alarm clock  5 – SatNav  6 – Books  7 – Handheld games consoles Page 66

© Simeon Keates 2008 Designing for Mobile devices  The “missing” lecture!  Excellent sets of slides at from MobileHCI 2008   Suggested reading: Scott Mackenzie’s presentation “Text input for mobile devices”… Page 67

© Simeon Keates 2008 Other slide sets from MobileHCI 2008  Mobile GUIs and Mobile Visualization Patrick Baudisch  Understanding Mobile User Experience Mirjana Spasojevic  Context ‐ Aware Communication and Interaction Albrecht Schmidt  Haptics, audio output and sensor input in mobile HCI Stephen Brewster  Camera ‐ based interaction and interaction with public displays Michael Rohs Page 68

© Simeon Keates 2008 Exercise Page 69

© Simeon Keates 2008 Exercise  Conclude your user trials  If you want feedback on your group presentation – then prepare slides for Friday NOTE – this is optional, but recommended Page 70