1. Design in the World of Business
Chapter 7
Design in the World of Business

2. Chapter 7 Design in the marketplace Featuritis
Timeline for moving from design to product
Incremental vs. radical design
As well as opinions related to
New media
Augmented humans
Design and the environment
The future of the Design of Everyday Things

3. Why, what, where and when to evaluate
Iterative design & evaluation is a continuous process that examines:
Why: to check that users can use the product and that they like it
What: a conceptual model, early prototypes of a new system and later, more complete prototypes
Where: in natural and laboratory settings
When: throughout design; finished products can be evaluated to collect information to inform new products
Designers need to check that they understand users’ requirements.

4. Bruce Tognazzini tells you why you need to evaluate
“Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process you are going to throw buckets of money down the drain.”
See AskTog.com for topical discussions about design and evaluation.

5. The language of evaluation
Analytical evaluation
Controlled experiment
Field study
Formative evaluation
Heuristic evaluation
Predictive evaluation
Summative evaluation
Usability laboratory
User studies
Usability studies
Usability testing
User testing

6. DECIDE: a framework to guide evaluation
Determine the goals.
Explore the questions.
Choose the evaluation approach and methods.
Identify the practical issues.
Decide how to deal with the ethical issues.
Evaluate, analyze, interpret and present the data.

7. Determine the goals What are the high-level goals of the evaluation?
Who wants it and why?
The goals influence the approach used for the study.
Some examples of goals:
Identify the best metaphor on which to base the design.
Check to ensure that the final interface is consistent.
Investigate how technology affects working practices.
Improve the usability of an existing product.

8. Explore the questions
All evaluations need goals & questions to guide them.
E.g., the goal of finding out why many customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub-questions:
What are customers’ attitudes to these new tickets?
Are they concerned about security?
Is the interface for obtaining them poor?
What questions might you ask about the design of a cell phone?

9. Choose the evaluation approach & methods
The evaluation approach influences the methods used, and in turn, how data is collected, analyzed and presented.
E.g. field studies typically:
Involve observation and interviews.
Do not involve controlled tests in a laboratory.
Produce qualitative data.

10. Identify practical issues
For example, how to:
Select users
Stay on budget
Stay on schedule
Find evaluators
Select equipment

11. Decide about ethical issues
Develop an informed consent form
Participants have a right to: - Know the goals of the study; - Know what will happen to the findings; - Privacy of personal information; - Leave when they wish; - Be treated politely.

12. Institutional Review Board
Purpose
To protect the rights and welfare of subjects in evaluations
Process
IRB reviews evaluation process to ensure it is safe and taking appropriate precautions
How it relates to CHI evaluations
Most CHI-related evaluations are in the “exempt” category
Still, there are issues to consider

13. Evaluate, interpret & present data
The approach and methods used influence how data is evaluated, interpreted and presented.
The following need to be considered: - Reliability: can the study be replicated? - Validity: is it measuring what you expected? - Biases: is the process creating biases? - Scope: can the findings be generalized? - Ecological validity: is the environment influencing the findings? - i.e. Hawthorn effect.

14. Evaluation approaches
Usability testing
Field studies
Analytical evaluation
Combining approaches
Opportunistic evaluations

15. Usability testing
Involves recording performance of typical users doing typical tasks.
Controlled environmental settings.
Users are observed and timed.
Data is recorded on video & key presses are logged.
The data is used to calculate performance times, and to identify & explain errors.
User satisfaction is evaluated using questionnaires & interviews.
Field observations may be used to provide contextual understanding.

16. Experiments & usability testing
Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more things – i.e., variables.
Usability testing is applied experimentation.
Developers check that the system is usable by the intended user population for their tasks.
Experiments may also be done in usability testing.

17. Usability testing & research
Improve products
Few participants
Results inform design
Usually not completely replicable
Conditions controlled as much as possible
Procedure planned
Results reported to developers
Experiments for research
Discover knowledge
Many participants
Results validated statistically
Must be replicable
Strongly controlled conditions
Experimental design
Scientific reported to scientific community

18. Usability testing
Goals & questions focus on how well users perform tasks with the product.
Comparison of products or prototypes common.
Focus is on time to complete task & number & type of errors.
Data collected by video & interaction logging.
Testing is central.
User satisfaction questionnaires & interviews provide data about users’ opinions.

19. Usability lab with observers watching a user & assistant

20. Portable equipment for use in the field

22. Testing conditions Usability lab or other controlled space.
Emphasis on:
selecting representative users;
developing representative tasks.
5-10 users typically selected.
Tasks usually last no more than 30 minutes.
The test conditions should be the same for every participant.
Informed consent form explains procedures and deals with ethical issues.

23. Some type of data Time to complete a task.
Time to complete a task after a specified. time away from the product.
Number and type of errors per task.
Number of errors per unit of time.
Number of navigations to online help or manuals.
Number of users making a particular error.
Number of users completing task successfully.

24. Usability engineering orientation
Aim is improvement with each version
Current level of performance
Minimum acceptable level of performance
Target level of performance

25. How many participants is enough for user testing?
The number is a practical issue.
Depends on:
schedule for testing;
availability of participants;
cost of running tests.
Typically 5-10 participants.
Some experts argue that testing should continue until no new insights are gained.

26. Experiments Predict the relationship between two or more variables.
Independent variable (e.g. which design) is manipulated by the researcher.
Dependent variable (e.g. time) depends on the independent variable.
Typical experimental designs have one or two independent variable.

27. Experimental designs
Different participants - single group of participants is allocated randomly to the experimental conditions.
Same participants - all participants appear in both conditions.
Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc.

28. Different, same, matched participant design

29. The Need to Balance Why do we balance Latin square study design
For learning effects and fatigue
For interactions among conditions
For unexpected features of tasks
Latin square study design
Used to balance conditions and ordering effects
Consider balancing use of two systems for two tasks for a same-participant study

30. Review of Experimental Statistics
For normal distributions
T test: for identifying differences between two populations
Paired T test: for identifying differences between the same population or a well-matched pair of populations
ANOVA: for identifying differences between more than two populations
For predicted distributions
Chi square: to determine if difference from predicted is meaningful

31. Field Studies Review Field studies are done in natural settings.
The aim is to understand what users do naturally and how technology impacts them.
Field studies can be used in product design to: - identify opportunities for new technology; - determine design requirements; - decide how best to introduce new technology; - evaluate technology in use.

32. Data collection & analysis
Observation & interviews
Notes, pictures, recordings
Video
Logging
Analyses
Categorized
Categories can be provided by theory
Grounded theory
Activity theory

33. Analytical Evaluation Overview
Describe inspection methods.
Show how heuristic evaluation can be adapted to evaluate different products.
Explain how to do doing heuristic evaluation and walkthroughs.
Describe how to perform GOMS and Fitts’ Law, and when to use them.
Discuss the advantages and disadvantages of analytical evaluation.

34. Inspections Several kinds.
Experts use their knowledge of users & technology to review software usability.
Expert critiques (crits) can be formal or informal reports.
Heuristic evaluation is a review guided by a set of heuristics.
Walkthroughs involve stepping through a pre-planned scenario noting potential problems.

35. Heuristic evaluation Developed Jacob Nielsen in the early 1990s.
Based on heuristics distilled from an empirical analysis of 249 usability problems.
These heuristics have been revised for current technology.
Heuristics being developed for mobile devices, wearables, virtual worlds, etc.
Design guidelines form a basis for developing heuristics.

36. Nielsen’s heuristics Visibility of system status.
Match between system and real world.
User control and freedom.
Consistency and standards.
Error prevention.
Recognition rather than recall.
Flexibility and efficiency of use.
Aesthetic and minimalist design.
Help users recognize, diagnose, recover from errors.
Help and documentation.

37. Discount evaluation
Heuristic evaluation is referred to as discount evaluation when 5 evaluators are used.
Empirical evidence suggests that on average 5 evaluators identify 75-80% of usability problems.

38. No. of evaluators & problems

39. 3 stages for doing heuristic evaluation
Briefing session to tell experts what to do.
Evaluation period of 1-2 hours in which:
Each expert works separately;
Take one pass to get a feel for the product;
Take a second pass to focus on specific features.
Debriefing session in which experts work together to prioritize problems.

40. Advantages and problems
Few ethical & practical issues to consider because users not involved
Can be difficult & expensive to find experts
Best experts have knowledge of application domain & users
Biggest problems:
Important problems may get missed
Many trivial problems are often identified
Experts have biases

41. Cognitive walkthroughs
Focus on ease of learning
Designer presents an aspect of the design & usage scenarios.
Expert is told the assumptions about user population, context of use, task details.
One of more experts walk through the design prototype with the scenario.
Experts are guided by 3 questions.

42. The 3 questions
Will the correct action be sufficiently evident to the user?
Will the user notice that the correct action is available?
Will the user associate and interpret the response from the action correctly?
Note the connection to Norman’s gulf of execution and gulf of evaluation.
As the experts work through the scenario they note problems.

43. Pluralistic walkthrough
Variation on the cognitive walkthrough theme.
Performed by a carefully managed team.
The panel of experts begins by working separately.
Then there is managed discussion that leads to agreed decisions.
The approach lends itself well to participatory design.

44. Predictive Models
Provide a way of evaluating products or designs without directly involving users.
Less expensive than user testing.
Usefulness limited to systems with predictable tasks - e.g., telephone answering systems, mobiles, cell phones, etc.
Based on expert error-free behavior.

45. GOMS Goals - the state the user wants to achieve e.g., find a website.
Operators - the cognitive processes & physical actions needed to attain the goals, e.g., decide which search engine to use.
Methods - the procedures for accomplishing the goals, e.g., drag mouse over field, type in keywords, press the go button.
Selection rules - decide which method to select when there is more than one.
Use at NYNEX and NASA

46. Keystroke level model
GOMS has also been developed to provide a quantitative model - the keystroke level model.
The keystroke model allows predictions to be made about how long it takes an expert user to perform a task.

47. Response times for keystroke level operators (Card et al., 1983)

48. Fitts’ Law (Fitts, 1954)
Fitts’ Law predicts that the time to point at an object using a device is a function of the distance from the target object & the object’s size.
The further away & the smaller the object, the longer the time to locate it and point to it.
Fitts’ Law is useful for evaluating systems for which the time to locate an object is important, e.g., a cell phone, a handheld devices.

49. Characteristics of approaches
Usability testing
Field studies
Analytical
Users
do task
natural
not involved
Location
controlled
anywhere
When
prototype
early
Data
quantitative
qualitative
problems
Feed back
measures & errors
descriptions
Type
applied
naturalistic
expert

50. Evaluation approaches and methods
Usability testing
Field studies
Analytical
Observing x
Asking users
Asking experts
Testing
Modeling

51. Chapter 7 Design in the marketplace Featuritis
Timeline for moving from design to product
Incremental vs. radical design
As well as opinions related to
New media
Augmented humans
Design and the environment
The future of the Design of Everyday Things