1. Incorporating Pragmatic Usability Testing Into a Software Test Plan Carla Merrill, Ph.D. Focused Design focuseddesign.com merrill@focuseddesign.com

2. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 2 Presentation Topics 1. How important is usability for a software product? 2. Why is it hard to incorporate usability testing into test plans? 3. A solution through design 4. Integrating the solution into an agile process

3. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 3 The Importance of Usability An interactive thought experiment for MS Word users FIRST:Think of something you liked about the Word. Why do you like it? (because it does what you need) What if I told you that the code implementing that feature was done in an old programming language and was inelegant, even sloppy. Knowing that, think about that feature again. Do you picture the code or the UI? (UI) Do you care about the structure of the code? (no) SECOND: Think about something you don’t like about the product. Why don’t you like it? (because it doesn’t do what you need) What if I told you that the code for that part of the product was extremely elegantly written in a programming language du jour. Would that fact make you like the feature? (no)

4. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 4 The Importance of Usability Paradigm Shift: 1. Users define and evaluate a software product in terms of what they experience: the user interface. 2. Long-term success of a software product depends on how well the user interface meets the needs of target users. 3. Good-quality code doesn’t matter to users if the features it implements don’t meet their needs. Users don’t experience the code.

5. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 5 Perceived Difficulties of Incorporating Usability Testing into a Software Test Plan 1. Specialized Human Factors skills required. 2. Specialized environments (test labs) required. 3. Appropriate test subjects required. 4. What is to be tested? 5. More project time and money required after standard test phase. 6. Test results and recommended changes are difficult to track and difficult to implement before product release.

6. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 6 Solution: Take Advantage of the Software Design Phase Require the user interface design process to deliver user scenarios that can be added to test plans.

7. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 7 Required Format for User Scenarios 1. Pragmatic: only scenarios that capture the primary tasks of your target users (contrast with use cases). 2. Created from the user’s perspective with user- based vocabulary that is implementation free. 3. Step-by-step format, limited to 10 steps per scenario. 4. Long tasks are chunked into a series of scenarios that are prioritized by importance to target users. 5. Scenarios are validated with users before testing begins.

8. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 8 Adding Scenarios To Test Plans 1. Tester can define a test procedure for each scenario by adding implementation language. 2. Tester can evaluate success/failure of a scenario by standard test methods: (time, system failure points, UI failure points, number of implementation steps required to complete the scenario). 3. Tester can log bugs for failure points in the test procedure.

9. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 9 Advantages of Incorporating User Scenarios: Usability Can Be Tested! 1. Testers have the skills. 2. Usability tests are clearly defined. 3. Standard test procedures are used. 4. Testing cycle allows usability bugs to be fixed before release. 5. Usability bugs can be triaged appropriately based on priority ratings of scenarios. 6. Usability of the product can be measured using standard testing metrics.

10. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 10 The Goal-Oriented Design Solution (GOD-S) Note: Slides 11-15 will not be discussed in this presentation. They are included for reference only.

11. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 11 User Profiling: What do we need to know about users?

12. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 12 User Profiling: How do we gather the information we need? Gain access to users Meet users in the right environment Prepare for the meetings Practice Guided Listening™ Create user reports

13. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 13 Persona and Scenario Development: How do we make effective use of the information we’ve gathered? The major challenges of UI design “Many users — one user interface” Feature-creep and elastic users Who’s to say what users want? What is a persona?* What are scenarios? A persona with scenarios can help solve the problems of UI design * Based on the usage of the term coined by Alan Cooper

14. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 14 Step 1. Construct a Matrix from User Profiles Step 2. Analyze the Matrix Step 3. Build Goal-Oriented Scenarios for a Persona Persona and Scenario Development: How do we create a persona and scenarios?

15. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 15 User Interface Design: Design the User Interface Based on Goal-Oriented Scenarios A sound strategy for UI design: Top-down design – follow the user 1 step at a time – focus on enabling scenarios There is no “right answer” – verify the design with users It’s creative It requires the UI design skillset It’s iterative and needs scheduled time in the project

16. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 16 How does GOD work with different development models? How to merge GOD with a Development Model Serial Development Model Agile Development Model

17. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 17 Serial Development Model Completeness, Stability Time Requirements Testing Design Development

18. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 18 Agile Development Model ( after Alistair Cockburn, http://alistair.cockburn.us )http://alistair.cockburn.us Completeness, Stability Time Requirements Testing Design Development

19. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 19 Potential Problems of the Merger Agile Development Bottom–up coding – don’t need the UI framework Use cases focus on system needs Code chunks are pieces isolated from the whole application UI chunks designed in context of a UI framework UI chunks based on user taskflows UI design is always ahead of the code GOD Top-down design – a UI framework first Scenarios focus on primary needs of a person Code chunks are based on programming methodology Coding of UI is simpler, less critical, and done last – the “lipstick”

20. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 20 Potential Synergies of the Merger Agile Development Bottom–up coding allows rapid progress, while absorbing thin topcoat Use cases can focus on user needs, but care for system needs Code chunks allow for iteration, absorbing usability feedback Top-down design cares for the Big Picture Scenarios keep the focus on user needs The Big Picture design can provide a logical scheme for chunking GOD

21. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 21 Strategies and Tactics for a Successful Merger Build UI design prework into the process Create persona/scenarios Create UI framework/primary navigation (inch-deep, top-down coding) Get buy-in on UI framework from stakeholders Create chunks based on UI framework Continue inch-deep top-down design and coding in each iteration Create user-focused use cases for chunks Include usability testing in each iteration and absorb the feedback Enable teaming between UI designers and UI coders Require UI design realism: designers use most efficient GUI controls Bleed UI deficits into the next iteration until UI is complete Agile Development + GOD

22. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 22 Timeline for a Successful Agile + GOD Project Project start Product delivery High-level business requirements GOD prework Schedule/chunking Team definition System framework design Chunking refinement as needed Agile iterations 1…n with multiple teams Integration Usability testing Beta

23. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 23 Timeline for a Successful Agile + GOD Iteration V starts End of V’s final QA GOD usability testing Reflections mtg Business reqs Chunk definition GOD V preworkGOD wireframing DEV post V cleanup/integrationDEV coding GOD UI deficit list QA post V bug tracking Use case development QA new V test scripting = code drop QA testing Start the next V = code test

24. October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 24 Summary 1.Usability testing is important for the success of a software product. 2.Pragmatic usability testing can be incorporated into the standard test plan through user scenarios developed in the design phase. 3.Testing can be done using existing QA resources and bug tracking systems. 4.Goal-Oriented Design supports pragmatic usability testing in an agile software development environment.