1. A Critical Look at “Best Practices” Jon Bach QE/PM Director jobach@ebay.com

2. “Best Practices and Context-Driven: Building a Bridge” From 2003, someone wrote…

3. Definition of “best” “Of the most excellent, effective, or desirable type or quality.” Mine: “Optimal available value”

4. Best Practices that aren’t For a keynote, it’s best to have audience seating? via @mariakedemo

5. Best Practice: Don’t be an asshole? But wait… “Someone has to be an asshole. Polite people aren't going to change the industry -- they made it the way it is.” James Bach, 8/3/13

6. How can you argue with these? Install the software before testing it – …but not best when testing install… – …but not best if you need a “dirty” config Use a keyboard to type characters into a field – …but not best if a stylus is needed… – …but not best if you’re testing buffered inputs When you find a bug, write a report – … but not best if the programmer is your brother who is sitting right next to you

7. Construx “Best practices are proven software engineering techniques, methods, lifecycles, etc. CxOne is based on software industry best practices provide insight and guidance into optimal selection, efficient deployment, and effective use of industry best practices.”

8. ConstruxConstrux (from their website) “Best practices are proven software engineering techniques, methods, lifecycles, etc. CxOne is based on software industry best practices provide insight and guidance into optimal selection, efficient deployment, and effective use of industry best practices.”

9. …such as…? “It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.” “Gathering and agreeing on requirements is fundamental to a successful project.” “Choosing the appropriate architecture for your application is key.” “A best practice for constructing code includes the daily build and smoke test.”daily build and smoke test “It is important to review other people's work.” “It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded.” © Construx (from their website)

10. …best, but why…? “It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.” “Gathering and agreeing on requirements is fundamental to a successful project.” “Choosing the appropriate architecture for your application is key.” “A best practice for constructing code includes the daily build and smoke test.”daily build and smoke test “It is important to review other people's work.” “It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded.” © Construx (from their website)

11. …best how, and according to whom…? It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process. Gathering and agreeing on requirements is fundamental to a successful project. Choosing the appropriate architecture for your application is key. A best practice for constructing code includes the daily build and smoke test. It is important to review other people's work. It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded.

12. IBMIBM – this link is broken, nice practice there http://www.standishgroup.com/sample_research/chaos_1994_1.php

14. http://www.construx.com/Practicing_Earl/Context_Matters/ In fact, context matters with any "best practice". The word "best" doesn't make it immune to the reality of the context. While it may be best somewhere, it could be a "worst practice" for your project. How do you know the difference? You can't until you fully comprehend the context and that usually means trying the practice out. A Construx employee writes critically about the term “Best Practices”, which goes against its own website (see previous slide).

15. Website bug … (10/15/13)

16. (link still broken as of 5/23/14)

17. Best Practice: Have Trained and Certified Test Teams “Certification can establish the minimum and expected skills needed for test positions.” “Introduction of ISTQB certification program is raising the standard for tester skills uniformly around the world.” http://www.rbcs-us.com/images/documents/testing%20best%20practices.pdf From Rex Black (RBCS Consulting) 2006 My opinion is that you do not need to be certified before you can be considered “skilled”. Certification is not a “practice”, nor does it build skill.

18. From STP: an article about not using “best practices” under the section titled “Best Practices” http://www.softwaretestpro.com/tag/best-practices

19. Context-driven testing http://context-driven-testing.com/ The value of any practice depends on its context. There are good practices in context, but there are no best practices. People, working together, are the most important part of any project’s context. Projects unfold over time in ways that are often not predictable. The product is a solution. If the problem isn’t solved, the product doesn’t work. Good software testing is a challenging intellectual process. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

20. “It depends” pendeo pendere pependi [to hang; to hang upon, depend on; to hang loose, hover; to be suspended, discontinued; be in suspense, uncertain, undecided]. http://catholic.archives.nd.edu

21. Me and Rob Sabourin I said to Rob: “How about a keynote that’s context-driven? Maybe the content can come from my workshop about exploring best practices the day before I present the keynote?” “I love this idea.” “This will be really cool.” “This has never been done before (that I know of).” So… What practices will you explore? What are the pros and cons of each? What are some interesting contextual considerations? “As a testing consultant, I don’t need the low-hanging fruit, I want the fruit that’s on top of the tree, that’s really hard to get!” “It would be cool to see the favorite 10 from the group.” “It would be cool to have an artifact -- a mind map, maybe?”

22. When hosting a workshop, it’s important that you face the audience and talk to them.

23. Mission

24. Help me build the closing keynote: “A Critical Look at Best Practices”A Critical Look at Best Practices

25. Tactics Part I Imagine that you are a council of elders. What so-called “best” software testing practices come to mind? Vote on a top 10 list. Part II Pros and cons of each Part III What should testers consider when choosing and evaluating practices?

26. Story Who was here? What practices did we explore? How did we explore them? Is there a “top 10”? What are the nuances testers should consider before using a practice? What are some real world examples and experiences? What conclusions did we make? What artifact do we have to share? Advice to our colleagues on the other sessions.

27. Who 16 people (8 male, 8 female) 8 countries Testers, Test Managers, Consultants 2 years’ experience, 38 years’ experience Games, Insurance, Regulated, Government, Banking, Anti-Virus, Big Data Dell, Nortel, Spotify, World of Tanks, Unity Neil Thompson…

28. From Neil’s paper… “Best Practices and Context-Driven: Building a Bridge” - 2003 Would not call them “always-good practices” now, but “considerations for any context”

29. Brainstorm of “good” practices

30. Full list (64) Learn how to tell a good testing story Use a test management system Carefully choose testing tools Vary your test techniques Bug advocacy: learn how to sell bugs Work smarter, not harder Make it clear to PM how much it costs to test or deliver proper software Have Dev test it first before it goes to Test

31. Full list (64) Don’t spend time doing things a machine can do better Make sure test coverage is approved by a stakeholder Plan the test environment and data needs early Document enough detail to save effort Work as a team an communicate Design code to be maintainable Use exploratory testing

32. Full list (64) Test early Be able to explain the value you add Work with Developers to know who tested what Work with Dev to build in testability and logging Weak or unclear requirements will cause problems Prioritize Always add time to estimates

33. Full list (64) Be aware of your assumptions Certify the testers Do research -- self education is important Become part of the community Create clear exit / entry / stop criteria Give yourself more time You can always find bugs in your software Be aware (observation vs. reference) Treat bugs as something positive

34. Full list (64) Be aware that metrics can be dangerous Be aware of pitfalls when communicating metrics Talk the same language (“test”, “integration”) Don’t let Dev verify bugs Use Session-Based Test Management Test software Check fixed bugs against new versions Ask questions

35. Full list (64) Send testers to Per Scholas Never trust developers Talk to each other Don’t plan too much; execute as well Get feedback often Write clear descriptions of bugs Use daily standups Automate Use prototypes

36. Full list (64) Communicate with end users Consult a variety of stakeholders Get good at using your product Be aware of your emotional responses Don’t be afraid of failing Gain and apply experience Look out for the unexpected Assume specs are incomplete Expect that you will never have time enough

37. Full list (64) Focus and defocus Think critically Use checklists Perform smoke tests first Use a bug-tracking system Discuss issues with Developers first before reporting Try to be effective and efficient

38. Vote on your 3 favorites PICTURE

39. Top 10 Ask questions Talk to each other Think critically Test early Learn how to tell a good testing story Don't waste time doing things a machine can do better Work as a team Be able to explain the value you add Don't be afraid of failing Prioritize

40. Now let’s see how these so-called “good” practices can go wrong…

41. “Ask questions” -- pitfalls Asking too many questions could be annoying Could take too much time from stakeholders Could offend political sensibilities The answer might provoke hidden agendas Not understanding importance of who you ask the question of Forgetting that you asked the same question of the same person The person being asked does not want to reveal they do not know the answer Did you ask enough questions? Someone could give the wrong answer, but you don't know

42. “Talk to each other” pitfalls Meetings for the sake of meetings Conflicts can come up from talking to each other Risk of too much information / confirmation bias Risk of interrupting people who do not want ot be interrupted Cultural differences could provoke conflict Speak too much -- garrulous Annoying volume to other colleagues Not easy with distributed teams, tech and time zones, language Unintentionally spread confidential info Can forget or have a lack of formal agreement what we talked about which means wasted time Book: “Quiet -- The Power of Introverts in a World That Can’t Stop Talking

43. “Think critically” pitfalls Don't think -- court case testimony Acceptance testing Testers can go too deep for minor issues Don't take your critical thinking home Dangerous for your health -- deep psychology problems, nothing is good enough Can be seen as over-perfectionism Can be seen as criticism Can be seen as culturally insubordinate

44. “Test early” pitfalls Testers report invalid bugs because product isn't ready If you don't have full specifications yet Project isn't yet approved by official stakeholders There are dependencies for certain types of testing (integration) -- preconditions Code base is too unstable, and code base rewritten in two weeks time anyway no matter what you find

45. Learn to tell a good testing story -- pitfalls Since learning takes practice, you could lose credibility as you learn The story isn’t necessary

46. Don't waste time doing things a machine can do better -- pitfalls It make take you some time to figure out that it couldn't be automated Takes too much time to automate cases when you can just check it by hand -- machine can do it better, but it's not worth Maybe its faster to do it manually for now BVT manual at first – easy, quick, painless to run UNTIL the build gets more mature and iterates every hour

47. “Work as a team” pitfalls Difficult to work as a team when the team dynamics might change Meetings might be forced on people who already know how to do it individually Einstein worked alone Sometimes you need to remove yourself from the team in order to focus

48. “Work as a team” pitfalls 2 Autism / Aspergberger's will get very stressed Dependency on each person of the team, varies on confidence, role, responsibility Maybe that you don't get along socially with the others on the team; less effective when negative energy If more than one "leader" on the team cause conflict Introverts

49. “Be able to explain the value you add” pitfalls Don't have to do that if it's known already and you know There are intrinsic things that may not need to be explained You may never know the value that you really add anyway You don't have to justify the job because the existance of the job itself has value

50. “Don’t be afraid of failing” pitfalls Some healthy fear is reasonable Don't be cocky Fear drives people to be better ("Expert Game") and do research Testing books on the shelf

51. “Prioritize” pitfalls Wasted energy because everything is equal and needs to be tested anyway Pressure to prioritize may cause you to incorrectly prioritize If the test approach is galumphing, prioritization is wasted effort If you're in modeling mode, there's nothing you need to do to prioritize There's an opportunity cost to any prioritize

52. Considerations for practices Social interactions Knowing what story you need to tell (depending on the domain of the project) Time How do you be not afraid of failure when your software is Mission critical? Size of project (2 or 200) Budget Locale

53. Considerations Lifecycle of project (Agile, waterfall, v-model) Your own technical skill Domain Legal (regulatory) Cultural Gender

54. In closing Best Practice lists hide ignorance and incompetence; but make money because consultants get paid for quick wins, BUT…. sometimes we just want to be told what to do Brainstorming practices that provide value is easy; scrutinizing those practices can be difficult, exhausting, and even disheartening You have a tester mindset; scrutinize any so- called “best practice” you want to get rid of.

55. I’m done. Let’s rumble…