Establishing an Agile Testing Team: Our Four Favorite “Mistakes” Kay Johansen Anthony Perkins

Slide 2 Our Task: Introduce a QA group to support frequent delivery at a web-hosting provider. An established company with large installed customer base and no existing QA group, delivering weekly updates! The “products” were really collections of open- source software customized by our developers; multiple operating systems (also customized); Some applications developed in-house One two-year old project not gone out the door.

Slide 3 We intended to automate heavily and focus on the four Agile values. Individuals and interactions We hired programmers, known team players, gave them no process, but encouraged collaboration. Working software We wanted to write automated regression tests. We made a clean test environment wth daily integrations, didn’t ask for documents. Customer collaboration We held bug prioritization meetings, talked with developers many times a day. Responding to change We reinterpreted our priorities over time.

Slide 4 We deliberately made 3 “mistakes” (matching “Lessons Learned in Software Testing”) We didn’t protect the customer “Never be the gatekeeper” “Don’t sign-off to approve the release of a product” We didn’t hire QA experience “Understand how programmers think” “Develop programmers’ trust” We gave in easily “Beware of becoming a process improvement group” “Adapt your processes to the practices that are actually in use”

Slide 5 We made friends with the product manager and developers, and integrated daily... Los of “face time” made friends out of the Product Managers (they controlled the release). Our “programmer/testers” were respected by the product developers. We found bugs in real code, so development paid attention to our findings. We always knew the state of the code, so the PM could make scope vs. schedule decisions. We refused to assign priorities to bugs, so development and the PM had to. The 2-year old product got shipped.

Slide 6 However, we didn’t actually automate very much, and half our people got laid off. We hadn’t actually automated many tests, still tested by hand. Our test coverage was lower than we wanted (much lower!) The product released later than we -- and management -- wanted. The company downsized, laid off half our team. (Hint: we were not perceived as valuable)

Slide 7 Our team decided to reprioritize, in order to demonstrate our value clearly. We had been too focused on the technical: (test automation and complete coverage) Needed more focus on the social / political: We worked on gaining still more trust from the developers We began status reports to management to show them what we were doing.

Slide 8 Even pushing trade-off decisions up made us look more valuable. We made deliberate “Mistake” Number 4: We chose not to test everything We gave management several alternatives of what to NOT test They chose to drop testing of weekly releases (From “Lessons Learned in Software Testing”:) “Beware of testing ‘completely’ ” “Don’t let yourself be abused”

Slide 9 We succeeded in being perceived as adding value. When bugs escaped on the untested products, the reaction was "How many more resources do you need?" instead of "How could you have let that happen?" Our programmers had a genuine interest in the developers’ products, so they went to lengths to help us in any way they could. We were accommodating where we could be, so people gave us more support when we had an issue we wouldn't give in on. Finally: We got to hire more people !

Slide 10 Reflection: Hiring programmers was great and prioritizing test coverage was useful. We liked these things that we did: Hiring programmers onto the test team Making friends with the product manager Prioritized, non-exhaustive testing Clean test environment Read “Lessons Learned in Software Testing” But what happened to the automated testing? Wasn’t as critical at that point... We did get around to it over time.