Using Managed Test Teams An Alternative to Out-sourcing or In-sourcing testing Ståle Amland, Hulda Garborgsv. 2, N-4020 STAVANGER, NORWAY Mob: , Phone: , Fax: ICSTEST 2001

©Amland Consulting2 Presentation Outline Managed test teams (MTT): What? Why? Experiences Alternative risk and reward profiles Alternative test organisations Summary

©Amland Consulting3 The Managed Team Concept Externally resourced – internally managed Best practice from various sectors Shared risk via mutual agreed objectives The team must understand their responsibility Optimise use of internal resources, IT & Business Procedural framework Guidelines Standards Templates Coverage criteria But: the team should use their expertise

©Amland Consulting4 The Cases Case 1 – Railway Industry Y2K programme 100+ Business Critical Applications USD30 million budget, 140 staff 4 x Managed Test Teams – 40 staff Case 2 – Telecommunication Industry Software house, product release USD2,2 million budget (testing) staff, 6 months 1 manager 1 operation 10 – 13 planning / preparation

©Amland Consulting5 Expectations Professional Approach Testing expertise and ability Professional team management Flexibility Team Rivalry Potential conflicts Delivery to timescale Commercial Inertia Concentrating on T&Cs instead of deliveries

©Amland Consulting6 Experiences in a Multi-Vendor Programme Background High profile, multi/thread, multi environment Time driven / no flexibility Internal resource heavily committed Testing a critical activity External expertise available External remediation / development

©Amland Consulting7 Experiences continued Issues Integration with internal resources Management (corporate and project) Test Environment / operation Users (availability) Integration with external resources Programme management / test authority Learning curve Standards, methods and procedures Applications Reporting

©Amland Consulting8 Experiences continued Issues (continued) Accountability Test Coverage Quality standards Team structure Permanent vs. contractor Team Expertise (leaders and doers) Combination of business knowledge and testing experience Number of teams (balance load, avoid management overhead)

©Amland Consulting9 Experience: Expectations vs. reality Professional Approach-Achieved Flexibility-Achieved Team Rivalry-Not Met Delivery to timescale-Achieved Commercial Inertia-Not met

©Amland Consulting10 Expectations vs. reality cont. Lessons learnt Ensure correct team structure Permanent / Contractors Business Knowledge / testing experience Retain experienced teams Benefit from external experience Challenged existing practise Ensure critical mass is retained Keep the core team together by providing new challenges continuously Avoid dilution of core team

©Amland Consulting11 Presentation Outline Managed test teams (MTT) Alternative risk and reward profiles Fixed Price Target Budget Time and Material Alternative test organisations Summary

©Amland Consulting12 Risk Profile – Fixed Price Agreed fixed price prior to test execution Used for System Test, Acceptance Test or equivalent “stand alone” test phases Price agreed as part of specification, design and/or coding Test organisation takes all risk Obvious problems Lack of information about quality Challenge to agree acceptance criteria Cooperation between client and test organisation Pros. & Cons. Easy budgeting Test organisation might take short cuts… Price will have to include high risk contingency

©Amland Consulting13 Risk Profile – Target Budget Incentives Time & Material Agree target budget If overrun: reduced payment to test organisation If under run: split the savings (”profit”) between client and test organisation Test organisation takes risk Obvious Problems Challenge to agree acceptance criteria Pros. & Cons. Incitements for both organisation to increase productivity Test organisation might take short cuts…

©Amland Consulting14 Risk Profile – Time & Material Testing take as long as it takes… Deadlines will be agreed in project plan Extensions must be documented Test organisation takes very limited risk Obvious problems To meet deadlines Agree criteria for extensions / acceptance criteria Pros. & Cons. Focus on quality and product risk, not deadlines only Complicated to control for client

©Amland Consulting15 Presentation Outline Managed test teams (MTT) Alternative risk and reward profiles Alternative test organisations Test Factory Outsourcing Summary

©Amland Consulting16 Test Factory – Objectives A permanent organisational unit, responsible for Test methodology Templates / standards Training Test environment (hw&sw) Services provided Training Test management Test environment management NOT responsible for Tests and test data Quality Hand-over to production

©Amland Consulting17 Test Factory Pros. & Cons. Benefits Small but permanent organisational unit Efficient / focusing on re-use across business Good control of test level in each project Efficient management of test environment Good integration with project management and development Drawbacks No additional resources, lack of flexibility Not focused on managing contractors

©Amland Consulting18 Outsourcing – Objectives Definition Tasks which are carried out by a third party Typically done offsite Option: Integrating tasks into current test process Building relations to an experienced test and project management organisation Providing experienced personnel when required

©Amland Consulting19 “Outsourcing” alternatives Outsourcing the total SQA function Adding Testing Expertise Adding Testers Insurance (“Double checking”) And variations…

©Amland Consulting20 Outsourcing Pros. & Cons. Benefits High flexibility regarding resources Test expertise availability Drawbacks Integration with internal organisation Team rivalry – lack of communication Potential contention between developers and testers Lack of business knowledge Must be internally managed

©Amland Consulting21 Summary Managed Test Team A viable Approach? Absolutely! Benefits Professional Expertise and flexibility Challenges Selecting the right teams and team structure Variation of Outsourcing

©Amland Consulting22 References Hamblen, G., Managing Outsourced Testing (On time and On Budget), Bampos, J., Outsourcing, STQE Magazine Nov./Dec. 1999, Volume 1, Issue 6,