1 The Pain and Gain of Test Automation – the early days Andy Redwood Portman Building Society

2 Portman Group 3 rd Largest Building Society in UK staff, 142 branches, HO in Bournemouth 2005 Total assets increased by 15% to £17.8 billion 2005 Residential mortgage balances up by 16% to £13.5 billion 2005 New residential lending up by 18% to £4.3 billion 2005 £237m of net retail inflows (Society) 2005 Retail savings balances increased to £11.4bn

3 Agenda Starting position Evaluation Buy-in Solution – first stage People What’s going well? Core issues Where are we going next?

4 The first 10 days 35 observations 43 suggestions Cross-referenced to risk table 19 actions agreed with Internal Audit 18 of 19 completed to date

5 Problem Statement June 2005 Need to be more ‘merger friendly’; Need to accommodate a diverse and changing architecture; Business applications must be resilient to change; Life-cycle processes are reactive; Testing takes too long; Accountability over the testing is weak; High Defect Rate in later test phases; High re-work rate; Testing Skill sets are being questioned;

6 Buy-in Management Bigger Picture CIO and executive Mandate for change Budget for tools Resource Strategy Training programme Team dynamic Test management Flexible process Multi-project focus Pragmatics Prioritising Communications

7 Gaining Control – Defining the End Game Define the problem – Decision point - Define the End Game Days Life-cycle changes Decision point – Automation (Winrunner/Quick Test Pro) months Revised Testing Strategy (by workshop) Pilot Programme – Champion required Decision point – good to go? Change Programme Staged Iterative Improvements By Programme Phased Improvement Quick Wins Action Now - Test Leadership, Proactive Risk-focus

Test Strategy 57 page Test Policy ‘The way we work’ ‘The way we want others to work with us’ Test Approach by exception Supported by Standards and User Guides

9 Testing – Red Flags TEST EXECUTION IMPLEMENTATION DEFINITION ANALYSIS DESIGNBUILD INITIATION Data Environment Test Manual Tests held In Access Process is Long & error prone Tests are duplicated Test does not Target risk Defects are missed Test can take months to complete Dependencies unknown Test Execution Restarts Every time 1 person 1 test Lack of early involvement Insufficient Reporting Environment Rebuild time Limited E2E Capability

10 June 2005 Team Structure 11 Manual Testers

11 IS Testing Flexible Working Group Business Manager IS Testing Test Manager Mortgage Channel Test Manager BAU Projects Test Manager Environment and Tools Test Manager Investment Projects Test Manager Regulatory Projects Test Manager Broker Channel Flexible Resource Pool – IS, IST, Business Secondments, External Resource

12 Stage1 Solution Write, publish and sign off the Strategy; Standardised but flexible process; Testing Services function; Accountability and ownership of the test; Flexible test resourcing from professional testing organisations – annual tenders; Integrated Test management Toolset – Test Director; Optimised Automated Functional and regression;

13 High Test Management focus Lower Test Management focus High Test Execution focus 1 Test Manager 3 Coordinators 12 Testers Defined Business Role Retained Expertise Hire Testers on demand Lower Test Execution focus Higher Cost Lower Cost No go zone Flexible Resourcing Risk-based Flexibility

14 Time Cumulative Defects High Detection Rate Low Detection Rate Test on or implement now Undetermined time saving Law of Diminshing Returns Risk Decision Point deploy now? Implementation date Requirements AnalysisDesignCodeTest Informed Deployments

15 Implementation date Requirements AnalysisDesignCodeTest No. of Tests Run Extended Testing Day Manual Testing In 8hr day Automated Testing In 14-18hr day Higher Test Volume Higher error detection rate Fewer Testers Production Stability Lower Costs

16 TEST EXECUTION IMPLEMENTATION DEFINITION ANALYSIS DESIGN BUILD INITIATION Data Environment Test Automated Where are we now? Input Test data Can be Scripted Tests Held in TD8 Tests defined to a multi-project Repository Real Time Prep by many Testers Data Load is Scripted Environment Performance Monitoring Automatic Environment Sanity Check Executions can be networked Execution is Unattended n-desktops On Demand Just enough Regression Testing Test Advise in Early stages Enterprise Test Strategy Continuity of Standards & Process Informed Decisions Measures & Metrics

17 Next Steps Integrate more across departments Roll out TD8 into development – move dynamic defect focus into Unit Testing Comparable execution toolsets across platforms Test Process Toolbox – more detail on the process Sort out our environments

18 Thanks for inviting us to your SiGiST Andy Redwood