1 Title slide Future for Functional Test Automation? TM Forum – April 2006 Susan Windsor Insight Through Intelligence WMHL Consulting Limited, MD

2 AGENDA Future for Functional Testing Challenges for Functional Automation Strategic Direction What does this mean to you?

3 Future for Functional Testing

4 Increased Testing Demand………..? Global development, with large projects having multi site, multi geography and multi suppliers to contend with. Corporate and regulatory requirements growing all the time Business demands new applications, faster and cheaper to obtain competitive advantage Closer alignment of IT and business to repair lack of business confidence

5 Or, increased competition for roles? Business testers using frameworks reduces the number of traditional functional testers required Agile development methods mean developers undertake more unit and component testing Growth of outsourced testing to different geographies Is the role of the UK functional tester (who is neither technical nor business specialist) dead? With the focus on skilling up functional testers, are we fiddling whilst Rome burns?

6 Challenges of Functional Automation

7 Record and Playback “all you need to do…….…!” Easy to record the scripts Extremely fragile Expensive to maintain On average, each test run requires at least 50% of the scripts to be recorded again Please tell me no one does this anymore!!!

8 Scripting Use the scripting language to write scripts that do what you want Build in as much robustness as possible But, you’re building an application to test an application, which also needs testing! Maintenance costs can be very high Needs programmer skills Cost of implementation not affordable at project level Challenging for Test Managers to incorporate in overall test plans

9 Table Driven Like scripting but more flexible and greater re-use Remove items from the script that change, e.g. data Reduces maintenance costs a bit Implementation costs about the same Still requires specialist skills to implement Still a challenge for Test Managers to control

10 Functional Test Automation is Broken! Focus on technology rather than business needs 80% of functional testing still manual 60% to 70% of automation tools used for non-functional testing Typically, traditional functional automation stops at 100 scripts, regardless of test coverage requirement Critical factors – Cost of implementation and maintenance prohibitive – Insufficient and expensive skills required – Inability to asset share over different technologies

11 Project Structure also a challenge Build & TestSys Test Int Test Support & Maintenance Acceptance Outsource Initiation Handover Milestones Different companies, different teams, different skill sets, tight budgets (cost of automation not always considered at initiation time!!) External Influences Consultants Suppliers Products Business Partners

12 Strategic Direction

13 Automation Frameworks Wouldn’t it be good if…….. – Tests could be documented in common format, regardless of whether they are manual or automated – The format for the tests resulted in quicker preparation time than traditional manual tests – Both developers and business testers could understand and use the same test format – Script maintenance was no longer a requirement – Tests could use any of the test execution tools required by the underlying technology Faster planning, faster execution, and far less technical skill required

14 Business Analysts already using them, and use will grow Home grown frameworks built within organisations to meet business demands Niche suppliers providing frameworks; try a Google search – 512,000 hits this week – Seen a handful that appear mature – Latest review by Paul Herzlich (OVUM analyst) Market Leaders such as Mercury developing Business Process Tester (BPT) This is the industry direction now

15 Manual Test Execution & Documentation UI Automated Test Execution Tools Test Results Non UI Component Test Execution Harness Requirements Test Management Governance Project ManagementAnalysis & Design Test Case Definition – Automation Framework Test Case Documentation Test Requirements Sign Off Where Frameworks Fit

16 Frameworks & Industry Needs It’s too early to automate in our project – Define the test cases in the Framework now, and automate later We haven’t got manual testing under control yet! – Use the Framework as the standard test case definition and use for manual and automated scripts We do Agile development, functional automation doesn’t fit – Define test cases before the development commences and use the documentation to obtain business signoff against requirements, as soon as code is written, you can automate

17 Automation consultants are too expensive to increase our automation test coverage the way we’d like to – You can fully utilise your manual testers to build test cases to generate automated scripts, your automation specialist can concentrate on the more exciting aspects of test automation and multiple projects can be support by a single technical resource. Our offshore partner’s use automation but we are concerned at the lack of visibility over exactly what their tests do – Full test documentation can be generated to demonstrate what the automated scripts will do so that you retain control Frameworks & Industry Needs - 2

18 Frameworks & Industry Needs - 3 We are tied into our Automation Supplier and even though we’re not comfortable with that, we can’t see a way out – Transfer your test case definitions into Framework now, initially prove your test cases via interaction with existing tools and you can switch when ever you like We can’t afford the cost of automated testing on this project – Using a Framework will reduce the cost of automation implementation by between 58% and 75% over traditional approaches We find the cost of maintenance prohibitive to extend our test automation further – Using a Framework will reduce the cost of automation maintenance costs by between 75% and 85%

19 What Does This Mean to You

20 Discussion Time What is your organisation’s direction? What is your view on Automation Frameworks? Do they have a future? Do you require different skills? What is your future?

21 Closing slide Thank You Susan Windsor WMHL Consulting Limited, MD