1. Regression Testing – What to Automate and how Assurance with Intelligence Paul Gerrard Gerrard Consulting 1 Old Forge Close Maidenhead Berkshire SL6 2RD UK e: paul@gerrardconsulting.com w: http://gerrardconsulting.com t: 01628 639173

2. Paul Gerrard Paul is the founder and Principal of Gerrard Consulting, a services company focused on increasing the success rate of IT-based projects for clients. He has conducted assignments in all aspects of Software Testing and Quality Assurance. Previously, he has worked as a developer, designer, project manager and consultant for small and large developments using all major technologies and is the webmaster of gerrardconsulting.com and several other websites. Paul has degrees from the Universities of Oxford and London, is Web Secretary for the BCS SIG in Software Testing (SIGIST), Founding Chair of the ISEB Tester Qualification Board and the host/organiser of the UK Test Management Forum conferences. He is a regular speaker at seminars and conferences in the UK, continental Europe and the USA and was recently awarded the “Best Presentation of the Year” prize by the BCS SIGIST. Paul has written many papers and articles, most of which are on the Gerrard website. With Neil Thompson, Paul wrote “Risk-Based E-Business Testing” – the standard text for risk-based testing. Assurance with IntelligenceSlide 2

3. In this session, I want to re-trace our early steps to automation Is life hard at the end because we didn’t think it through the beginning? Assurance with IntelligenceSlide 3

4. Automating regression tests Regression tests are the most likely to be stable and run repeatedly so: – Automation promises big savings of time – Automation ensures reliable execution and results checking – Stable tests/software are usually easiest to automate Hold on! Paradox ALERT!!!! – We regression test because things change – Chaotic, unstable environments need it the most – When things change automation is HARD! Slide 4 Assurance with Intelligence

5. What is the regression testing thought process? Assurance with IntelligenceSlide 5

6. How I think about regression testing Establish what we mean by regression testing Understand why regressions occur Combating regression – there are choices Who are the stakeholders for regression testing and what do they want? Context of regression testing Regression test policy Selecting what to regression test Selecting what regression tests to automate Assurance with IntelligenceSlide 6

7. A common experience? Regression testing is a)Painful b)Boring c)A waste of time “Let’s get a tool in” Automating 10% of tests is easy Automating 90% of tests is hard (impossible?) “Why are we doing this?” “Let’s do it manually” Assurance with IntelligenceSlide 7

8. Let’s look at each thought process Assurance with IntelligenceSlide 8

9. (What is...) regression testing? When a software fault is fixed, it often happens that 'knock-on' effects occur We need to check that when changing the faulty code, the developer has not introduced new faults There is a 50% chance of introducing regression faults Regression tests tell us whether new faults have been introduced – i.e. whether the system still works after a change to the code or environment has been made. When environments are changed, we might also regression test. Slide 9 Assurance with Intelligence

10. Regression testing 2 “Testing to ensure a change has not caused faults in unchanged parts of the system" Not necessarily a separate stage Regression testing most important during maintenance activities Regression tests are usually the first to be considered for automation. Slide 10 Assurance with Intelligence

11. How do regressions occur? Something changes and affects ‘working’ software An enhancement is implemented A bug is fixed Two bug fixes interfere with each other One programmer overwrites another programmer’s fix (configuration management) Something in the environment setup Test data changes in some way A test changes in some way Test configuration/setup changes Etc. etc. Can you think of others? Assurance with IntelligenceSlide 11

12. Code fixes have a 50% chance of introducing side-effects in working software Ancient history – who cares whether it’s 10% or 90%? Regressions have a disproportionate impact on effort, confidence, morale Assurance with IntelligenceSlide 12

13. Combating regression - choices Several options: – Don’t change anything! – Impact analysis (of code/data//test changes) – Regression testing (to detect unplanned changes) Impact analysis is tricky; not 100% reliable but... – Compare source code – see what’s changed – Module dependencies – see what’s affected – Rigour in managing environments and data – Rigour in managing source code Regression testing choices – Manual v automated – Test first or test last – All tests or a selection of tests – Others? Assurance with IntelligenceSlide 13

14. Who are our stakeholders? Stakeholders are people interested in the outcome of test What do they want? Ideally, nothing! Sponsors, customers – Regressions are an immediate setback/delay just when they thought you could move the project forward Users and testers – “All our good work is undermined!” Developers – They’ve missed something obvious, embarrassing; unplanned work – could be trivial or substantial Tech support – “Users are going to be furious!” – “We feel really let down by dev, maintenance, testers” Assurance with IntelligenceSlide 14

15. Context of regression testing Test- or behaviour-driven development: – All tests are automated; these are our safety-net – Tests allow us to refactor with confidence Integration/System/Acceptance testing 1 st pass – RT after main testing done – confirmation that all tests pass with all fixes in place Maintenance phase – Minor fixes (don’t get impact analysed) – Enhancements fit with existing system – Refactoring/conversion/interface changes. Assurance with IntelligenceSlide 15

16. Regression test policy Common: – Business critical, high risk, error-prone areas – FIFA – first in, first to automate? – LIFA – last in first to automate – Easily automatable But let’s step back a little – Focus on stakeholders – What do THEY actually want? Assurance with IntelligenceSlide 16

17. What would stakeholders want? Sponsors, customers – Want to avoid delays – Is the most critical functionality working? Tech support – Are we inconveniencing the users? – Is the most critical functionality working? Users and testers – Can we trust the results of our early (high priority tests) – (Maybe we aren’t so worried about the rest?) – Is the most critical functionality working? Developers – Want instant feedback (so they can fix with least hassle) – The best case for them doing their own automated tests! Assurance with IntelligenceSlide 17

18. If a developer ran your test team... Whose problem is regression anyway? It’s developers’ problem not the testers’! Dev have been asked to fix/enhance/refactor code without screwing up How would a developer address the regression problem? – Design a harness and code to execute transactions at the API level – Easier, more maintainable, less sensitive to cosmetic changes A reusable asset that allows refactoring. Assurance with IntelligenceSlide 18

19. The real problem to solve? If the problem is identifying change where there shouldn’t be any... – With a few caveats, it doesn’t matter what tests we run as long as we cover the important functionality – and that it behaves the same way as it did last time – We still need to demonstrate the user interface isn’t adversely affected and the end-to-ends still work – Is it POSSIBLE and BETTER to do this manually? – Why use a framework on top of a test execution tool on top of a browser to test server-based code? Too complex! Where does that leave automated test execution – on the shelf? (It’s probably there already!) Assurance with IntelligenceSlide 19

20. Assurance with IntelligenceSlide 20

21. Techniques for Testing GUI Applications Eurostar 1997: the most popular paper on my website is referenced by Wikipedia http://gerrardconsulting.com/sites/default/files/ Techgui.pdf http://gerrardconsulting.com/sites/default/files/ Techgui.pdf Won’t repeat the whole story here... But most of the 1997 slides are still relevant Have we made NO progress since 1997? Assurance with IntelligenceSlide 21

22. Selecting what regression tests to automate 1997 Assurance with IntelligenceSlide 22

23. Manual v automatic scripts - very rough guideline

24. What aspects to automate? Pareto law: – go for the easy scripts first – don’t waste time scripting low volume complex scripts at the expense of high volume simple ones Consider hybrid approach: – use the tool to do navigation and data entry – do the checking on-line or at end of test.

25. Automated test scripts Coded scripts – navigation through the system, some checklist test cases can be can be coded directly – relatively easy to maintain Recorded scripts – automated scripts should be documented – then used to record the automated script – less easy to maintain Integrate all scripts into automated suite.

26. Migration of manual scripts Automated scripts are easier to maintain – if designed to be maintained – Means you have to build test automation scaffolding before you script tests – Automation projects are software development projects, not test projects Allow for maintenance cost before deciding to migrate functional scripts – delay migration until system is stable – soak and regression tests, as required.

27. Conclusion The regression problem is a developers’ problem that can be solved by: 1.Developer discipline 2.Better impact analysis 3.Automated testing at an API level 4.Manual/automated testing (critical/E2E level) for visibility to stakeholders Is this what the big product companies do? Assurance with IntelligenceSlide 27

28. Regression Testing – What to Automate and how Assurance with Intelligence Paul Gerrard Gerrard Consulting 1 Old Forge Close Maidenhead Berkshire SL6 2RD UK e: paul@gerrardconsulting.com w: http://gerrardconsulting.com t: 01628 639173