1. Test Automation Failures - lessons to be learned DEV-TEST (SPRING) ‘99, UNICOM Seminar, Experiences with Test Automation Commonwealth Institute, London, 28 May 1999 Ståle Amland, Amland Consulting, Hulda Garborgsv. 2, 4020 STAVANGER, NORWAY Mob:+47 905 28 930, Phone: +47 51580587, Fax: +4751585524 e-mail: stale@amland.no WEB: www.amland.nostale@amland.no

2. Test Automation FailuresAmland Consulting2 Presentation Outline Introduction - The Projects Problems Test Tool Problems The Testers and their Problems Test Environment Problems Timing and “volume” of test automation Recommendations Test Tools - recommendations The Testers - recommendations The Environment - recommendations Timing and “volume” - recommendations Summary

3. Test Automation FailuresAmland Consulting3 Introduction - Project A Retail banking applications (server only) IBM Mainframe, CICS / DB2, Client / Server, “waterfall” Test bed: PC terminal emulator Test Phase: System Test, 6 months, 3,000 man-days Test tool: AutoTester Test Documentation: What to test! NOT How to test. Product specialist used for testing “Manual” Test Process: Recording for audit trail and “manual playback” Recording of scripts to be transformed into automated test scripts Resources (plan: 100% automation) 15%of all transactions were tested 25% of available testing resources 2.5% of all faults were detected

4. Test Automation FailuresAmland Consulting4 Introduction - Project B “Home Banking” application for small businesses Windows multi platform Client/Server application, RAD project 100 windows, 5 hardware platforms, 15+ mainframe systems 25 developers in 6 months, 25 testers in 3 months 8 mainly test management 2 test automation Test Phase: User Acceptance Testing (UAT) Test tools: SQA Suite Test Documentation: Structured documentation: Input / Output Used Test Tool for management Automated Test Process: Based on record and playback

5. Test Automation FailuresAmland Consulting5 Test Tool Problems The Test Tool does not support the Development Tools: Will not recognise objects Must rely on relative positioning Different Tools = Different Problems If you do Automation “wrong” - selecting the “right” tool will not save you!

6. Test Automation FailuresAmland Consulting6 The Testers and their Problems Product people as testers Conflict: Testing the system “manually” vs. Recording tests for later automated regression testing Result: demotivation - poor testing! Project B - RAD: Instability of software in UAT

7. Test Automation FailuresAmland Consulting7 Test Environment Problems Project A: Data Interference between testers Database “unique key” problems Relative Positioning: If resized: Tests recorded on one PC would not replay on other PCs Project B: …same problems + Test tool did not support development tools without separate.DLL Because of outsourcing: Supplier would not include test tool.DLL

8. Test Automation FailuresAmland Consulting8 Timing and “Volume” Problems Tried to automate too much at the same time Hectic recording process Lost too much information Massive Test script maintenance effort Only managed to automate 15% 100% automation objective became demotivating No prioritisation took place

9. Test Automation FailuresAmland Consulting9 Test Tools - Recommendations Prerequisites: Tool must support development and test environment and cover all (planned) production environments Next - must handle... documentation, planning, tracking and defect handling simultaneously from automated testing semi-automated testing manual testing Ability to record and log user input For QA or “Manual Playback”

10. Test Automation FailuresAmland Consulting10 The Testers - Recommendations Allow time for training! “On the job training” is needed First project will have low productivity Process should Combine Manual and Automated Testing Dedicated testers for automation Benefit to know programming Build a maintainable test suite Record manual tests: “Readable” not necessary “replayable”

11. Test Automation FailuresAmland Consulting11 Test Environment - Recommendations If possible: integrate test automation into development Separate environments for manual and automated testing Implement Configuration Management Allow enough time for Test Environment Maintenance It is very time consuming! Document shared test data carefully Periodically restore to initial state

12. Test Automation FailuresAmland Consulting12 Timing and “Volume” - Recommendations Start with “simple” functions I.e. changes to design / code / user interface are unlikely No “unique key” problems “Prove” benefits Identify time-consuming tests Pilot Project: Training of testers Build training material Do NOT plan 100% coverage first time!

13. Test Automation FailuresAmland Consulting13 Summary Automated Testing Benefits? YES Complicated? YES High resource requirements? YES Tools very useful in planning progress tracking defect tracking documentation management …as well as automation All projects will benefit from the use of a recording tool as part of test documentation and audit trail / due diligence

14. Test Automation FailuresAmland Consulting14 If you want to automate your testing, listen to people with experience before you start!

15. Test Automation FailuresAmland Consulting15 Ståle Amland Amland Consulting Hulda Garborgsv. 2, N-4020 STAVANGER Norway Mob: +47 905 28 930 Phone: +47 51 58 05 87 Fax: +47 51 58 55 24 e-mail: stale@amland.no Contact Details