1. Automated Testing April 2001WISQA Meeting Ronald Utz, Automated Software Testing Analyst April 11, 2001

2. Agenda  Reasons not to Automate  Reasons for Automation  Areas to Automate  Areas not to Automate  Automation Tools Used at Esker Software  Summary  Questions

3. Reasons not to Automate  Heavily weighted up front development times to plan and develop automated tests  Added costs for software, training and salary  Most automated tests can be done manually  Automated tests may contain flawed designs resulting in incorrect results  Although the tests themselves may be faster, the results must still be analyzed by a human  Missed bugs are possible if automated test results are not analyzed thoroughly

4. Reasons for Automation  Tests can be reused over and over again  Testers are better able to multitask  Testers are not required to be experts on every aspect of the product  Great for use on revisions to existing products  Eliminates missed errors that testers have looked at over and over but yet go unseen  Faster Testing (After development stage)  Regression testing

5. The Automation Decision The decision can be a difficult one. You must commit to it and provide the proper resources for it to be successful.

6. Areas to Automate  Repetitive tasks  Graphic Comparisons  Acceptance Tests  Regression Tests  Load/Stress Tests  File Transfers  Link Checks  Duplicate Shortcut keys  File Comparisons

7. Areas not to Automate  UI verifications,especially early in product life  Spelling/Grammar checks  Areas within the product that offer infinite choices/branches to follow  Areas of product that change often  Anything having to do with branding  Anything to do with Copyrights  Anything without well defined specifications  Help files verifications

8. Types of Tests we Run  We run host scripts on the server that should elicit a certain response in our program, then we verify if it occurs correctly by: — Getting screen text and compare to known value — Comparing screen graphic to a blessed image — Verifying correct settings on application dialogs  Verify that file transfers occur properly  Verify keystrokes are interpreted correctly  Check registry settings  Check memory usage

9. Automation Tools used at Esker  Borland Delphi used to create Front End App  Test scenarios written in Rational Visual Test  DDE used for ASCII text comparisons  OLE used for Macro language testing  Microsoft Access Database for data storage  VB 6 used to develop our add on utilities  Logging to ASCII text files

10. Esker’s Automated Testing Front End Program

11. Front End Features  Able to test multiple emulation types  Allows for the use of profiles  Creates log files for all selected tests  Optional Database logging  Database query tools built in  Includes a bitmap comparison utility  Optional registry comparison  It provides a lot of flexibility for our testing

12. Rational Visual Test Interface

13. Rational Visual Test Features  Integrates with Source Safe  Includes a scenario recorder  Language is a hybrid, some similarities to VB others to C  Has many built in functions designed to easily test PC and Web software  Able to step through and break at specific lines of code  Can be run on client PC’s with a small install

14. DDE Comparison to Text file We can scrap text from the screen using DDE and then do a text comparison to a known to be correct text file.

15. OLE Testing Interface

16. Database Query Utility Visual Basic program designed to perform detailed queries of the Access Database.

17. Bitmap Comparison Utility

18. Extract from a Log File

19. Summary  Automated testing is a long term commitment  Not everything can be automated but most anything that can be automated can be done manually  Chart out a long term plan as to what and when you want to automate things over time  Keep track of all of the pertinent data so you have baselines that can be compared to future tests  Acceptance tests are a good place to begin  Don’t automate anything that you’re not sure how to manually test  Don’t automate areas subject to change or that have loose and/or changing specifications

20. Questions?