Automated Testing April 2001WISQA Meeting Ronald Utz, Automated Software Testing Analyst April 11, 2001

Agenda  Reasons not to Automate  Reasons for Automation  Areas to Automate  Areas not to Automate  Automation Tools Used at Esker Software  Summary  Questions

Reasons not to Automate  Heavily weighted up front development times to plan and develop automated tests  Added costs for software, training and salary  Most automated tests can be done manually  Automated tests may contain flawed designs resulting in incorrect results  Although the tests themselves may be faster, the results must still be analyzed by a human  Missed bugs are possible if automated test results are not analyzed thoroughly

Reasons for Automation  Tests can be reused over and over again  Testers are better able to multitask  Testers are not required to be experts on every aspect of the product  Great for use on revisions to existing products  Eliminates missed errors that testers have looked at over and over but yet go unseen  Faster Testing (After development stage)  Regression testing

The Automation Decision The decision can be a difficult one. You must commit to it and provide the proper resources for it to be successful.

Areas to Automate  Repetitive tasks  Graphic Comparisons  Acceptance Tests  Regression Tests  Load/Stress Tests  File Transfers  Link Checks  Duplicate Shortcut keys  File Comparisons

Areas not to Automate  UI verifications,especially early in product life  Spelling/Grammar checks  Areas within the product that offer infinite choices/branches to follow  Areas of product that change often  Anything having to do with branding  Anything to do with Copyrights  Anything without well defined specifications  Help files verifications

Types of Tests we Run  We run host scripts on the server that should elicit a certain response in our program, then we verify if it occurs correctly by: — Getting screen text and compare to known value — Comparing screen graphic to a blessed image — Verifying correct settings on application dialogs  Verify that file transfers occur properly  Verify keystrokes are interpreted correctly  Check registry settings  Check memory usage

Automation Tools used at Esker  Borland Delphi used to create Front End App  Test scenarios written in Rational Visual Test  DDE used for ASCII text comparisons  OLE used for Macro language testing  Microsoft Access Database for data storage  VB 6 used to develop our add on utilities  Logging to ASCII text files

Esker’s Automated Testing Front End Program

Front End Features  Able to test multiple emulation types  Allows for the use of profiles  Creates log files for all selected tests  Optional Database logging  Database query tools built in  Includes a bitmap comparison utility  Optional registry comparison  It provides a lot of flexibility for our testing

Rational Visual Test Interface

Rational Visual Test Features  Integrates with Source Safe  Includes a scenario recorder  Language is a hybrid, some similarities to VB others to C  Has many built in functions designed to easily test PC and Web software  Able to step through and break at specific lines of code  Can be run on client PC’s with a small install

DDE Comparison to Text file We can scrap text from the screen using DDE and then do a text comparison to a known to be correct text file.

OLE Testing Interface

Database Query Utility Visual Basic program designed to perform detailed queries of the Access Database.

Bitmap Comparison Utility

Extract from a Log File

Summary  Automated testing is a long term commitment  Not everything can be automated but most anything that can be automated can be done manually  Chart out a long term plan as to what and when you want to automate things over time  Keep track of all of the pertinent data so you have baselines that can be compared to future tests  Acceptance tests are a good place to begin  Don’t automate anything that you’re not sure how to manually test  Don’t automate areas subject to change or that have loose and/or changing specifications

Questions?