1. CEN 5070 – Software V&V Automation for Program Testing © 2001-2008, E.L. Jones

2. October 2008Program Test Automation2 Purpose This lecture introduces design patterns to automate the testing of executable programs (or applications). The testing lifecycle phases involved are implementation, execution and evaluation.

3. October 2008Program Test Automation3 Agenda Revising the Test Script The Need for Shell Scripts Putting it All Together Invoking program under test Set-up: required files Insertion of stored test cases Capture and formatting of test results Test execution instructions Verification of test results

4. October 2008Program Test Automation4 Shell Scripts Shell scripts automate the use of operating system commands and utilities for program execution, and management of test data and results. We begin with the test environment.

5. October 2008Program Test Automation5 Application Test Environment Application Driver Test data Application Under Test Test History Index ‘thx_xxx.txt’ Verification Actual results Test results ‘utr_xxx.txt’ Test History Index ‘thx_xxx.txt’ Test Case Data ‘tdi_xxx.txt’ Test Script ‘uts_xxx.txt’ Test Log ‘utl_xxx.txt’ 1 3 4 2 5 6 7 8 9

6. October 2008Program Test Automation6 0. Application Driver Specification Application Test Driver (1) tdi_ filename (2) thx_ filename (3) utr_ filename thx_ file utr_ file tdi_ file thx_ file Application.run file

7. October 2008Program Test Automation7 1. Automated Invocation of Interactive Application echo 5 4 |./Add.run echo “5 4” |./Add.run  Pipe inputs into stdin stream. Add.run reads inputs from stdin.  WARNING: Piping as a quoted string does not work properly.

8. October 2008Program Test Automation8 2. Automated Insertion of Stored Test Cases set testcase=(`cat tdi_Add.txt`) set num1=$testcase[1] set num2=$testcase[2] set expected=$testcase[3] echo $num1 $num2 |./Add.run shift testcase  Create list of all test cases in in file. A list is an array $testcase[1..$#testcase]  Extract test case into application specific variables.  Pipe first two stimui as input to stdin and invoke Add.run  Each shift command removes one (used) value from front of the list.

9. October 2008Program Test Automation9 3. Automated Capture of Execution Results set res = `echo $num1 $num2 |./Add.run` Set sum=`echo $res | awk ‘{print $5}’`  Variable $res contains application output (prompts and results).  Based on the FORMAT of output, “parse for” the results using awk.. Example: x=## y=## SUM = 14  The answer (sum) is the 5th token in the output.

10. October 2008Program Test Automation10 @ tcnum++ echo –n “$runid $tcnum num1=$num1 num2=$num2 “ echo “SUM= ” >> utrFile NOTES:The $runid must be managed (see next slide). The name of the utr_ file is an argument. After the testcase data have been used, remove them in preparation for the next test case.  Create utr record showing stimuli and results. Record unique by testrun# and testcase#. 4. Store Formatted Test Result Record

11. October 2008Program Test Automation11 5. Set the Test Run# if (! –e thx_File) then #-| generate header lines. endif set runid=`tail -1 thxFile | awk –F# ‘{print $1}’` #-| The current runid is 1 +previous runid. @ runid++  If the thx_ file does not exist, create the file with the required header lines. NOTE: The thx_ file name is the second argument.  the last line in the thx_ file contains the runid.

12. October 2008Program Test Automation12 6. Update Test History Index #-| Write runid to thx_ file. echo “#$runid $user `date`“ >> thx_File #-| Write runid to thx_ file. echo “#$runid $user `date`“ >> utr_File  Append a line to the thx_ file to record the fact that this test run is beginning.  Also append a line to the utr_ file to create a “section heading” for the results from the test run.

13. October 2008Program Test Automation13 7. Update Test Script SETUP Section TEST RUN Section B.Unit Execution C.Test Log Preparation D.Test Sequence  Include run_ shell script, test log, and the files used as inputs,  Modify to reflect automated steps.  Include invocation of run_ script.  Log is required if run_ script does not contain an oracle.  Remove individual test cases: these will now be read from the tdi_ file.

14. October 2008Program Test Automation14 7. Update Test Script (cont’d) CLOSE-OUT Section A.Test Log B.Test Documentation Files C.Saving Artifacts to Repository  This is the EVALUATION and ARCHIVAL activity.  Explain how to update test log from trs_ and thx_ files, if oracle not implemented. MUST describe each discrepancy and propose probable cause of defect.  Identify each artifact and how it is to be used/handled. Some may have to be modified.  All go to TS repository.file.

15. October 2008Program Test Automation15 8. Update Test Log Test Log MUST include: a) Verification Summary; b) Description of each defect/ hypothesis of root cause. c) Describe any “fixes” made to enable the test to be run again.  Count all defects (unexpected results/inability to perform steps)  Description should be specific and general (if a pattern is seen).  Hypothesis is an educated guess as to the root cause.  Program is wrong  Test case is wrong  Test automation is wrong  Fixes to test cases or various scripts are the responsibility of the tester.

16. October 2008Program Test Automation16 9. Implement Test Oracle #-| Case 1: One exected result. if ($res == $expected) then echo “ PASSED” >> utr_File else echo “ FAILED” >> utr_File endif echo –n “#$runid $user `date` “ >> utr_File echo “$passed PASSED $failed FAILED“ >> utr_File  Compare actual result to expected result, and append “PASSED” or “FAILED” to test record.  NOTE: Include COUNTERS for passes and failures. These can be included in FOOTER records in both utr_ and thx_ files.

17. October 2008Program Test Automation17 9. Implement Test Oracle (cont’d) #-| Case 2: Multiple expected #-| results (e.g. 3). @ match1 = ($res1 == $exp1) @ match2 = ($res2 == $exp2) @ match3 = ($res3 == $exp3) @ matched = $match1 + $match2 \ $match3 If ($matched == 3) then echo “ PASSED” >> utr_File else echo “ FAILED” >> utr_File endif  Note that the result from each comparison is 1 (true) or 0(false). ALL must be true to PASS.  NOTE: Include COUNTERS for passes and failures.

18. October 2008Program Test Automation18 Shell Scripts for Testing Programs -- Summary Unix provides a powerful set of primitive operating system and scripting facilities for automating the testing of programs. This lecture has shown that automation can be applied to support most of the SPRAE principles. The design patterns shown can be extended to programs of moderate complexity.