1. 4 Nov 2000CCSC SE Grading Student Programs: A Software Testing Approach 14th CCSC Southeastern Conference November 3-4, 2000 Edward L. Jones CIS Department Florida A&M University Tallahassee, FL ejones@cis.famu.edu

2. 4 Nov 2000CCSC SE Outline Motivation Tester’s Approach Assignment Life Cycle An Example Lessons Learned Future Work

3. 4 Nov 2000CCSC SE In 25 Words or Less Automation is possible Requires up-front teacher investment Requires behavior adjustment by students Problem is well-structured High potential for reuse Just Do It!!

4. 4 Nov 2000CCSC SE Motivation Enhance the learning experience Grading is labor-intensive Difficulty of consistency Class size increases workload Redundancy of explaining deductions

5. 4 Nov 2000CCSC SE A Tester’s Approach Attitude -- finding what’s wrong! Consistent, repeatable error search process Careful documentation of results/findings Automation to reduce time investment Black-box testing based on specification.

6. 4 Nov 2000CCSC SE Programming Environment PR (public) Teacher SR (submission) Student submit getcopy postgrade getassignment viewgrade postassignment

7. 4 Nov 2000CCSC SE An Example COBOL programming assignment Example of HOW TO do it Good sense of up-front effort Repeatable/reusable pattern Expose difficulties Insights into possibilities

8. 4 Nov 2000CCSC SE Program Grading Life Cycle Prepare AssignmentImplement GraderGrade Programs Write Program Grading Log Grade Report Student Program Assignment Specification Assignment Specification Test plan Test cases Test driver Checker script Automated Grader

9. 4 Nov 2000CCSC SE Lessons Learned Student shock & outrage at exactness Specification must be better than average! Front-end loaded: Assure testability up front First-time automation is costly Amortized via similar assignment styles Students need the grader before submitting Students need to learn how to read, interpret and satisfy a specification

10. 4 Nov 2000CCSC SE Automation Challenges Does the teacher have the time? Is the automated grader TOO STRICT for CS1/CS2? What to ignore? Deduction schedules become complex – aggregate vs specific search strategy The “just a little more automation” trap!! The guts to do it!!

11. 4 Nov 2000CCSC SE Future?? Do it again! Do it on another class, CS1-CS3! Investigate table-driven checker scripts – reduce costliest step. Distribute grader along with assignment! Sell colleagues on the idea!

12. 4 Nov 2000CCSC SE Questions?

13. 4 Nov 2000CCSC SE Thank You

14. 4 Nov 2000CCSC SE Interactive Input Requirements (1) (1) The program must be menu-driven based on TRANSACTION CODE. ----------------------------------------------- ENTER TRANSACTION CHOICE: 1 -- Open Account 2 -- Deposit 3 -- Withdraw 4 -- Close Account 5 -- Check Balance 9 -- QUIT PROGRAM ------------------------------------------------

15. 4 Nov 2000CCSC SE Interactive Input Requirements (2) (2) Sequence of interactive data entry, per transaction code: Open (01) : Account#  Amount  Lastname  FirstName Deposit (02) : Account#  Amount Withdraw (03) : Account#  Check#  Amount Close (04) : Account#  LastName  FirstName BalCheck (05) : Account#

16. 4 Nov 2000CCSC SE Output Requirements - Messages A. VALIDATION ERRORS - BAD TRANSACTION CODE - DEPOSIT AMOUNT TOO HIGH - DEPOSIT AMOUNT TOO LOW - MISSING ACCT OWNER NAME - MISSING INITIAL DEPOSIT B. UPDATE ERRORS - - ACCOUNT ALREADY EXISTS - ACCOUNT DOES NOT EXIST - ACCOUNT HAS NON-ZERO BALANCE - INSUFFICIENT FUNDS C. UPDATE COMPLETION - DEPOSIT MADE - WITHDRAWAL MADE - BALANCE DISPLAYED - ACCOUNT OPENED - ACCOUNT CLOSED

17. 4 Nov 2000CCSC SE Processing Requirements BUSINESS RULES (excerpts) 2. An initial deposit AMOUNT must be given for OPEN. 3. The initial deposit AMOUNT must be at least $20.00. 5. The deposit AMOUNT must NOT exceed 10,000.00. 6. A withdrawal AMOUNT may not exceed the Balance. 7. An account can not be closed unless it has a ZERO balance. 10. The OPEN account must FAIL if the account already exists.

18. 4 Nov 2000CCSC SE Test Plan (1) 1. There will be three test runs 1.1 Nominal -- valid transactions, successful updates Goal: Demonstrate correct behavior in "perfect" world. 1.2 Update Errors -- valid transactions, update failures. Goal: Demonstrate ability to detect and correctly respond to master file conditions that prevent update. 1. 3 Transaction Errors -- invalid transactions Goal: Demonstrate ability to detect and correctly respond to invalid transactions.

19. 4 Nov 2000CCSC SE Test Plan (2) 2. All test runs use the same master file. 3. Test Execution & Results Verification 3.1 For each test run there will be an input test data set 3.2 For each test run there will be a verification script that scans the program output file (p7_audit.rpt) for expected results. 3.3 Each verification script tallies deductions for failure to produce expected output. 3. The verification scripts write results to log file p7_user.log.

20. 4 Nov 2000CCSC SE Test Plan (3) 4. File Naming Conventions Test run 1 2 3 Test Data Settestdata1 testdata2 testdata3 Verification Script checkscript1 checkscript2 checkscript3

21. 4 Nov 2000CCSC SE Test Driver # ------------------------------------------------- # Test Driver: Perform 3 test runs. # ------------------------------------------------- set slog=p7_{$1}.log echo "p7 Grading Log for student $1" > $slog set totpen=0 foreach run ( 1 2 3) #-| Run program & check results. cobrun $1 < testdata$run csh checkscript$run $1 p7_audit.rpt set runpen = $status @ totpen=$totpen + $runpen echo “$1–RUN $run PENALTY = $runpen" end # for loop echo “Student $1 – TOTAL PENALITIES = $totpen" >> $slog exit

22. 4 Nov 2000CCSC SE Test Cases Run #1 NOMINAL Test Cases VALID transaction, SUCCESSFUL update Test Sequence / Expectation: 1) BAL CHECK -- ACCT (1111) / Bal = 250.55 2) DEPOSIT -- ACCT (1111) 249.45 3) BAL CHECK -- ACCT (1111) / Bal = 500.00 4) CLOSE -- ACCT (8888) Wine Brandy 5) WITHDRAW -- ACCT (9999) 26.00 6) BAL CHECK -- ACCT (9999) / Bal = 175.00 7) OPEN -- ACCT (5555) 876.54 Jones Ed 8) BAL CHECK -- ACCT (5555) / Bal = 876.54 ------------------------------- When Master File is: ------------------------------- 1111 000025055 010190 McNair Stub 2222 000560000 020293 Simmons Joe 3333 000000100 123199 Fisher Kelly 4444 000000750 012500 Broke Eyem 6666 000998900 071092 Beuche Bobby 8888 000000000 011595 Wine Brandy 9999 000002100 020100 Faulk Mark

23. 4 Nov 2000CCSC SE Checker Script (1) # ------------------------------------------------------- # Filename checkscript1 # Purpose: Determine deductions for wrong output. RETURN #points deducted. # # Invocation: checkscript1 student resultsfile # ------------------------------------------------------- set pen=0 set log=p7_{$1}.log #-| AGGREGATE counts -- transaction completion messages. #-| Search results written to student's grading log. foreach msg (OPENED CLOSED DISPLAY WITHDRAW DEPOSIT) grep $msg $2 >> $log if ($status) then @ pen = $pen + 1 echo "$1 - WRONG - missing $msg MESSAGE" >> $log endif end

24. 4 Nov 2000CCSC SE Checker Script (2) #------------------------------------------------------------------------ #-| SPECIFIC Search: 249.45 DEPOSIT accepted. egrep "[0]*2 " $2 | grep 249.45 >> $log if ($status) then @ pen = $pen + 1 echo "$1 - WRONG - missing DEPOSIT 249.45 MESSAGE" >> $log endif #-| SPECIFIC Search: 1111 DEPOSIT applied correctly. egrep "1111" $2 | grep 500.00 >> $log if ($status) then @ pen = $pen + 1 echo "$1 - WRONG - BALANCE after DEPOSIT not 500.00" >> $log endif … #-| Return the penalty points. echo "$1 - script1 PENALTY POINTS = $pen" >> $log exit $pen

25. 4 Nov 2000CCSC SE Program Grading Log (1) p7 Grading Log for tmorris ____________________________ 007 SUCCESS ACCOUNT OPENED tmorris - WRONG - missing CLOSED MESSAGE 001 SUCCESS BALANCE DISPLAYED 003 SUCCESS BALANCE DISPLAYED BALANCE DISPLAYED 005 SUCCESS WITHDRAWAL POSTED 006 SUCCESS BALANCE DISPLAYED 007 SUCCESS ACCOUNT OPENED 008 SUCCESS BALANCE DISPLAYED 001 VALID 002 VALID 003 VALID 004 INVALID MISSING AMOUNT 008 VALID 002 2 1111111 249.45 003 1111111 500.00 McNair Stub 007 1 5555555 30300 876.54 Jones Ed 75555555 876.54 000000 Jones Ed tmorris - script1 PENALTY POINTS = 1

26. 4 Nov 2000CCSC SE Program Grading Log (2) 003 VALID 4 ACCOUNT CLOSED tmorris - WRONG - extraneous CLOSED MESSAGE tmorris - WRONG - missing FAIL MESSAGE 001 INVALID ACCT DOES NOT EXIST 002 INVALID ACCOUNT ALREADY EXI tmorris - WRONG - CLOSE -- missing NON-ZERO BALANCE message 004 INVALID INSUFFICIENT FUNDS 005 INVALID ACCT DOES NOT EXIST tmorris - script2 PENALTY POINTS = 3 001 SUCCESS ACCOUNT OPENED 003 SUCCESS ACCOUNT OPENED tmorris - WRONG - extraneous OPENED MESSAGE 004 INVALID DEPOSIT AMOUNT TOO HIGH 005 INVALID DEPOSIT AMOUNT TOO LOW tmorris - WRONG - extraneous DEPOSIT MESSAGE 004 INVALID DEPOSIT AMOUNT TOO HIGH 001 1 7777777 30300 5.00 OPENDeposit TooRLOW 001 7777777 5.00 000000 OPENDeposit TooRLOW tmorris - WRONG - OPEN -- missing MISSING ACCT OWNER NAME message 004 INVALID DEPOSIT AMOUNT TOO HIGH 005 INVALID DEPOSIT AMOUNT TOO LOW tmorris - WRONG - OPEN -- missing MISSING INITIAL DEPOSIT message tmorris - script3 PENALTY POINTS = 5 tmorris - TOTAL RUN PENALITY POINTS = 9

27. 4 Nov 2000CCSC SE Program Grading Life Cycle Prepare AssignmentImplement GraderGrade Programs Write Program Grading Log Grade Report Student Program Assignment Specification Assignment Specification Test plan Test cases Test driver Checker script Automated Grader

28. 4 Nov 2000CCSC SE SPRAE - A Tester’s Framework Specification: basis for testing Premeditation: no plan, no test Repeatability: deterministic outcome. Accountability: full disclosure Economy: cost-effectiveness.