1. 15 Nov 2005 Jones - IASTED/SEA 1 Automated Support for Test-Driven Specification Edward L. Jones IASTED SEA Conference Phoenix, Arizona October 14-16, 2005

2. 15 Nov 2005 2Jones - IASTED/SEA Outline Test-Driven Specification Paradigm Test-Driven Specification Paradigm Background Background An Example An Example Automation Scheme Automation Scheme Future Work Future Work

3. 15 Nov 2005 3Jones - IASTED/SEA The Paradigm: Test Driven Specification Test Cases + Assess Test Cases Specification Assess Specification Software Development Software Testing Time

4. 15 Nov 2005 4Jones - IASTED/SEA Background Early Attention to Testing is Good Early Attention to Testing is Good V-Model of Software Testing (planning) V-Model of Software Testing (planning) W-Model of Software Testing (verification) W-Model of Software Testing (verification) Test-Driven Development (coding phase) Test-Driven Development (coding phase) Requirements/Specification remain the hard part of software engineering Requirements/Specification remain the hard part of software engineering Verification options – prototyping, testing, model checking, automated test case generation Verification options – prototyping, testing, model checking, automated test case generation

5. 15 Nov 2005 5Jones - IASTED/SEA Specifics of Our Work Black-box: focus is software function Black-box: focus is software function Specification Language = Decision Table Specification Language = Decision Table Decision table-based test adequacy criteria Decision table-based test adequacy criteria Automated conversion of decision table to analyzer program Automated conversion of decision table to analyzer program Data-driven verification to reveal model anomalies and test data inadequacies Data-driven verification to reveal model anomalies and test data inadequacies

6. 15 Nov 2005 6Jones - IASTED/SEA Decision Tables Logic based, cause-effect specification Logic based, cause-effect specification Within family of state-based specifications Within family of state-based specifications Old technology from 1950s-70s Old technology from 1950s-70s Automated programming systems Automated programming systems Basis for software testing Basis for software testing Specification language for rule based systems Specification language for rule based systems Vanthienen’ Prologa tool, other decision management tools for decision science Vanthienen’ Prologa tool, other decision management tools for decision science

7. 15 Nov 2005 7Jones - IASTED/SEA An Example Specification Calculate employee pay, including overtime paid at 1.5 times the hourly rate of hourly employees for qualifying hours. The normal work week is 40 hours; the maximum number of paid hours is 80. Salaried employees earn over $30 per hour, and are paid overtime only for hours in excess of 60 hours. Salaried employees working less than 40 hours are paid for the normal work week. Hourly employees earn less than $30 per hour, and are paid for all hours worked up to the maximum; they are paid overtime for hours in excess of 40.

8. 15 Nov 2005 8Jones - IASTED/SEA Stimulus-Response View of Specification Calculate Pay hours rate regular_pay overtime_pay

9. 15 Nov 2005 9Jones - IASTED/SEA Decision Table for Specification (Incomplete) ---------------------------- ---------- hours > 40 | N N rate >= 30 | Y N ----------------------------- --------- overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X ---------------------------- ---------- Conditions Decision Rules Actions Action Rules Computation Rule

10. 15 Nov 2005 10Jones - IASTED/SEA Decision Table based Adequacy Criterion Possible adequacy criteria w.r.t. conditions, actions, rules Possible adequacy criteria w.r.t. conditions, actions, rules Functional Coverage Functional Coverage Measures the thoroughness of testing w.r.t. decision table rules Measures the thoroughness of testing w.r.t. decision table rules FC = # computation rules satisfied FC = # computation rules satisfied # computation rules in DT # computation rules in DT

11. 15 Nov 2005 11Jones - IASTED/SEA Automated Checking Specification anomalies Specification anomalies Ambiguous rules - test tuple satisfies multiple rules Ambiguous rules - test tuple satisfies multiple rules Missing rule -- test tuple satisfies no rule Missing rule -- test tuple satisfies no rule Test data set adequacy Test data set adequacy Superfluous test data – rule satisfied by multiple test tuples Superfluous test data – rule satisfied by multiple test tuples Incomplete test data – rule not satisfied by any test tuple Incomplete test data – rule not satisfied by any test tuple Computation rule defines test oracle Computation rule defines test oracle

12. 15 Nov 2005 12Jones - IASTED/SEA Automation Build Analyzer Generate Analyzer Decision Table Run Analyzer Test Data Set Test Coverage Report Spec Coverage Report

13. 15 Nov 2005 13Jones - IASTED/SEA DT Version #1 ---------------------------- ---------- hours > 40 | N N rate >= 30 | Y N ----------------------------- --------- overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X ---------------------------- ---------- Missing Rules

14. 15 Nov 2005 14Jones - IASTED/SEA Analysis Reports - Version #1 TEST SET hours rate 39 9 41 29 40 30 45 40 SPEC COVERAGE REPORT VALID 35 9 315 0 MISSING 41 29 MISSING MISSING VALID 40 30 1200 0 MISSING 45 40 MISSING MISSING TEST COVERAGE REPORT Rules --> Test -- -- Case 1 2 ==== -- -- 1 X 2 3 X 4 RULES NOT SATISFIED: FUNCTIONAL COVERAGE = 2/2

15. 15 Nov 2005 15Jones - IASTED/SEA DT Version #2 ------------ ------------------------------------------ N N Y Y - - | hours > 40 - - - - Y - | hours > 60 - - - - - Y | hours > 80 N Y N Y Y Y | rate >= 30 ------------ ------------------------------------------ X - - - - - | regular_pay = hours * rate; - X X X X X | regular_pay = 40 * rate; X X - - - - | overtime_pay = 0; - - X - - - | overtime_pay = 1.5 * rate * (hours - 40); - - - X X X | overtime_pay = 1.5 * rate * (hours - 60); ------------ ------------------------------------------ TEST SET hours: 35 41 40 45 70 90 rate: 9 29 30 40 40 50

16. 15 Nov 2005 16Jones - IASTED/SEA Analysis Reports - Version #2 TEST COVERAGE REPORT Rules --> Test -- -- -- -- -- -- Case 1 2 3 4 5 6 ==== -- -- -- -- -- -- 1 X 2 X 3 X 4 X 5 X X 6 X X X RULES NOT SATISFIED: FUNCTIONAL_COVERAGE = 6 / 6 SPEC COVERAGE REPORT VALID 35 9 315 0 VALID 41 29 1160 43.5 VALID 40 30 1200 0 VALID 45 40 1600 -900 AMBIGUOUS 70 40 1600 600 AMBIGUOUS 90 50 2000 2250

17. 15 Nov 2005 17Jones - IASTED/SEA DT Version #3 ------------ ------------------------------------------ N N Y - - - | hours > 40 - - - Y Y - | hours > 60 - - - - N Y | hours > 80 N Y N Y Y Y | rate >= 30 ------------ ------------------------------------------ X - - - - - | regular_pay = hours * rate; - X X X X X | regular_pay = 40 * rate; X X - - - - | overtime_pay = 0; - - X - - - | overtime_pay = 1.5 * rate * (hours - 40); - - - X X X | overtime_pay = 1.5 * rate * (hours - 60); ------------ ------------------------------------------ TEST SET hours: 39 41 40 45 70 90 rate: 9 29 30 40 40 50 No similar rule for salaried employee.

18. 15 Nov 2005 18Jones - IASTED/SEA Analysis Reports - Version #3 TEST COVERAGE REPORT Rules --> Test -- -- -- -- -- -- Case 1 2 3 4 5 6 ==== -- -- -- -- -- -- 1 X 2 X 3 X 4 X 5 X X 6 X X RULES NOT SATISFIED: FUNCTIONAL_COVERAGE = 6 / 6 SPEC COVERAGE REPORT VALID 35 9 315 0 VALID 41 29 1160 43.5 VALID 40 30 1200 0 MISSING 45 40 MISSING MISSING AMBIGUOUS 70 40 1600 600 AMBIGUOUS 90 50 2000 2250

19. 15 Nov 2005 19Jones - IASTED/SEA Conclusion Have presented a black-box approach Have presented a black-box approach Test-driven specification as a paradigm for interaction between developer and tester Test-driven specification as a paradigm for interaction between developer and tester Demonstrated benefits of automation Demonstrated benefits of automation Claim that approach is applicable to existing decision based specification methods Claim that approach is applicable to existing decision based specification methods

20. 15 Nov 2005 20Jones - IASTED/SEA Future Work Extend to State-Based Specifications Mill’s State-Box model SCR state tables Pre-/Post-condition specifications Consider random generation of test data used to verify specification Use tool in software engineering courses

21. 15 Nov 2005 21Jones - IASTED/SEA Questions?

22. 15 Nov 2005 22Jones - IASTED/SEA Motivation / References Hoffman & Strooper’s 2001 paper Hoffman & Strooper’s 2001 paper “Prose + Test Cases = Specification” Pamela Zave’s IEEE-TSE 1991 paper, a post- mortem on her 1980’s work on the Paisley operational specifications system Pamela Zave’s IEEE-TSE 1991 paper, a post- mortem on her 1980’s work on the Paisley operational specifications system “An Insider’s View of Paisley” “An Insider’s View of Paisley”