15 Nov 2005 Jones - IASTED/SEA 1 Automated Support for Test-Driven Specification Edward L. Jones IASTED SEA Conference Phoenix, Arizona October 14-16, 2005

15 Nov Jones - IASTED/SEA Outline Test-Driven Specification Paradigm Test-Driven Specification Paradigm Background Background An Example An Example Automation Scheme Automation Scheme Future Work Future Work

15 Nov Jones - IASTED/SEA The Paradigm: Test Driven Specification Test Cases + Assess Test Cases Specification Assess Specification Software Development Software Testing Time

15 Nov Jones - IASTED/SEA Background Early Attention to Testing is Good Early Attention to Testing is Good V-Model of Software Testing (planning) V-Model of Software Testing (planning) W-Model of Software Testing (verification) W-Model of Software Testing (verification) Test-Driven Development (coding phase) Test-Driven Development (coding phase) Requirements/Specification remain the hard part of software engineering Requirements/Specification remain the hard part of software engineering Verification options – prototyping, testing, model checking, automated test case generation Verification options – prototyping, testing, model checking, automated test case generation

15 Nov Jones - IASTED/SEA Specifics of Our Work Black-box: focus is software function Black-box: focus is software function Specification Language = Decision Table Specification Language = Decision Table Decision table-based test adequacy criteria Decision table-based test adequacy criteria Automated conversion of decision table to analyzer program Automated conversion of decision table to analyzer program Data-driven verification to reveal model anomalies and test data inadequacies Data-driven verification to reveal model anomalies and test data inadequacies

15 Nov Jones - IASTED/SEA Decision Tables Logic based, cause-effect specification Logic based, cause-effect specification Within family of state-based specifications Within family of state-based specifications Old technology from 1950s-70s Old technology from 1950s-70s Automated programming systems Automated programming systems Basis for software testing Basis for software testing Specification language for rule based systems Specification language for rule based systems Vanthienen’ Prologa tool, other decision management tools for decision science Vanthienen’ Prologa tool, other decision management tools for decision science

15 Nov Jones - IASTED/SEA An Example Specification Calculate employee pay, including overtime paid at 1.5 times the hourly rate of hourly employees for qualifying hours. The normal work week is 40 hours; the maximum number of paid hours is 80. Salaried employees earn over $30 per hour, and are paid overtime only for hours in excess of 60 hours. Salaried employees working less than 40 hours are paid for the normal work week. Hourly employees earn less than $30 per hour, and are paid for all hours worked up to the maximum; they are paid overtime for hours in excess of 40.

15 Nov Jones - IASTED/SEA Stimulus-Response View of Specification Calculate Pay hours rate regular_pay overtime_pay

15 Nov Jones - IASTED/SEA Decision Table for Specification (Incomplete) hours > 40 | N N rate >= 30 | Y N overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X Conditions Decision Rules Actions Action Rules Computation Rule

15 Nov Jones - IASTED/SEA Decision Table based Adequacy Criterion Possible adequacy criteria w.r.t. conditions, actions, rules Possible adequacy criteria w.r.t. conditions, actions, rules Functional Coverage Functional Coverage Measures the thoroughness of testing w.r.t. decision table rules Measures the thoroughness of testing w.r.t. decision table rules FC = # computation rules satisfied FC = # computation rules satisfied # computation rules in DT # computation rules in DT

15 Nov Jones - IASTED/SEA Automated Checking Specification anomalies Specification anomalies Ambiguous rules - test tuple satisfies multiple rules Ambiguous rules - test tuple satisfies multiple rules Missing rule -- test tuple satisfies no rule Missing rule -- test tuple satisfies no rule Test data set adequacy Test data set adequacy Superfluous test data – rule satisfied by multiple test tuples Superfluous test data – rule satisfied by multiple test tuples Incomplete test data – rule not satisfied by any test tuple Incomplete test data – rule not satisfied by any test tuple Computation rule defines test oracle Computation rule defines test oracle

15 Nov Jones - IASTED/SEA Automation Build Analyzer Generate Analyzer Decision Table Run Analyzer Test Data Set Test Coverage Report Spec Coverage Report

15 Nov Jones - IASTED/SEA DT Version # hours > 40 | N N rate >= 30 | Y N overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X Missing Rules

15 Nov Jones - IASTED/SEA Analysis Reports - Version #1 TEST SET hours rate SPEC COVERAGE REPORT VALID MISSING MISSING MISSING VALID MISSING MISSING MISSING TEST COVERAGE REPORT Rules --> Test Case 1 2 ==== X 2 3 X 4 RULES NOT SATISFIED: FUNCTIONAL COVERAGE = 2/2

15 Nov Jones - IASTED/SEA DT Version # N N Y Y - - | hours > Y - | hours > Y | hours > 80 N Y N Y Y Y | rate >= X | regular_pay = hours * rate; - X X X X X | regular_pay = 40 * rate; X X | overtime_pay = 0; - - X | overtime_pay = 1.5 * rate * (hours - 40); X X X | overtime_pay = 1.5 * rate * (hours - 60); TEST SET hours: rate:

15 Nov Jones - IASTED/SEA Analysis Reports - Version #2 TEST COVERAGE REPORT Rules --> Test Case ==== X 2 X 3 X 4 X 5 X X 6 X X X RULES NOT SATISFIED: FUNCTIONAL_COVERAGE = 6 / 6 SPEC COVERAGE REPORT VALID VALID VALID VALID AMBIGUOUS AMBIGUOUS

15 Nov Jones - IASTED/SEA DT Version # N N Y | hours > Y Y - | hours > N Y | hours > 80 N Y N Y Y Y | rate >= X | regular_pay = hours * rate; - X X X X X | regular_pay = 40 * rate; X X | overtime_pay = 0; - - X | overtime_pay = 1.5 * rate * (hours - 40); X X X | overtime_pay = 1.5 * rate * (hours - 60); TEST SET hours: rate: No similar rule for salaried employee.

15 Nov Jones - IASTED/SEA Analysis Reports - Version #3 TEST COVERAGE REPORT Rules --> Test Case ==== X 2 X 3 X 4 X 5 X X 6 X X RULES NOT SATISFIED: FUNCTIONAL_COVERAGE = 6 / 6 SPEC COVERAGE REPORT VALID VALID VALID MISSING MISSING MISSING AMBIGUOUS AMBIGUOUS

15 Nov Jones - IASTED/SEA Conclusion Have presented a black-box approach Have presented a black-box approach Test-driven specification as a paradigm for interaction between developer and tester Test-driven specification as a paradigm for interaction between developer and tester Demonstrated benefits of automation Demonstrated benefits of automation Claim that approach is applicable to existing decision based specification methods Claim that approach is applicable to existing decision based specification methods

15 Nov Jones - IASTED/SEA Future Work Extend to State-Based Specifications Mill’s State-Box model SCR state tables Pre-/Post-condition specifications Consider random generation of test data used to verify specification Use tool in software engineering courses

15 Nov Jones - IASTED/SEA Questions?

15 Nov Jones - IASTED/SEA Motivation / References Hoffman & Strooper’s 2001 paper Hoffman & Strooper’s 2001 paper “Prose + Test Cases = Specification” Pamela Zave’s IEEE-TSE 1991 paper, a post- mortem on her 1980’s work on the Paisley operational specifications system Pamela Zave’s IEEE-TSE 1991 paper, a post- mortem on her 1980’s work on the Paisley operational specifications system “An Insider’s View of Paisley” “An Insider’s View of Paisley”