1. Requirement-based Testing A Presentation by Harry Sneed for the John von Neumann Society Budapest, 27 May 2009

2. The many Dimensions of Software Testing Test Objects Units (Modules/Classes) Components Subsystems Systems Test Types Functional Testing Performance Testing Load Testing Usability Testing Test Approaches Random Testing Creative Testing Systematic Testing Test Bases Code-based testing Interface-based testing Reference-based testing Model-based testing Require.-based testing

3. Software Units Components Systems Functional Specification (Oracle) In 1978 Bill Howden introduced the notion of a test oracle in connection with Functional Testing. It means testing the System against the Requirement Specification (functional & non-functional Requirements) A Test is always a Test against Something Oracle

4. Code Test inputs Test results & Actual paths Specification Expected results Expected paths Potential Paths Specified Paths compare Results & Paths Corresponds to a Unit Test (White-Box) Code-based Testing = Testing the Program against itself

5. CodeTest inputsTest results SpecificationExpected results compare Data Representative Values Interfaces/Files/Tables Specified Results Interfaces/Databases Corresponds to an Integration Test (Grey-Box) Data-based Testing = Testing the Program against the Data Interfaces/Files/Tables

6. CodeTest inputsTest results Specification Expected results compare Files/Tables/Panels Specified Results Corresponds to a System Test (Black-Box) Results Files/Tables/Panels Specified Inputs Requirement-based Testing = Testing the Program against the Specs

7. Static Analysis Execution Test Verf. Cond. Generator Syntax Analysis SyntaxErrorsSemantic ErrorsExec Errors Struct Errors Graph Checking + Call Checking + Units Consistency + Mode Checking + Asserted/Actual Use + Set/Use Checking Test Results + Execution Errors + Coverage Report + Assertion Exceptions Annotated Listing + VC'S + Symbolic Execution of Expressions + Simplified VC'S Indented Program Listing + Diagnostics Correct Source Program with Assert "Correct" Source Prog. Assertions taken from Specs(RSL) The Mother of all Test Tools = RXVP Requirement Evaluation & Verification Package FORTRAN Dr. Edward Miller from General Research Corporation The idea was to put dirty software into a washing maschine and get out cleaned software.

8. P 1 P 2 P 3 P 4 Entry Exit Test Object = Procedure or Module Test Case = Path thru Test Object 1. Path 2. Path 3. Path 5. Path Test Cases were extracted from the Code 4. Path Preconditions Postconditions Before State After State

9. The Budapester Testlabor The Budapester Test Laboratory was set up by Harry Sneed and Dr. Ed Miller from Software Research Inc in cooperation with the SZKI and Szamok in May, 1978. This was the first known attempt to outsource the testing activity. The customer was Siemens, Munich, which was developing a large scale, distributed software system for the German Railroad – the Integrated Transport System (ITS).

10. Test Process in the Test Labor Compilable Modules were delivered by Siemens every week on a magnetic tape. Sources were statically analyzed to produce flow and structure diagrams. Paths were marked thru the flow diagrams with magic markers. Conditional variables were assigned values to confirm or negate every decision. Values were assigned at test time to the test driver before each test case execution. Results were logged to be checked later against the expected results.

11. Experience with the Test Labor In the first half year 128 modules with 60881 SPL statements passed through the test labor. 4378 code branches were tested with a coverage of 89.7 % by 1544 test cases. 192 program errors were discovered via dynamic analysis. 1396 design and code deficiencies were discovered via static analysis. The costs of the quality assurance remained below 4 DM pro statement. This was less than 10% of the development costs.

12. Summary of first 6 Months CompModulesStmtsTest cases BranchesCoverageDesign & Code Defis Program Errors A640291385 B37758823260391%13022 K71407351064284387.5%868143 N6284710114094%11014 S8568214779295%1506 Total128608811544437891.9%1396190

13. Resume on Module Testing In one year 281 Modules with 134 K stmts were tested with 3864 test cases at a cost of DM 75,- per test case = DM 289.800 The minimum branch coverage was 85% 403 Errors were reported at a cost of DM 150,- per error = DM 60.450 Later in the system test by the customer, another 386 errors were uncovered. The intensive module test had uncovered only 51% of the total errors found. This demonstrates the fact that module testing of the code against itself is not sufficient. The software as a whole must be tested against the requirements.

14. Store Measure R-Net Receive Signals ASSERT Report Engine ok ASSERT Check Measure Report Signal Error ASSERT send Alarm ASSERT Invalid Measure Valid Measure Validation Points MonitoringEngine Safe Range Out of safe Range RXVP progressed to test the systems against Requirements Nets

15. Receive Signal Measuring device & Store Measure Message Log Check Measure Report ok Alarm Nurse on Duty Alarm Display Report Measure Error System Display 3. test case 2. test case 1. test case R-Net-Entry invalid Measure Valid Measure and or or Engine problemEngine ok Engine Monitoring System Requirement Specification with RSL Extracting Test Cases from Requirement Nets

16. Action Description IfAccount_Balance minus the withdrawal is more than the credit limit allow the withdrawal In case the withdrawal causes the Account_Balance to fall below the credit limit refuse the withdrawal. Account_Balance minus the withdrawal is more than the credit limit Withdrawal causes the Account_Balance to fall below the credit limit Allow the withdrawal Refuse the withdrawal Test Case1234 YYNN YNYN -X-- X-X- Service FunctionTest Case Generation Conditional keywords are scanned. Each Condition is treated separately and combined with the other conditions in the same action to give the number of testcases = nr_conditions 2 Extracting Test Cases from Business Rules

17. One Use Case can have one or more Test case Normally, there are at least 2 Test Cases Positive outcome Negative outcome Cash Withdrawal Not enough cash Account overdrawn Cash delivered Wrong PIN Number Invalid Card Use Case = Test Cases = Use Case Paths Extracting Test Cases from Use Case Descriptions

18. Test Cases Functional Non-Functional Objects Actions States Conditions Requirements Text Extracting Test Cases from Requirement Documents

19. Actions upon objects: The customer account should be deleted. The invoice should be sent within 30 days. Object states: The customer account is overdrawn. The customer is an employee. Rules: If the account is overdrawn a notice should be sent. Whenever someone fails to pay within 30 days, a reminder should be sent. Customer accounts should be deleted, if they are overdrawn more than 3 months. In case of a system crash, the recovery routine should be started. Grammar types indicating Declarations to be tested

20. [RF 41] Notification can be divided into action and non-action (=information) notification. [RF 42] In case of an action needed notification, the system provides a link to the place where the action is needed. [RF 43] Notifications are added to a notification interface, which contains information about The reason of the notification The kind of the notification The type of the notification The date/time of the notification If action needed: A link to the place where the action is needed. [RF 44] The user is able take a notification (locked for others), do the action and mark the notification as finished. If the action is finished, the user name is stored by the notification. [RF 45] If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF 43]) so that each user can see the notification. The notification is only added to the notification table if the data is unequal to the current event data (see also [RF 30]). Sample Requirements Text

21. TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0163 Require RF_43 perform action Notifications are added to a notification interface, which contains information about: The reason of the notification The kind of the notification The type of the notification The date/time of the notification DFI0164 Require RF_43 checks rule If action is needed a link to the place where the action is needed. -------------------------------------------------------------------------------------------------------------------------------------------------------- TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0165 Require RF_44 perform action The user is able take a notification (locked for others), do the action and mark the notification as finished. DFI0166 Require RF_44 checks rule If the action is finished, the user name is stored in the notification. -------------------------------------------------------------------------------------------------------------------------------------------------------- TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0167 Require RF_45 checks rule If at least one provider changed his feed data changing starting time, a notification is added to the notification table (see [RF_43]) so that each user can see the notification. DFI0168 Require RF_45 checks rule The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]). DFI0169 Require RF_45 checks rule The user choose Provider A as primary source, so the event time is the time of Provider A. DFI0170 Require RF_45 checks rule If Provider B updates his time to the time of the event, no notification is added. DFI0171 Require RF_45 checks rule If Provider B sends a time different to the event time, a notification is sent. Test Cases extracted from the Requirement Text

22. The order entry system must be able to handle at least 1000 customer web pages at one time. A customer order must be processed within 2 seconds. The article database may contain up to 10.000 articles. The system should be able to produce as many as 25.000 bills in one billing run. A billing run should not take more than 2 hours. If the system crashes it should be back up and in operation within 15 minutes. Database backups should take place every 4 hours. Customers should be checked both by number and by password. Passwords should be rejected after 3 tries, customer number should be rejected after 2 tries. Sample Non Functional Requirements

23. 1.999 customer orders at one time 2.>1000 customer orders at one time 3.response time per order < 2 sec. 4.response time per order >2 sec. 5.9.999 article records 6.> 10.000 article records 7.24.999 bills in one billing 8.>25.000 bills in one billing 9.run time of billing < 2 hours 10.run time of billing >2 hours 11.If system crash, recovery < 15 minutes 12.If > 4 hours, database secured 13.Submit 3 passwords where 3rd password is correct 14.Submit 3 passwords where all 3 are incorrect 15.Submit 2 customer number where 2nd number is correct 16.Submit 2 customer number where both are incorrect Typical Non-Functional Test Cases

24. A Test Case: has a unique identifier serves a purpose refers to a Testobject refers to a requirement or use case has a precondition state has a postcondition state has a set of required input parameters has a set of expected results has a predecessor test case has a given environment has an owner has a status has a date of execution Properties of a Test Case An international standard is required for the minimum Properties of test cases!!!

25. TestCase ID: IC-Order-7 Test Case – Purpose: to test a fulfilled order Test-Object: Customer – Order Precondition: Ordered Article is on stock. Article. Amount is > Order.Position.Amount (Article. Amount - Order. Position.Amount ) > minimum – Amount Postcondition:Article.Amount = a Pre.Article.Amount – Order.Position.Amount No supply order Inputs: Order.Cust-Nr = 200100 Customer.Credit = A Order.Position-Nr = 2 Order.Position.Art – Nr = 4711 Order.Position.Amount = 3 Article.Amount = 10 Minimum – Amount = 7 Results: Text-Msg= Article order fulfilled Environment: MS – Internet, Windows – XP Usecase tested: Customer.Order.Processing Status: tested Date: 29. 8. 2005. Functional Test Case Specification

26. Text Analysis Test Cases Repository Loader Test Case DB Repository Unloader Test Scripts Test Tester enhances & refines Test Cases Tester executes Test Cases Text Analyzer generates Test Cases Test Script Generator generates Test Scripts for Test Automation From Requirements via Test Cases to Test Scripts

27. Text Processor TxtAnal Text Pre Processor TxtScan Object List Object Model Test Cases Rule Violations Spec Metrics Repository Entries Objects - Use Cases - Actions - States - Rules Text Excel XML Format Rules Rupp Rules Size Quantity Complexity Sections - Use Cases - Objects - Actions - States - Rules TextAnal = Tool for extracting Test Cases from English and German Language Documents

28. Requirements Analysis Demo Bet&Win Requirement Document Setting Key Words Scanning for Objects Analyzing the Document Extracting Test Cases Collecting Requirement Metrics Documenting Cross References Generating a test design

29. Egovernment Web Site with three applications and circa 33 K Java Stmts 1844 Test cases were specified of which 1272 were generated 1495 Test cases were executed = 81% Test Case coverage 352 Test cases discovered at least one error = 24% Hit rate 450 Errors were reported by testers = 13 per KDSI 56 Errors were reported by users after release = 11% of all errors System test discovered 89% of all reported errors = test efficiency System test lasted 4 months from January to May 2005 392 Test days were booked against the project = 3.8 Test cases per Testertag = 1.1 Errors reported per tester day Customer was satisfied with the test result. Experience with Requirement-based Testing of the Public Internet Portal for the state of Saxony

30. Conclusions Test methods have not really improved since the 1970s. We are still using the basic approaches introduced back then. Test tools have however improved significantly. Today there are more and better testing tools. The future lies in test automation. Only through automation can we really reduce test costs and increase the test ROI. Test ROI = (TestBenefits – TestCosts) / TestCosts