1. Bridging the Gap between logical requirements-based Test Cases and physical Test Data Generation Presentation for Swiss Testing Day by Harry M. Sneed, ANECON, Vienna The unresolved problem of system testing Requirement-based testing From Requirements to logical Test Cases From logical Test Cases to a Test Design From logical Test Cases to Test Data From logical Test Cases to Test Results Generating Test Data Validating Test Results Can the Gap between the test concept and test execution be bridged???

2. The Unresolved Problem of System Testing At the top of system testing there are logical test cases formulated in natural language based on the requirement specifications. At the bottom of system testing there are physical test data objects such as HTML pages, XML interfaces and SQL databases. The problem in system testing is how to go from the logical test case level to the physical test data, i.e. to bridge the gap between test case specification and test data generation. SWISS-1

3. Logical Test Cases Design Document Data Analysis Data Schema If article_stock<minimum_stock { create supply_order where supply_id = asupply_id + 1; supply_article = article_id; supply_name = article.name; supply_amount = supply.stock * 2; } Requirement Document Requirement Analysis Test Data Miracle Objects & Attributes The Miracle of System Testing SWISS-2 Test if when amount on stock exceeds minimum stock level a supply order is produced

4. Object Assignment Test Data Procedures Extracts Logical Test Cases From Requirements to Test Data SWISS-3 Test Data Generator Logical Test Cases Text Analysis Database Schemata Interface Schemata GUI Definitions Value Assignment Value Tables Existing Data Requirement Document DB Tables Interface Files GUI Inputs SQLXML/WSDLHTML / XSL Human Transactions With User Objects & Preconditions Pre Asserts SQLXMLHTML Input Objects Script English/GermanHuman Interaction 1 23 4 5 Tester

5. Steps to Test Data Generation SWISS-4 The TestAnalyzer extracts logical test cases from the requirement text - one for the object action, object state and object condition. Tester refines the test cases by adding pre conditions to each test case. The precondition refers to the objects used by the system as inputs. Tool displays all logical objects from the test cases and all physical input objects from the data model. The tester assigns logical objects to physical objects. Tool generates an assertion procedure for each physical object. Tool displays attributes of all physical objects. The tester assigns them value domains, specific values, relations or expressions. The tool generates tables of test data. Tool joins generated test values with existing test data to generate system inputs - GUI contents, data base tables and messages. 1 2 3 4 5

6. Object Assignment Test Data Procedures Extracts Logical Test Cases From Requirements to Test Results SWISS-5 Test Result Validator Logical Test Cases Text Analysis Database Schemata Interface Schemata GUI Definitions Value Assignment Value Tables Existing Data Requirement Document DB Tables Interface Files GUI Inputs SQLXML/WSDLHTML / XSL Human Transactions Logic Test Cases with User Objects & Post Conditions Post Asserts SQLXML/WSDLHTML Script English/GermanHuman Interaction 1 23 4 5 Tester Data Model

7. Steps to Test Result Validation SWISS-6 The TestAnalyzer extracts logical test cases from the requirement text - one for the object action, object state and object condition. Tester refines the test cases by adding post conditions to each test case. The precondition refers to the objects used by the system as outputs. Tool displays all logical objects from the test cases and all physical output objects from the data model. The tester assigns logical objects to physical objects. Tool generates an assertion procedure for each physical object. Tool displays attributes of all physical objects. The tester assigns them value domains, specific values, relations or expressions. The tool generates tables of expected values. Tool compares system outputs with previous outputs and with the expected values. 1 2 3 4 5

8. Refinement of Test Cases SWISS-7 Test CaseTest PurposeTest ObjectsPre/Post Conditions OrderEntry01Order should be rejected if customer is not known Order Customer Rejected (post) Unknown(pre) OrderEntry02Order should be rejected if customer credit rating is too low Order Customer Credit Rating Rejected (post) Known (pre) Too low (pre) OrderEntry03Order should be rejected if article ordered is not on stock. Order Article Rejected (post) Not exists (pre) OrderEntry04Order should be backlogged, if amount on stock is less than order amount. Order Amount on Stock Order Amount Backlogged (post) < order_amount (pre) => amount_on_stock (pre) OrderEntry05Order should be fulfilled if amount on stock is adequate. Order Amount on Stock Fulfiled (post) > order_amount (pre) OrderEntry06Amount on stock should be reduced by amount of order. Input on Stock Amount of Order - order_amount (post) < amount_on_stock (pre) OrderEntry07When amount on stock is less than minimum on stock level, a supply order is produced. Amount on Stock Minimum Stock Supply Order < minimum_stock (pre) > amount_on_stock (pre) Exists (post) 2

9. Assigning Logical Objects to Physical Objects SWISS-8 Test CaseTest PurposeTest ObjectsUsage OrderEntry09When amount stock is less than minimum stock level, a supply order is produced Amount on stock Minimum stock level Supply order Input Output OutputInput 3

10. Assigning Value Domains to Physical Data Fields SWISS-9 4 Article Article_id Article_name Article_stock Minimum_stock Supply_stock Supply_id Supply_article_id Supply_name Supply:amount Integer String Integer String Integer Supply_Order Article_Ids Begin = 0 Internal = +10 Article_Names Book Magazine CD_Disk Article_Stock Minimum = 50 Maximum = 9999 Minimum_Stock Constant = 50

11. [RF 41] Notification can be divided into action and non-action (=information) notification. [RF 42] In case of an “action needed” notification, the system provides a link to the place where the action is needed. [RF 43] Notifications are added to a notification interface, which contains information about The reason of the notification The kind of the notification The type of the notification The date/time of the notification If action needed: A link to the place where the action is needed. [RF 44] The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. If the action is finished, the user name is stored by the notification. [RF 45] If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF 43]) so that each user can see the notification. The notification is only added to the notification table if the data is unequal to the current event data (see also [RF 30]). Original Requirements Text SWISS-10

12. Function your_name defined RF_43 Notifications are added to a notification interface, which contains information about * The reason of the notification * The kind of the notification * The type of the notification * The date/time of the notification * If action is needed: A link to the place where the action is needed. Function your_name defined RF_45 If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF_43]) so that each user The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]). Provider B has an other starting time than provider A. The user chose Provider A as primary source, so the event time is the time of Provider A. If Provider B updates his time to the time of the event, no notification is added. If Provider B sends a time different to the event time, a notification is send. Requirements extracted from the Requirement Text SWISS-11

13. TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0163 Require RF_43 perform action Notifications are added to a notification interface, which contains information about: The reason of the notification The kind of the notification The type of the notification The date/time of the notification DFI0164 Require RF_43 checks rule If action is needed a link to the place where the action is needed. -------------------------------------------------------------------------------------------------------------------------------------------------------- TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0165 Require RF_44 checks rule If the action is finished, the user name is stored in the notification. -------------------------------------------------------------------------------------------------------------------------------------------------------- TestCase-Id BaseType Requirement Relation TestType TestCase Description -------------------------------------------------------------------------------------------------------------------------------------------------------- DFI0166 Require RF_45 checks rule If at least one provider changed his feed data changing starting time, a notification is added to the notification table (see [RF_43]) so that each user can see the notification. DFI0167 Require RF_45 checks rule The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]). DFI0168 Require RF_45 checks rule The user choose Provider A as primary source, so the event time is the time of Provider A. DFI0169 Require RF_45 checks rule If Provider B updates his time to the time of the event, no notification is added. DFI0170 Require RF_45 checks rule If Provider B sends a time different to the event time, a notification is sent. Test Cases extracted from the Requirement Text SWISS-12

14. TestCase Id = "DFI0167" > Require RF_45 High Analyst rule check:The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]). MS-Windows notification notification_table event DFI0166 Notification is received the data is unequal to the current event data(see also [RF_30]). notification is only added to the notification table Test Cases in XML Format with Objects SWISS-13

15. Oracle Data Tables /*==============================================================*/ /* Table: BASEEVENT */ create table DFI.BASEEVENT ( ID NUMBER(18) not null, EVENTNAME varchar(250), STARTDATETIME date not null, CANCELLED date, UPDATEABLE int not null, EVENTDATE date not null, PROGRAM varchar(100), ERRORCODE number, ERRORTEXT varchar(2000)) /*==============================================================*/ /* Table: FEEDEVENT */ create table DFI.FEEDEVENT ( ID NUMBER(18) not null, BASEEVENT_ID NUMBER(18), FEEDEVENTID varchar(100), EVENTNAME varchar(250), STARTDATETIME date, CANCELLED date) /*==============================================================*/ /* Table: NOTIFICATION */ create table DFI.NOTIFICATION ( ID NUMBER(18) not null, EVENTNAME varchar(30), EVENTDATE date not null, EVENTSPONSOR CHAR(10)) Physical Objects defined in SQL Database Schema SWISS-14

16. Physical Objects defined in XML Input Data Schema SWISS-15

17. BASEEVENT ID EVENTNAME START DATETIME CANCELLED UPDATEABLE EVENTTDATE PROGRAM ERRORCODE ERRORTEXT number(18) not null, varchar(250), date not null, int not null) date not null, varchar(100), number, varchar(2000 NOTIFICATION ID EVENTNAME EVENTDATE EVENTSPONSOR number(18) not null, varchar(30), date not null, char(10)) Competition Id Sponsor Name StartDate EndDate Idtype string date Date Round Number Name StartDate EndDate Legs Replayspossible Idtype string date Idtype boolean EventBASEEVENT Notification Table NOTIFICATION NotificationCompetition Round Assignment of Logical Objects to Physical Objects SWISS-16

18. file: DFI_Test_02; if ( object = "BASEEVENT" ); assert new.ID = “4711"; assert new.EVENTNAME = “XXXXXXXXXXXXXXXXX"; assert new.STARTDATETIME = “2006-10-07"; assert new.CANCELLED = "2006-10-05"; assert new.UPDATEABLE = "99999999999999"; assert new.EVENTDATE = “2006-10-07"; assert new.PROGRAM = “YYYYYYYYYYYYYYYYYYY"; assert new.ERRORCODE = “77"; assert new.ERRORTEXT = “ZZZZZZZZZZZZZZZZZZ“; endObject; if ( $object = „ComFEEDEVENT" ); assert new.ID = “4712"; assert new.BASEEVENT_ID = BASEEVENT.ID; assert new.FEEDEVENTID = FEEDEVENT.ID; assert new.EVENTNAME = BASEEVENT.EVENTNAME; assert new.STARTDATETIME = “2006-10-07"; assert new.CANCELLED = "2006-10-05"; endObject; if ( $object = “NOTIFICATION" ); assert new.ID = “4713"; assert new.EVENTNAME = “XXXXXXXXXXXXXXXXX“; assert new.EVENTDATE = “2006-10-15"; assert new.EVENTSPONSOR = “YYYYYYYYYYYYYYYYYYYY"; endObject; End DFIDB; SWISS-17 Initial Assignment of Values to Database Attributes

19. file: DFIDB; if ( $Test = " DFI0166 " ); if ( $object = "BASEEVENT" ); assert pre.ID = “4711"; assert pre.EVENTNAME = “Manchester/Liverpool Match"; assert pre.STARTDATE = “2006-10-07"; assert pre.CANCELLED = "2006-10-05"; assert pre.UPDATEABLE = „6856"; assert pre.EVENTDATE = “2006-10-07"; assert pre.PROGRAM = “English football league"; assert pre.ERRORCODE = “77"; assert pre.ERRORTEXT = “Match result not verified“; endObject; if ( $object = „Competition" ); assert pre.ID = BASEEVENT.ID; assert pre.Sponsor = “McDonalds“; assert pre.name != BASEEVENT.EVENTNAME; assert pre.StartDate != BASEEVENT.STARTDATE; assert pre.EndDate = “2006-10-07"; endObject; if ($object = “NOTIFICATION" ); assert post.ID = Competition.ID; assert post.EVENTNAME = Competition.name; assert post.EVENTDATE = Competition.StartDate; assert post.EVENTSPONSOR = Competition.Sponsor; endObject; endTest; SWISS-18 Specific TestCase related Assignment of Values

20. To test the Automatic Data Feed Function To test if the XML files from the Providers are processed correctly To test if the DFI Database is updated correctly To test if the correct Notifications are sent Sun-Sparc Server DFI Database must be initialized with Testdata One or more XML Files must be generated DFI System must be put into operation XML Files must be copied into the system queue RF-01-Existing_Functionality_is_covered RF-45-Notification_is_generated_if_at_least_one_Provider_changes_his_feed_data RF-46-User_can_restrict_data_by_multiple_Filters UC-01-Import_Data_Feed UC-01-01-Propagate_new_Feed_Event BaseEvent FeedEvent Notification Rule TeamLeague Feed Files have been parsed and separated Feed Events have been added Feed Events have been updated Event Notifications have been stored Update Notifications have been stored UpdateEvent Notification has been sent SWISS-19 Assignment of Use Cases to a Test Process

21. Generate an XML feed file Load the XML feed file System parses the XML Feed file System stores good Parse results for processing System stores bad Parse results as error file System creates new event System adds Feed Event reference System sets Base Event properties System checks active Rule System checks if active Rule is fulfilled System sends Create Notification to Bookie System updates base event attributes System sends non-action Notification to Bookie SWISS-20 Assignment of Use Case Steps to Process Steps

22. DFI0166 If at least one provider changed his feed data e.g. changing starting time, a notification is added to a notification interface RF-45 UC-01-01 DFI0167 The notification is only added to the notification table if the data is unequal to the current event data RF-45 UC-01-01 DFI0168 The user choose Provider A as primary source, so the event time is the time of Provider A RF-45 UC-01-02 All Feed File variations have been processed All invalid Feed data has been recognized All target database tables have been updated All database attribute values correspond to expected values All feed data has been matched All unmatched data has lead to a notification All update events have been sent to V5 All notification data values correspond to expected values All notifications have been acknowledged by the bookie SWISS-21 Assignment of Logical Test Cases to a Test Process

23. Test Process Data Flow Data Feed UC-01 UC-16 UC-17 Update Event Validiere mit WSDLVal XML Interface WSDL Interface Feed File Generiere mit XMLGen XML Rule BaseEventDeliveredNotification Team League FeedEvent Delivered FeedEvent Notification Type Validiere mit CSVVal Generiere mit CSVGen SWISS-22

24. Test Result Validation against the Post Conditions SWISS-23

25. The Solution to System Testing The solution to the system testing problem lies in the association of objects and events at the logical level to objects and events at the physical level. The logical objects referred to in the system requirements must be linked to the physical objects of the system architecture – the GUIs, interfaces and database tables. The logical events referred to in the system requirements, e.g. use cases, must be linked to the physical events of the system – the online transactions and batch processes. SWISS-24