D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Software Testing in a Collaborative B2B Environment Brian McMillan Verizon Communications Brian McMillan Verizon Communications SEAS Alumni Conference 2002

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Agenda What do we mean by B2B applications? Software Development Lifecycle Testing Overview Test Environment Test Planning Test Execution Multi-Business Test Planning & Execution Automation via tools What do we mean by B2B applications? Software Development Lifecycle Testing Overview Test Environment Test Planning Test Execution Multi-Business Test Planning & Execution Automation via tools

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY What do we mean by B2B Applications? A business to business (B2B) application automates a business process across two or more companies Example: Ariba Buyer A packaged software solution that automates, expedites, and controls the purchase of goods and services Allows collaboration with suppliers over the internet End-users view on-line catalogues from their desktop web-browser Order products and services “real time” under contract with authorized suppliers Example: VendorLink Verizon custom written application that uses WebMethods technology to allow engineers from Verizon and Major Equipment Manufacturers to collaborate on switch design.

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Iterative Development In an iterative model, good requirements and testing impact all phases and future iterations in the product lifecycle

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY The Importance of Testing Create process to eliminate defects from the product Identify defects early (1:10:100) Verify product quality continuously

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Availability Results For Previous 24 hrs Prorated Results Month To Date Best Results Which Can Be Achieved For The Month 99.23%99.95%99.97%

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Testing and Quality Verifies the software meets customers needs - Deming Verifies the specific requirements are satisfied - Philip Crosby Verifies achievement of producing a product that meets or exceeds agreed-on requirements –Measured by agreed-on measures and criteria –Produced by an agreed-on process - Rational

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Measurements Key measures of tests include - coverage and quality –Test coverage Measurement of test completeness Coverage of requirements and test cases, or the coverage of executed code –Quality based upon Evaluation of test results Analysis of change requests (defects) identified during testing –Quality is a measure of the system or application-under-test Reliability Stability Performance

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Types of Testing Functional Testing (does it work?) –Unit Testing –System Testing –Integration Testing –Acceptance Testing Performance Testing (does it scale?) –Load Testing –Stress Testing –Configuration Testing –Contention Testing

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Environment Development System and User Acceptance Test Production Standard Test Environment Install package (test deployment plan) Perform installation testing, configuration testing, fail over/fail back, load testing Remove and re-install package (test back out procedures) Multiple project team developing on both desktops and laptops Unit test in the desktop environment Consolidate on a shared server for integration test Execute test plan and test scripts. Fix problems, re-test, log results Build System V package for push into STE environment Execute deployment plan Perform post-installation testing Remove package if testing fails FTP code to UAT Server FTP Package to STE Server FTP Package to Production Server Developer has access to code and servers Developer does not have access to code and servers

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Environment: Ariba Implementation Development System and User Acceptance Test Production Standard Test Environment 4 2-Way HP L-Class UNIX Servers (2 application, 2 database) clustered for high availability 4 8-Way HP N-Class UNIX Servers (2 application, 2 database) clustered for high availability 2 8-Way HP N-Class UNIX Servers (1 application, 1 database) used for performance testing and disaster recovery. Multiple 4-way HP L-Class UNIX Servers Multiple Windows NT Workstations

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Planning Test Strategy/Testing Types Standard User Acceptance Testing Negative Data Testing Regression Testing Free Form Testing Load Testing Stress Testing Volume Testing Failover and Recovery Testing Resources Roles and Responsibilities Workers Testing Approach and Schedule Description of Test System Cycles Success Criteria For User Acceptance Testing User Acceptance Test Logins and Data Deliverables Test Plan Test Cases Mapping of Test Cases to Requirements Test Logs Defect/Issues Log Tools

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Planning: Roles & Responsibilities

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Planning: The Test Calendar

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Planning: Weekly Activities Week One Activities

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Cases and Scripts Create Order for Staples Test Script #1 (partial):

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Traceability Requirement Design Feature Test Case One to Many A Test Case that does not map back to a Requirement is considered incremental to the scope of the project

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Use Cases = Test Cases

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Enter Order Line Item Use Case Basic Flow of Events 1.System prompts user to enter the item ID. 2.Customer enters item ID. 3.The system validates and finds the item ID in the Item Catalog. 4.Systems loads item information from Item Catalog. 5.System determines that the Customer is authorized to order the item. 6.System prompts the Customer to enter the quantity for this line item. 7.User enters the desired quantity. 8.The system validates item quantity according to the criteria in System plays back the information for the current line and prompts the Customer to confirm or cancel the line. 10.Customer confirms the line. 11.System tells the Customer they have entered [num lines] and stores the line item in the current order. Alternative Flows 1.Entered item ID is not in the Catalog 2.Substitute Item Accepted 3.Substitute Item Rejected 4.Attempt to Order Unauthorized Item 5.Item Price Limit Exceeded 6.Invalid Item Quantity Exception Flows 1.Entered item ID is not in the catalog (third time) 2.Entered item's IVRU-Code is invalid (for the third time) 3.User chooses to cancel this line item

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Test Evaluation Summary The Test Evaluation Summary collects, organizes, and presents the Test Results and key measures of test to enable objective quality evaluation and assessment. –Test Metrics (including coverage analysis) –Test Results (test case Id, pass/fail) –Analysis of change requests (scope change, defects) identified during testing –Additional key findings

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Testing Across Multiple Businesses Integration between two companies: Requires additional coordination between the two teams Develop your test and deployment plans jointly Test all transactions end to end Integration between a few companies: Requires significant coordination between multiple teams Integration between many companies: Requires complex coordination between the many teams Best accomplished with a separate dedicated test environment Consider a conference approach over an extended test period

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Automation via testing tools

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Script recording and playback

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Script recording and playback

D E P A R T M E N T O F COMPUTER SCIENCE AND SYSTEMS ANALYSIS SCHOOL OF ENGINEERING & APPLIED SCIENCE O X F O R D O H I O MIAMI UNIVERSITY Questions?