Black-box Testing for Evolving COTS-Based Software Ye Wu Information and Software Engineering Department George Mason University

Agenda Introduction Challenges Methodology Conclusion IWICSS © Dr. Ye Wu

What is COTS? COTS – Commercial-Off-The-Shelf COTS describes ready-made products that can easily be obtained. IWICSS © Dr. Ye Wu

Main characteristics It exists a priori Used without source code modification It is available to the general public It can be bought (or leased or licensed). Supported and evolved by vendor COTS vs. Component Based vs. Non-developmental item(ndi). IWICSS © Dr. Ye Wu

Challenges Facing Evolving COTS-based Software What should COTS-users do if COTS products are changed? COTS-user transparent changes Changes that affect COTS product users, but do not affect user system architecture Changes that affects COTS user architecture IWICSS © Dr. Ye Wu

COTS-user transparent changes Mainly corrective maintenance and a small part of the changes in perfective and adaptive maintenance activities No change is required for COTS users IWICSS © Dr. Ye Wu

COTS-user aware changes Poor design in the previous version Technology changes Changes of standards Sometimes a strategic move to lock in customers IWICSS © Dr. Ye Wu

COTS-user aware changes Changes that affect COTS users, but do not affect COTS users’ architecture Changes that affect COTS users’ architecture IWICSS © Dr. Ye Wu

Black-box Testing Strategies Random Testing Partition Testing Boundary value Testing Decision Table Testing … … IWICSS © Dr. Ye Wu

Black-box Testing for Evolving COTS-based Software Information about COTS products to perform adequate black-box testing is not always available Customization needs to be done on some COTS products IWICSS © Dr. Ye Wu

COTS Product Modifications Generalization Modifications Specialization Modifications Reconstruction Modifications IWICSS © Dr. Ye Wu

Black-box Testing Methods for Components Partition Testing of Generalization Modification: IWICSS © Dr. Ye Wu

Black-box Testing Methods for Components Partition Testing for Specialization modification: IWICSS © Dr. Ye Wu

Black-box Testing Methods for Components Partition Testing for Reconstruction modification: IWICSS © Dr. Ye Wu

Initial Results Total 48 partitions – old version 17 changed or added partitions – new version 34 faulty versions (each with one inserted fault) 21 faults identified 13 faults unidentified 5 faults can be detected during Unit-testing 8 faults are caused by boundary value or inaccurate partition IWICSS © Dr. Ye Wu

Testing Framework for Evolving COTS-based Software Delivered Version Delivered Version Normal Interfaces Normal Interfaces Client Client Test Case Analyzer Test Case Analyzer Testing Interfaces Testing Interfaces Test Specification Old Version Old Version Test Case Repository Test Case Repository Component Developer COTS Developer new Version new Version IWICSS © Dr. Ye Wu

Conclusion and Future Work Adopting evolving COTS products can be risky if without adequate testing Black-box testing can be adopted, but need to be adequately adjusted Automated test framework is required IWICSS © Dr. Ye Wu