1. Black-box Testing for Evolving COTS-Based Software
Ye Wu
Information and Software Engineering Department
George Mason University

2. Agenda Introduction Challenges Methodology Conclusion IWICSS
© Dr. Ye Wu

3. What is COTS? COTS – Commercial-Off-The-Shelf
COTS describes ready-made products that can easily be obtained.
IWICSS
© Dr. Ye Wu

4. Main characteristics It exists a priori
Used without source code modification
It is available to the general public
It can be bought (or leased or licensed).
Supported and evolved by vendor
COTS vs. Component Based vs. Non-developmental item(ndi).
IWICSS
© Dr. Ye Wu

5. Challenges Facing Evolving COTS-based Software
What should COTS-users do if COTS products are changed?
COTS-user transparent changes
Changes that affect COTS product users, but do not affect user system architecture
Changes that affects COTS user architecture
IWICSS
© Dr. Ye Wu

6. COTS-user transparent changes
Mainly corrective maintenance and a small part of the changes in perfective and adaptive maintenance activities
No change is required for COTS users
IWICSS
© Dr. Ye Wu

7. COTS-user aware changes
Poor design in the previous version
Technology changes
Changes of standards
Sometimes a strategic move to lock in customers
IWICSS
© Dr. Ye Wu

8. COTS-user aware changes
Changes that affect COTS users, but do not affect COTS users’ architecture
Changes that affect COTS users’ architecture
IWICSS
© Dr. Ye Wu

9. Black-box Testing Strategies
Random Testing
Partition Testing
Boundary value Testing
Decision Table Testing
… …
IWICSS
© Dr. Ye Wu

10. Black-box Testing for Evolving COTS-based Software
Information about COTS products to perform adequate black-box testing is not always available
Customization needs to be done on some COTS products
IWICSS
© Dr. Ye Wu

11. COTS Product Modifications
Generalization Modifications
Specialization Modifications
Reconstruction Modifications
IWICSS
© Dr. Ye Wu

12. Black-box Testing Methods for Components
Partition Testing of Generalization Modification:
IWICSS
© Dr. Ye Wu

13. Black-box Testing Methods for Components
Partition Testing for Specialization modification:
IWICSS
© Dr. Ye Wu

14. Black-box Testing Methods for Components
Partition Testing for Reconstruction modification:
IWICSS
© Dr. Ye Wu

15. Initial Results Total 48 partitions – old version
17 changed or added partitions – new version
34 faulty versions (each with one inserted fault)
21 faults identified
13 faults unidentified
5 faults can be detected during Unit-testing
8 faults are caused by boundary value or inaccurate partition
IWICSS
© Dr. Ye Wu

16. Testing Framework for Evolving COTS-based Software
Delivered Version
Delivered
Version
Normal Interfaces
Normal Interfaces
Client
Client
Test Case
Analyzer
Test Case
Analyzer
Testing Interfaces
Testing Interfaces
Test
Specification
Old
Version
Old
Version
Test Case
Repository
Test Case
Repository
Component
Developer
COTS
Developer
new
Version
new
Version
IWICSS
© Dr. Ye Wu

17. Conclusion and Future Work
Adopting evolving COTS products can be risky if without adequate testing
Black-box testing can be adopted, but need to be adequately adjusted
Automated test framework is required
IWICSS
© Dr. Ye Wu