Acceptance Testing Strategy Guided by Evolutionary Operational Profiles Authors: Maria de Fatima Mattiello-Francisco fatima@iss.inpe.br Ana Maria Ambrosio National Institute for Space Research Edgar Toshiro Yano Aeronautical Institute of Technology Industry Track ISSRE2007

System’s stakeholders satisfaction Expected testing quality Motivation Acceptance Process Service Profiles Test Strategy Results The target is the Payload Embedded Software on board X-Ray Astronomy Satellite System’s stakeholders satisfaction Expected testing quality Confidence on critical software operation takes into account: Stakeholders involved in Development & Operation Processes Principal Investigator Quality is related to Test Strategies which includes Test Techniques Environmental facilities Testing time duration

Acceptance Process Motivation Service Profiles Test Strategy Results Mission Integration Stages Instrument ECSS- European Cooperation for Space Standardization Subsystem System CDR – Critical Design Review QR – Qualification Review AR – Acceptance Review

Acceptance Process Acceptance Process Motivation Service Profiles Test Strategy Results Mission Integration Stages The target software is integrated with other mission's components in the three mission stages Instrument Subsystem System (Payload) (Satellite Platform) (Space & Ground Segment) During each stage the payload software will be an encapsulated component being validated under the different tester’s point of view Acceptance Process A formal testing to determine whether or not a systems satisfies acceptance criteria based on the software problems reports resulting from verification and validation activities [IEEE-Std1059-1993] In our approach the acceptance criteria is based on the number of faults revealed by the services execution during each integration stages.

Acceptance Process U1 U2 U3 U4 U2 U3 U4 U5 U2 U3 U4 U5 U6 U7 Motivation Service Profiles Test Strategy Results Testers involved in the Mission Integration Stages Instrument Subsystem System Payload Satellite Platform Space & Ground Segment U1 U2 U3 U4 U2 U3 U4 U5 U2 U3 U4 U5 U6 U7 U1- Payload Software Engineer (focus on software requirement validation) U2 – Payload Engineer (focus on interfaces performance) U3 – Principal Investigator (Scientist focus on operational modes) U4 – Software Engineer (Regression Test ) U5 – On Board Data Handling Computer Engineer (focus on communication) U6 – Mission System Engineer U7 – Ground Segment Engineer (Mission Center Operation) Testers with different needs carry on the mission integration stages

Service Profiles Operational Profiles Services Motivation Acceptance Process Test Strategy Results Operational Profiles External user-oriented test approach – specifies the intended usage of the system in terms of operations and their occurrence probabilities [MUSA93] Services Space system operations are service-oriented. A set of on-board performing functions is always related to one service purpose. On-board embedded software is a component that collaborates on service provider by executing services activities, being destination of service requests and the source of service reports [ECSS-E-70-2003]. Operation problems can be anticipated by TESTERS performing service-oriented test cases during the acceptance testing

The proposed Evolutionary Operational Profile Service Profiles Motivation Acceptance Process Test Strategy Results The proposed Evolutionary Operational Profile Aggregates to Musa’s approach the service operational profiles, adding the concept of service and the usage of system architectural elements Such new approach aims at supporting a test strategy for embedded space software acceptance along the evolutionary integration stages Denotes the growth in complexity and autonomy of the environment with which the embedded software interacts as far the integration stages evolves. At each integration stage the environmental facilities are incrementally substituted by the real subsystems. Software functionality does not change but service operation does due to the test scripts evolution

Identify software operational modes profile Service Profiles Motivation Acceptance Process Test Strategy Results SERVICE PROFILES Is a concept which depends on software behavior in terms of operational mode and software architecture Services are performed within system modes by component and their interfaces. They are conceived taking into account the following steps Identify software operational modes profile Identify software architectural profile List the SERVICES to be performed Establish relationship between element and services concerning USAGE INTENSITY Calculate the service operational profiles

Service Profiles Mode profile Motivation Acceptance Process Test Strategy Results Mode profile Is defined as the probability of occurrence of each operational mode in the system use. A service might be provided by more than one software operational mode and its occurrence probabilities being different in each mode. Software operations and services are performed within operational modes and by architectural elements. Case Study: MIRAX Payload Software - SWPDC operational modes

Architectural Element profile Service Profiles Motivation Acceptance Process Test Strategy Results Architectural Element profile Case Study: MIRAX Payload Software - SWPDC Architecture

Architectural Element profile Service Profiles Motivation Acceptance Process Test Strategy Results Architectural Element profile Given a category of architectural element, one can attribute probabilities of usage of e in system operation lifetime as a whole. i The occurrence probability estimation depends on high knowledge of the application and historical data from previous experience in similar projects SERVICE USAGE MATRIX The binary cell a indicates the relative use of the architectural element by each service ij

Service Profiles Calculation Motivation Acceptance Process Test Strategy Results Service Profiles Calculation Service profile is defined as the occurrence probability of each service in terms of using a particular element. Where weight w represents the degree of usage intensity and/or criticality of the element e on such service q ij i j MODE USAGE INTENSITY MATRIX

Test Strategy Test Strategy Motivation Acceptance Process Service Profiles Results Test Strategy Based on SERVICES PROFILES PSe, one selects the number of TEST CASES Ntc provided by U testers per service in order to match the previous total time T allocated for each mission integration stage Ek There is a set of services Si that exercise better some elements Ek than other. Therefore, priority on performing particular test cases is recommended whenever reduction on testing duration are imposed to the testers Uj Uj The test strategy effectiveness will be demonstrated in our case study with data from the mission integration stage at Instrument level provided by the testers. All test cases associated to the mentioned services were executed in order to evaluate this approach. f ( S4,U2,E1) = ??? Si

Test Strategy Motivation Acceptance Process Service Profiles Results Instrument Ntc provided by tester per service U1 U2 U3 U4 A 101 636 46 88 = 870 Considering that the average time of 2 test cases execution (including analyses) is 1 hour for group A, this stage will require 435 hours to be concluded. If the available time is 220 hours (50%) only, our test strategy approach reduces respectively the number of test cases as showed in group B: 72 285 29 54 = 440 B One observes that the reduced test set (60%) is not proportional to the schedule constraints (50%). There is a gain in terms of test set to be performed. 72% 45% 63% 61% 60% Reduction Percentage

Results Motivation Acceptance Process Service Profiles Test Strategy Faults detected by A and B U1 U2 U3 U4 A 147 failures B 74 failures 7 bad configuration on test execution 2 source code malfunction 7 document inconsistencies A particular system failure can be detected by several Test Cases provided by different testers. In such case study it was observed that 147 failures detected by the test set A were caused by the following faults: The results showed that redundancies in terms of test cases coverage are given by the service-oriented test case generation. Such redundancy allows to choose test case which covers intrinsically more than one service, without losing test efficiency. The 74 failures detected by the reduced test set B were caused by the same faults

Results Questions??? Motivation Acceptance Process Service Profiles Test Strategy Questions??? Financial support to the QSEE Project Quality in Space Application Embedded Software http://www.cea.inpe.br/qsee/