1. SOLAR ORBITER SOC Test Plans Nana Bach SOWG#7 – 6-9 July 2015

2. Contents Solar Orbiter SOC Test Plan Identification of different Test Levels SOC Internal Testing Process Continuous Delivery approach Requirements Tracing SOC Interface Testing Interaction with different groups System Test Conclusions SOWG#7 – ESAC – 6-9 July 2015

3. Solar Orbiter SOC Test Plan Describes the strategy, scope, approach, resources and schedule of the testing activities within the SOC. Identifies all the needed levels of testing, processes, actors and harnesses. Identifies the deadlines for readiness of the SOC. Identifies the roadmap to fulfil the System Requirements and calculate the traceability. Describes the testing roles and responsibilities, along with the technical activities, interfaces, and scheduling. Is based on SVVP (System Verification and Validation Plan) which identifies the Verification and Validation Strategy and classifies the SRD requirements and their methods of Verification and Validation. Will trace all the requirements identified in SVVP as to be verified by “Testing”. SOWG#7 – ESAC – 6-9 July 2015

4. Solar Orbiter SOC Test Plan SOWG#7 – ESAC – 6-9 July 2015 SVVP Software Product Test Specification Template Software Test Specification 1 Software Test Specification 2 Sub-System Test Specification Template Sub-System 1 Test Specification Sub-System 2 Test Specification External Interfaces Test Specification Template Interface to MOC Test Specification Interface to Instrument Teams Test Specification System Test Specification Template Operational Rehearsal 1 Test Specification End-to-end Test Specification SOC Test Plan Software Product Test Report Template Softwar e Test Report 2 Software Test Report 1 Sub-System Test Report Template Sub-System 1 Test Report Sub-System 2 Test Report External Interfaces Test Report Template Interface to Instrument Teams Test Report Interface to MOC Test Report System Test Report Template Operational Rehearsal 1 Test Report End-to-end Test Report Document Tree:

5. Identification of different Test Levels SOWG#7 – ESAC – 6-9 July 2015 System Sub- System 1 Software Product 1 Software Product 2 Sub- System 2 Software Product 3 External Interface Software Product 4 The Solar Orbiter SOC is identified as a System composed by different Sub- Systems. Each Sub-System can have one or multiple Software Products beneath. Smallest identified components to be tested are the Software Products. The functionalities, processes and different Software Products are grouped by the Sub-Systems into one functional/procedural unit. The Sub-Systems which require involvement of External Parties are called External Interfaces. We distinguish between externally and internally developed SP.

6. Identification of different Test Levels Therefore the 4 identified levels of Testing are: 1.Software Product Level Test  Unit Tests  Functional Tests  Non-Functional Tests Performance Usability Tests Reliability Tests Robustness Tests Load Tests Stress Tests  Integration Tests  Regression Tests SOWG#7 – ESAC – 6-9 July 2015

7. Identification of different Test Levels 1.z 2.Sub-System Level Test  Use Case Testing  Sub-System Validation  Sub-System Integration 3.External Interfaces Level test  MOC-SOC Interface Tests  Instrument Teams Interface Tests  Science Archive Interface Tests 1.System Level Test  End-to-End Tests  Operational Rehearsals  SOVTs / Post-SVTs SOWG#7 – ESAC – 6-9 July 2015

8. SOC Internal Testing Process SOWG#7 – ESAC – 6-9 July 2015 System SRD Sub-System 1 Use Cases Software Product 1 User Stories Software Product 2 User Stories Sub-System 2 Use Cases Software Product 3 User Stories External Interface Use Cases Software Product 4 User Stories Each logical group of functionalities have been described in Use Cases. The Sub-Systems Test Level can be split into the Sub-System Use Case testing as well as the Sub-System Validation. Use Cases can be integrated into a bigger functional pieces, building up to full Sub-System. Tests against the external interfaces to the SOC like e.g. The Instrument Teams or MOC are External Interfaces Tests. System Level Tests involve the whole System e.g. the SOVTs/SVTs, Operational Rehearsals, End-to-End tests etc. The SRD is linked to the Use Cases. The Use Cases are split into User Stories and implemented within SCRUM development technology.

9. SOC Internal Testing Process Steps of the development/testing process: 1.In Sprint Planning Meeting decompose Use Cases into functional blocks. 2.Implement the code corresponding to each functional block. 3.Implement the acceptance test (Cucumber) corresponding to the peace of code. 4.Run the test in Hudson/Jenkins as part of automated build. 5.When the entire Use Case is ready perform the test on Sub- System Level. 6.Validate the Sub-System when all the Use Cases implemented. 7.Integrate Sub-Systems into the System (or parts of it). 8.Perform the External Interfaces tests. 9.Perform the System Test. SOWG#7 – ESAC – 6-9 July 2015

10. Continuous Delivery Approach SOWG#7 – ESAC – 6-9 July 2015 Continuous Integration and Delivery approach. Automated build, test and deployment approach. Behaviour Driven Development strategy (BDD). Continuous integration and test framework Hudson and Jenkins. Test integration and functional framework Cucumber. Feature implementation leads to automated testing. Releases of Software Products on automated basis. The Sub-System testing automated as much as possible. Automated Use Case testing whenever possible. SW Develop ment Unit Test Suite of Feature Accepta nce Tests Release/ Deploym ent

11. Requirements Tracing SOWG#7 – ESAC – 6-9 July 2015 SIRD Requirements are linked to the System Requirements. The System Requirements are summarized after their scope. No breaking down by the Sub-Systems. Test Specification Documents will contain traceability to the SRD. Each of the Test Cases will link to one or multiple System Requirements. Test Reports will show which of the Requirements have been tested. The System Requirements will be classified into different categories. Requirement Verification and Validation Matrix in the SVVP.

12. Requirements Tracing SOWG#7 – ESAC – 6-9 July 2015 No further breakdown on the System Requirements. Software Requirements for each of the Software Products summarized in the Cucumber Features. The Use Cases are going to be decomposed into different User Stories (within the Jira, SCRUM methodology) and summarize one single functionality or group of functionalities, which can be considered as “functional” Software Requirements. Described in the Feature Description within the automated Test framework Cucumber. For non-functional Software Requirements a Cucumber feature will also exist. The summary of all the Cucumber features for a given Software Product will represent its Software Requirements.

13. SOC Interface Testing Identified Interfaces: MOC (SOC External Interface) GFTS EDDS Instrument Teams (SOC External Interface) GFTS Server Low Latency Pipelines (also considered as externally delivered Software Product) Low Latency Pipelines Visualizations Science Archive (SOC Internal Interface) SOWG#7 – ESAC – 6-9 July 2015

14. Interaction with different groups Different groups: SOC MOC Instrument Teams/PIs External developers/possible COTS Define the Interfaces (ICDs) Assure the Interfaces have been implemented and tested correctly (External Interfaces Test Specifications and corresponding Test Reports). Define the communication channels between the different groups. Define Operational Procedures. Validate Operational Procedures. Define Procedures to be followed in cases of anomalies/disaster. Identify clear milestones in the schedule. SOWG#7 – ESAC – 6-9 July 2015

15. System Test 1.End-to-End tests Test the System as a hole Use when possible realistic (or even real) data sets 2.Operational Rehearsals Test Operational Procedures Define Operational Scenarios/Environments 3.SOVTs/ Post-SVTs Verify the readiness of the SGS with the SOVTs Use the output of the SVTs run by the MOC => Formal TRR/TRB procedures to be held for each of the System Tests SOWG#7 – ESAC – 6-9 July 2015

16. Conclusions A lot of work still to be done … SOWG#7 – ESAC – 6-9 July 2015