1. SAS_06_STOL_Tool_Cooper Automated Systems Test and Operations Language (STOL) Analysis Tool Jason G. Cooper jason.cooper@matricresearch.com July 20, 2006

2. 2SAS_06_STOL_Tool_Cooper Outline Problem/Approach Overview Problem/Approach Overview General Discussion General Discussion Technology Readiness Level Technology Readiness Level Potential Applications Potential Applications Data Availability Data Availability Barriers Barriers In-Depth Project Plans In-Depth Project Plans

3. July 20, 2006 3SAS_06_STOL_Tool_Cooper Problem / Approach The IV&V community has expressed a need for automated assistance in regards to STOL script analyses activities The IV&V community has expressed a need for automated assistance in regards to STOL script analyses activities Currently, practitioners have no automated methods to assist in analysis triage for STOL scripts Currently, practitioners have no automated methods to assist in analysis triage for STOL scripts To research algorithms and methodologies most suitable for STOL test script analysis Develop and evaluate an automated analysis tool to assist practitioners with STOL-related tasks Phase One Activities: Initial Knowledge Discovery Tool Software Development Verification and Test Evaluation Recommendations

4. July 20, 2006 4SAS_06_STOL_Tool_Cooper Outline Problem/Approach Overview Problem/Approach Overview General Discussion General Discussion Technology Readiness Level Technology Readiness Level Potential Applications Potential Applications Data Availability Data Availability Barriers Barriers In-Depth Project Plans In-Depth Project Plans

5. July 20, 2006 5SAS_06_STOL_Tool_Cooper Technology Readiness Level (TRL) This project begins at TRL 2 This project begins at TRL 2 TRL 2 (Technology Concept Formulated) is focusing theories and principles on the specific application area TRL 2 (Technology Concept Formulated) is focusing theories and principles on the specific application area In this project’s case, the team will be determining complexity and behavioral metrics to be used for automated analysis techniques In this project’s case, the team will be determining complexity and behavioral metrics to be used for automated analysis techniques

6. July 20, 2006 6SAS_06_STOL_Tool_Cooper Potential Applications The development of such an automated analysis tool has the potential for usage in STOL script activities across multiple domains. The development of such an automated analysis tool has the potential for usage in STOL script activities across multiple domains. A commonly phrased mantra is “test like you fly, fly like you test.” In following this testing framework, an analysis tool could offer benefits not only to verification of flight software test activities, but also to command scripts used for mission operations. A commonly phrased mantra is “test like you fly, fly like you test.” In following this testing framework, an analysis tool could offer benefits not only to verification of flight software test activities, but also to command scripts used for mission operations.

7. July 20, 2006 7SAS_06_STOL_Tool_Cooper Data Availability The research team is currently identifying which STOL version to focus initial efforts, as well as most commonly-occurring STOL script errors in the IV&V domain The research team is currently identifying which STOL version to focus initial efforts, as well as most commonly-occurring STOL script errors in the IV&V domain Several potential projects, which utilized STOL, are available for retrospective analyses Several potential projects, which utilized STOL, are available for retrospective analyses The specific project for test and evaluation will be determined in the latter stages of the software development phase The specific project for test and evaluation will be determined in the latter stages of the software development phase

8. July 20, 2006 8SAS_06_STOL_Tool_Cooper Barriers STOL Version: determining the most common version of the scripting language will limit the applicability of the initial tool. In the second phase, this may be mitigated by supporting multiple versions of the scripting language. STOL Version: determining the most common version of the scripting language will limit the applicability of the initial tool. In the second phase, this may be mitigated by supporting multiple versions of the scripting language. Metrics: standard complexity metrics will apply to portions of this scripting language. However, certain aspects of the language require new metrics to be assessed. Metrics: standard complexity metrics will apply to portions of this scripting language. However, certain aspects of the language require new metrics to be assessed.

9. July 20, 2006 9SAS_06_STOL_Tool_Cooper Outline Problem/Approach Overview Problem/Approach Overview General Discussion General Discussion Technology Readiness Level Technology Readiness Level Potential Applications Potential Applications Data Availability Data Availability Barriers Barriers In-Depth Project Plans In-Depth Project Plans

10. July 20, 2006 10SAS_06_STOL_Tool_Cooper Initial Knowledge Discovery Determine behavioral knowledge to extract from STOL constructs Determine behavioral knowledge to extract from STOL constructs Determine complexity metrics to apply Determine complexity metrics to apply Capture most commonly occurring STOL errors in IV&V project for domain knowledge Capture most commonly occurring STOL errors in IV&V project for domain knowledge

11. July 20, 2006 11SAS_06_STOL_Tool_Cooper Tool Software Development Requirements Definition Requirements Definition Architecture Definition Architecture Definition Implementation Implementation Unit Testing and “Seed” Testing Unit Testing and “Seed” Testing

12. July 20, 2006 12SAS_06_STOL_Tool_Cooper Verification and Test Evaluation Select project and underlying STOL scripts Select project and underlying STOL scripts IV&V practitioner and tool analyze 50% of selected scripts; results comparison IV&V practitioner and tool analyze 50% of selected scripts; results comparison Practitioner utilizes tool for remaining 50% of scripts; feedback and usability captured Practitioner utilizes tool for remaining 50% of scripts; feedback and usability captured

13. July 20, 2006 13SAS_06_STOL_Tool_Cooper Questions?