1. Value-based Empirical Study for IV&V ISERN2005 Masa Katahira, JAXA Daniel Port, Univ. of Hawaii

2. Background IV&V is a common method applied to safety-critical software in order to gain quality and safety. NASA, ESA, and JAXA have started the study of strategic IV&V together. IV&V is a common method applied to safety-critical software in order to gain quality and safety. NASA, ESA, and JAXA have started the study of strategic IV&V together. Main issue: strategic planning of IV&V activities Main issue: strategic planning of IV&V activities Balancing safety vs. costBalancing safety vs. cost

3. Generating an Optimal Strategy What is an optimal strategy with respect to cost? What is an optimal strategy with respect to cost? 1. For each attribute, pick technique that gives maximum benefit 2. Order these pairs from max cost-benefit (RE reduction / cost) to lowest

4. Comparison of Strategies strategic

5. Important Problems How much is enough IV&V to perform? How much is enough IV&V to perform? How can we make best use of IV&V efforts with limited resources (budget, schedule, project constraints)? How can we make best use of IV&V efforts with limited resources (budget, schedule, project constraints)? Must provide rationale for plans and budgets Must provide rationale for plans and budgets Explain in a tangible way why IV&V is important and its benefits to managers, customers, and developers Explain in a tangible way why IV&V is important and its benefits to managers, customers, and developers How can we select most cost-effective techniques from hundreds of IV&V techniques? How can we select most cost-effective techniques from hundreds of IV&V techniques? Which organization should perform which techniques? Which organization should perform which techniques? When should we stop IV&V? When should we stop IV&V? How can you assess IV&V Return On Investment? How can you assess IV&V Return On Investment? IV&V Is clearly not Value neutral!

6. Current Case Study   System Characteristics   Budget Limitation   Environment It must be addressed on evidence and past results, not just expert opinion and “best guess.” We must incorporate a continuous improvement and control program based comparing expected results with actual results. IV&V activity  Objectives  Attributes  Techniques MODEL Strategic Planning  Reduced Risk  Effectives  Cost Real Issues after delivery  Problems  Incidents  Accidents Planning is by expert opinion and best guess

7. Current Study Collaboration Study has been just started Collaboration Study has been just started Daniel Port, University of HawaiiDaniel Port, University of Hawaii Masa Katahira, JAXAMasa Katahira, JAXA Haruka Nakao, JAMSSHaruka Nakao, JAMSS Expectation of ISERNers Expectation of ISERNers Opinions for value based IV&VOpinions for value based IV&V Suggestion for gathering the empirical data on IV&VSuggestion for gathering the empirical data on IV&V We will show our data collection format if requested We will show our data collection format if requested

8. Back Up Slides

9. JAXA IV&V techniques structure Main consideration is built in the strategic planning mechanism which need to indicate the attributes having system parameters (numeric numbers) and techniques having IV&V conditions (numeric numbers) Main consideration is built in the strategic planning mechanism which need to indicate the attributes having system parameters (numeric numbers) and techniques having IV&V conditions (numeric numbers) TargetObject Attribute Object Attribute Technique System Parameters IV&V conditions

10. Ｓｙｓｔｅｍ Ｃｈａｒａｃｔ ｅｒｉｓｔｉｃｓ 13 characteristics 13 characteristics (1) Autonomous Control (2) Fault Tolerance (3) Functional Role (4) Dealing Data (5) Relation of Hazard (6) Hazard Control Function (7) Execution Architecture (8) Sub Architecture (9) Time Criticality (10) Number of Component (11) Life Cycle (12) Reuse (13) Methodology Functional Architecture Development Process

11. Potential risk for each IV&V attribute (1) Autonomous Control (2) Fault Tolerance (3) Functional Role (4) Dealing Data (5) Relation of Hazard (6) Hazard Control Function (7) Execution Architecture (8) Sub Architecture (9) Time Criticality (10) Number of Component (11) Life Cycle (12) Reuse (13) Methodology + AutonomousO Not Autonomous O 0FT+ 1FTO 2FT + Central Controller+ Device Controller + Data Relay + System Data+ Mission Data O Cause of Hazard + Hazard Controller O Indirectly Hazard Control O Must Work Function+ Must Not Work Function O Single Task+ Multi Task + SequenceO Event Driven O Hard Real-time+ Soft Real-time O Many+ Not Many + In DevelopmentO In Operation O Exist+ None + Water FallO Spiral Characteristics parameter +: Positive 0: Negative 1 0 1 0 0 0 0 0 0 0 0 1 (A5) Completeness of state transition Total Potential risk if the attribute is not assessed : Potential risk VALUE 3

12. IV&V conditions 8 IV&V conditions 8 IV&V conditions (1) A kind of target document (2) Development phase (3) Period for IV&V (4) Knowledge of System or Operation (5) Developer Support (6) Usability of source code (7) Usability of electrical document (8) Size of document / LOC

13. Efficiency of IV&V techniques Ex. System A Efficiency of Completeness Analysis(T6) for Completeness of state transition(T6) (1) A kind of target document + Natural Language + Flow Chart O Source Code (2) Development phase + Requirement + Design O Manufacture O Test (3) Period of time for IV&V + Enough O Not Enough (4) Knowledge of System or Operation + Enough O Not Enough (5) Support by Developer + Enough O Not Enough (6) Usability of source code + OK O NG (7) Usability of e-document + OK O NG (8) Size of document / LOC + Much O Not Much 1 0 1 0 0 0 1 0 3