Balancing Practices: Inspections, Testing, and Others JAXA scenario (formal method) Masa Katahira Japanese Space Agency

Strategy to select methods Methods Compliment –Inspection/Review –Testing –Formal Method Theorem Proving Auto Model Checking Inspections/Reviews –Hard to cover all aspects Testing –Not complete, too late in some case Formal Method –TP: Complex for practical use –MC: State explosion possible We realize that –the correct use of particular methods, –the combination of several methods are very important. But how? –Quality Goals –Budget Limitation –System Characteristics –Data availability, Development Phase

Selection and Scalability of Methodologies (sample) Completeness/Consistency Selection (Depth) Light Full set Phases Modeling/Model Checking Inspection/Review (Check List) Simulation Interface Validation Design Coverage & Timing Verification Coverage Auto Test Case Generation & Robustness Evaluation Test Case & Test Test Result Review Compliance/Traceability Risk Analysis (Robustness) Process & Quality Static analysis (Problem Reports) In line Process Monitor (SMIP) Manual Check (Tools Support) Auto Equivalency Check Hazard Analysis/SFMEA REA Assessment Attributes (Sample)

Still we don’t know how to decide methods’ selection!

Fig.1 Each methods’ effectiveness among all significant issues Fig.2 Each methods’ effectiveness among all Editorial Errors Fig.3 Each methods’ effectiveness among all Significant issues and Editorial errors Consideration For projects A,B,C, there is not enough time to perform model checking. Review with check list instead. For Project D, a checklist is useful for the data correctness and consistency of data handling system. For Project E, the effective of model checking is confirmed due to having enough time. Review also shows important role in erroneous description in the specification For design phase such as C,E, model checking and a checklist help finding errors which can not be found by review. For projects A,B,C, there is not enough time, and the review with check list shows efficiency. For Project D, Review as well as review with checklist shows the efficiency for data handling system. For such as Project E case, when enough time is assigned, the model checking shows good results.

Lessons Learned Summary in JAXA case study MethodAdvantageDisadvantage Formal Model/ Model Checking ● It is useful to find the problem concerning the complicated state/mode transition and processing timing issues which is hard to be found by manual. ● Erroneous description in the spec. at modeling which is more effective than normal review. ● A certain mount of time are necessary to modeling and model checking. ● Need to know modeling and model checking knowledge. ● Low cost effectiveness for software which does not have complicated logic such as data handing, or transformation Review with Checklist (Inspection) ● High cost effectiveness even if there is very short time to access it. ● Erroneous descriptions are covered by check items in the list. ● It does not depend on the skill of evaluators than model methods. ● There are limited based on the items in the checklist. ● It is hard to check the detailed behaviors in the complex system and to cover the possible combination.

Boundary of Formal Method Application Available man-month Safety Critical System Characteristics Complexity Important Border

JAXA formal method activity

Needs and remaining issues of formal method [Problem Statement] Need to assure the high reliability of spacecraft Facing to the difficulty to prove the goodness only by test and inspection because of system complexity and safety requirements such “must not work” Large number of defects are introduced mostly at the Requirement Phase or the Early Design Phase. Unintended or Unexpected system/software behavior is difficult to be found at the inspection/review by manual. [Challenge] Knowledge base inspection/review is still very important, but model checking gives a chance to detect important findings which are easy to miss by the reviewer. Modeling task itself gives a chance to think enough deeply what the specification really says as if the reviewers build the software by themselves. [Remaining Issues] Quality of Model and model checking task –Large amount of time is spent to correct the erroneous model Abstraction and partitioning techniques –To avoid state explosion, and missing the scenarios Better Productivity –Hard to find real problems from thousands of auto checking results Personnel skill

Modeling and model checking in JAXA Requirement Model (SpecTRM *1, Uppaal) Design Model (SpecTRM, Uppaal) Req. Spec. ICD Hazard Report Design Spec. ICD Hazard Report Modeling Natural Language Input Tools Uppaal Flow Diagram Tool Flow Diagram Tool Model Checking (Static Analysis) Completeness Consistency Reachability Completeness Consistency Reachability Equivalency SMV, SPIN Executable Code Simulation (Dynamic Analysis) Robustness Behavior Analysis Findings Test CaseProposal Operation Model Procedure Ope. Scenario Consistency Task Model Tool *1:SpecTRM: Specification Tools and Requirements Methodology Direct Modeling

Productivity improvement Modeling Task Cost

Real issues from the results of the modeling and model checking Identified issues in the specification

Modeling –Can organize information and execute the modeling in the brain –Identify lots of basic problems in the specification (ambiguous descriptions, inconsistency of the contents, unclear data definition) as to make the accurate model of the specification in the formal language Automated Consistency analysis –Effective to identify the inconsistency in the requirement specification –Identify the inconsistency among the procedures in case that multiple tasks executions are allowed simultaneously in the design level. Automated Completeness analysis –Check whether the nodes after the transition at the branch in the flowchart meet the number of the transition conditions and its contents, and whether all error handling and exceptional procedure are covered in the design level. Formal Validation of the functional behavior using SPIN –Effective to validate whether the procedures are executed without stagnation and those behavior meet the requirements for the procedure flow in the detailed design –Effective to verify whether hazard control function/failure recovery functions are working without unintended stop in the real time Lessons Learned from industry use of modeling and model checking

Questions? (Formal Method) A)What is the role of formal method (Theorem Proving, Model checking etc.) in many quality practices? B)When is a Formal Method necessary or efficient? C)What is a Formal Method useful for? Specific Aspects? D)What are most important research issues to deploy the method into real projects? Industry Needs? E)What empirical data gathered at the industry will be useful to future research? F)What is an expected benefit from use of formal method?

Backup Slide

Findings from each methods (Spacecraft’s Projects) Significant: Signification Issues to be modified such as incorrect or missing functions/logic/data Editorial: Editorial Errors in the specification No Issue: Non real problem (misunderstanding/modeling mistake) Type of system Findings type and number from each methods ReviewFormal Modeling and Model Checking Review with Checklist SignificantEditorialNo IssueSum System A (Controller) /Req System B (Controller) /Req System C (Controller)/Design System D (DataHandling), Design System E (Controller)/Design SignificantEditorialNo IssueSumSignificantEditorialNo IssueSum

SpecTRM (Model) Based Robustness Test Environment (SpecRobusT) Outline: By using specification models, the important test cases are generated for full software simulation during development contractor ’ s test phase automatically and comparing results. Especially, all inputs are verified in the model to generate the test cases. Auto tests are performed at 10,000 – 100,000 cases / sec. Results of Project application: # of Test Case ： 550,870,000,000 Benefits: –Verification at very early phase –Introduction to automated test environment –Introduce “ Test Before Development ” paradigm into development process Implementation Procedure