Assurance techniques for code generators Ewen Denney USRA/RIACS, NASA Ames Bernd Fischer ECS, U Southampton

Assurance problem Safety/mission-critical software requires assurance that it meets a certain level of “quality” What are the issues in assuring automatically generated code? –Different forms of assurance –Different assurance techniques –Diverse generator paradigms

Forms of assurance What exactly might we need to assure? Compliance with requirements Compliance with spec/model Certification standards Coding standards Absence of run-time errors Traceability Appropriate documentation  Minimize “automation surprises” Correctness Reliability Legibility

Code generators in practice Practitioner survey carried out in March 2006 (Code Generators in Safety-critical Applications, J. Schumann, E. Denney); 23 responses from NASA and industry. How are ACGs used for safety-critical applications at NASA and in industry? Which are the primary application areas and domains? Which tools are used? Challenges, benefits and problems? How could ACGs be extended to be more useful in safety-critical applications?

Tools and languages The Big Three: Real-Time Workshop MatrixX SCADE

Domains and criticality levels Principle domains: –control –modeling/simulation Many highly critical applications ACG used for –production code (74%) –prototyping (52%) –simulation (48%) –testing (30%) –glue/interface code (30%)

System components

Weaknesses Steep Learning Curve –applicable problems, features, correct usage, architecture, implied methodology, semantic ambiguities, … –substantial impact on development process ACG customization –necessary in 1/3 of cases –often (2/3) done by tool vendor ACG bugs –in 2/3 of applications, bugs were found in ACG

Qualification A code generator is qualified –with respect to a given standard –for a given project if there is sufficient evidence about the generator itself so that V&V need not be carried out on the generated code to certify it Must be done for every project, version Can obtain verification credit Generators are rarely qualified Examples: ASCET-SE (IEC 61508), SCADE, VAPS (DO-178B)

Certification and V&V Auto-generated code must be certified for safety-critical use Techniques used: –testing(90%) –static analysis(58%) –simulation(52%) –manual review(48%) No formal verification No review of generator code

Safety properties

Generator features

Domain-specific analyses Mostly numeric issues: stability (root locus, Lyapunov) robustness convergence transience Some domain-specific design rules: “forbidden” constructs block structure

Documentation Design information Code derivation Configuration management information (to “replay” generation) Safety information Tracing information Interface definitions, requirements User manuals Installation information  Should be customizable

Traceability Most important:model  code Secondary:code  V&V artifacts

Tool integration Also workflow and process tools tools for integrating legacy code

Survey summary Integrated modeling, analysis, and simulation tools are most common in control domain In-house extensions common for modeling and verification issues Natural synergy between code generation and certification activities –perceived but not realized –autocode often treated like manual code Iterative customization of generator should be seen as integral part of development process

Assurance techniques Testing the generator (qualification) –for all specs, blocks, configurations, backends, … Post factum verification / certification –verify / certify generated programs individually Correctness by construction –generator inherently guarantees certain properties Documentation Traceability

Discussion questions What are the interesting assurance artifacts, properties, etc. in your target domains? What are suitable notions of documentation, traceability, development process? What assurance techniques have you tried? How is the generative knowledge represented (templates, transformation rules, etc.) and how can it be combined with assurance information? Can we apply Design for Verification (D4V) to generators?