Notes on: Is Proof More Cost- Effective Than Testing? by Steve King, Jonathan Hammond, Rob Chapman, Andy Pryor Prepared by Stephen M. Thebaut, Ph.D. University of Florida

UK Defense Standard For procurement of safety critical software in defense equipment Perceived emphasis on formal methods (formal specification and design, proofs of correctness) Purpose of this paper: report on work showing that using formal methods on a large scale is practical and effective at both the specification and code level

SHOLIS Ship Helicopter Operating Limits Information System – safety critical system aiding safe operation of helicopters on naval vessels Z-based formal specification and SPARK programming language (subset of Ada) –Omitted features: gotos, aliasing, default parameters for subprograms, side-effects in functions, recursion, tasks, user-defined exceptions, exception handlers, & generics –simplification of other features

Proofs Specification-Based –Approx. 150 proofs in 500 pages –Consistency of global variables and constants, existence of initial states and checking of preconditions Code-Based –Approx. 9,000 verification conditions verified –Predicate transforms used for non-looping programs –Invariants used to prove partial (weak) correctness of looping programs + separate arguments for termination

Some Observations Skills needed: knowledge of Z, proof by cases and contradiction, understanding of imperative programming constructs, familiarity with proof concepts such as loop invariants (not that much!) Proof validation: formal peer + IV&V reviews; proofs were not inspected by the customer Z proof appeared to be substantially more efficient at finding faults than the most efficient testing phase.

Some Observations (cont’d) Code proofs appeared to be more efficient than unit testing, despite the fact that substantial amounts of unit testing were completed before the bulk of code proof started.

Specification using “Z” Z is a mature and expressive notation for model-based specification. It combines formal and informal descriptions and incorporates graphical highlighting. The basic building blocks of Z-based specifications are schemas. Schemas identify state variables and define constraints and operations in terms of those variables.

Schemas Schemas are comprised of three parts: a name, a signature, and a predicate. –The signature declares (state) entities introduced in a schema and their type. –Signature declarations are given in the form identifier: type. –The predicate defines logical relationships between entities in the declaration. Z is based on the predicate calculus + typed sets.

Graphical Layout of a Schema Signature Predicate

Schemas (Cont.) Schemas describe state entities or operations on state entities. –To describe state entities, declared variables comprise the entities while predicates specify their invariant properties. –To describe operations, declarations consist of initial and final state variables, inputs, and outputs, while predicates specify the relations among them.

Example 1 Container contents: N capacity: N contents  capacity Specification of a generic container entity: modeled as natural numbers invariant property

Example 1 (Cont.) Indicator light: {off, on} reading: N danger_level: N light = on  reading  danger_level Specification of a generic indicator entity:

Example 1 (Cont.) Storage_tank Container Indicator reading = contents capacity = 500 danger_level = 50 Specification of a storage tank entity: Predicates on separate lines are separated by an implicit “AND”.

Example 1 (Cont.) Storage_tank contents: N capacity: N light: {off, on} reading: N danger_level: N contents  capacity light = on  reading  danger_level reading = contents capacity = 500 danger_level = 50 Expanded specification of a storage tank entity:

Example 1 (Cont.) Fill-OK  Storage-tank amount?: N contents + amount?  capacity contents’ = contents + amount? Specification of a nominal storage tank fill operation: “dash” indicates the value of contents AFTER the operation “?” indicates that amount is an INPUT. “Delta” indicates that values of one or more variables will be changed.

Example 1 (Cont.) OverFill  Storage-tank amount?: N r!: seq CHAR capacity < contents + amount? r! = “Insufficient tank capacity – Fill cancelled” Specification of an exceptional storage tank fill operation: “Xi” indicates that NO variables will be changed by the operation. “!” indicates that r is an OUTPUT.

Example 1 (Cont.) Fill Fill-OK V OverFill Complete specification of the storage tank fill operation:

Key Points Model-based specification relies on a state model of the system using mathematical entities such as the predicate calculus and typed sets. Functions / operations may be specified by defining their effect on system state. It is normal to specify functions / operations incrementally and then combine the fragments to produce a complete specification.

Key Points (Cont.) Z specifications are comprised of a number of schemas that introduce typed names and define predicates over those names. They are distinguished from surrounding text by graphical highlighting. Schemas are building blocks which may be combined and used in other schemas. The effect of including schema A in schema B is that schema B inherits the names and predicates of schema A.

Notes on: Is Proof More Cost- Effective Than Testing? by Steve King, Jonathan Hammond, Rob Chapman, Andy Pryor Prepared by Stephen M. Thebaut, Ph.D. University of Florida