1 Using Formal Models of Utility to Guide the Development of Safety-Critical Systems Chris Johnson University of Glasgow, Scotland.

2 Can PRA Guide Formal Methods?

3

4 Classical Decision Theory (f 1,p 1 ;f 2,p 2 ;…;f n,p n )  s  S: V(s) = (  n i=1 p i ).u(f 1  f 2  …f n )  s, s 1  S: s 1  risk s  V(s) > V(s 1 ).

5 Applications of Classical Decision Theory pump_v(exchanger_error, , bbv_error, )  risk analyser_v(analyser_error, ) compound_failure  exchanger_error  bbv_error  AX(display_exchanger_warning  display_bbv_warning) ordered_response_failure  compound_failure  analyser_failure  AX(start_standby_pump  EF(display_analyser_warning  reroute_analysis))

6 Decision Theory and Formal Methods s o |= AX (f ; y) iff: #{  F k  F| (s 0, s 1 )  F k  s 1 |= f  F k } = y #{  F j  F| (s 0, s 1 )  F j  F j } s o |= AX [f P y] iff: #{  s x  P| s x |= f  s x } = y #{  s x  P  s x }

7 RPN Paradoxes

8 Decision Theory and Formal Methods Issues with probability: –limited incident data; –relational databases; –poor interpretation.

9 Why Bother with Utility?

10 Why Bother with Utility?

11 Why Bother with Utility? H. Kortner and A. Kjellsen, Det Norske Veritas

12 Why Bother with Utility

13 Standard Models of Utility Users have a consumption set X. Trade-offs exist between elements of X: There are preference relations over X: –(x 1, x 2 )   “x 1 is at least as good as x 2 ” Axioms avoid paradoxes & define “rationality”.

14 Rationality Axiom 1: Completeness For any x 1  x 2  X either x 1  x 2 or x 1  x 2 Implication 1. The Completeness Axiom makes an unrealistic assumption that designers will be able to distinguish between the different strategies or plans that they can exploit.

15 Rationality Axiom 2: Reflexivity For all x  X x  x Implication 2The Reflexivity Axiom states that any alternative is at least as good as itself but designers may associate different values with different means of obtaining the same outcome.

16 Rationality Axiom 3: Transitivity For any x 1,x 2, x 3  X if x 1  x 2 and x 2  x 3 then x 1  x 3 Implication 3The Transitivity Axiom makes an unrealistic assumption that users act as “rational” consumers in a technical environment that they may not fully understand.

17 Preference Topologies Definition 1 (  preference): constrained to satisfy rationality axioms. Definition 2 (>> strict preference): x 1 >> x 2 iff x 1  x 2 and  (x 2  x 1 ) Definition 3 (~ indifference): x 1 ~ x 2 iff x 1  x 2 and x 2  x 1

18 Preference Topologies For some point x 0 = (x 0 1, x 0 2 ): At least as good as: {x | x  X, x  x 0 }. No better than: {x | x  X, x 0  x} Worse than: {x | x  X, x 0 >> x} Preferred:{x | x  X, x >> x 0 } Indifferent: {x | x  X, x ~ x 0 }

19 Preference Topologies - Shows X as a 2D vector of reals. - Paradox to left of x 0 - So introduce additional axioms.

20 Axioms of Taste: Continuity For all x  R n both  and  are closed. Implication 4The Continuity Axiom ensures topological nicety and is neutral with respect to safety-critical development.

21 Axioms of Taste: Strict Monotonicity For all x 0, x 1  R n + if x 0 is greater than or equal to x 1 then x 0  x 1 while if x 0 is strictly greater than x 1 then x 0 >> x 1. Implication 5The Axiom of Strict Monotonicity fails to characterise certain aspects of safety-critical development in which more of a resource can yield a worse design.

22 Axioms of Taste: Strict Monotonicity Continuity reduces indifference region. Monotonicity ensures all preferred sets are strictly above indifference sets (non-satiation).

23 Axioms of Taste: Strict Convexity If x 1  x 0 and x 1  x 0 then tx 1 +(1-t)x 0 >> x 0 for all t  [0, 1] Implication 6The Axiom of Strict Convexity reflects a “balanced” approach to resource allocation or substitution. As one of the preference axioms of taste, however, it is inappropriate for all forms of safety-critical development.

24 The Way Forward Perhaps I’m missing the point. Quantitative analysis unimportant? But we keep getting PRA wrong: –Formal methods might help?

25 Wider Conclusions Question use of convex utility curves: –in risk analysis and decision theory; –(in stochastic multiplexing; caching etc.) Things are more complex than I thought: –subjectivity; ceteris paribus; risk homeostasis.

26

27 National Attitudes to Risk

28 Caveat Preference relation orders the consumption set, X. Utility functions map preferences onto numeric scale. Utility functions “inherit” complete, reflexive, transitive, continuous and strictly monotonic properties. Time to examine these assumptions...