1. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-1 The Insufficiency of Formal Design Methods - the necessity of an experimental approach for the understanding and control of complex MAS Bruce Edmonds Centre for Policy Modelling Manchester Metropolitan University Joanna Bryson Department of Computer Science, University of Bath

2. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-2 Main Themes The need for a shift of emphasis: –From verification towards validation –From the “10%” towards the “90%” (in the adage) –From engineering (of new systems) to adaptation (of existing systems) This goes against implicit assumptions (in SE): –Design/engineering is “better” than testing/adaptation –Computational systems are deterministic (in reality) –Since (aspects of) software systems can be represented as formal objects that formalist approaches will be able to deliver (increased) reliability Whilst this shift may be resistible with single, closed systems it will be unavoidable with MAS

3. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-3 Outline of the talk 1.Some formal limitations to formal approaches (why a “Hilbert Programme” for MAS won’t work) 2.Software production strategies (the primary SE goal and strategies to get there) 3.A more “Scientific” approach for SE (utilising the classic experimental method)

4. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-4 Part 1: Some Formal Limitations of Formal Methods - how simple MAS can be and it still be impossible to bridge the specification  program gap

5. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-5 Formal Approach Idea is to write specifications in a formal language Often of a logical or set-theoretic nature Two undisputed advantages: 1.Specification is unambiguous and concise 2.Specifications can be syntactically manipulated (as in proofs, checkers etc.) Is thus a sort of lingua franca for software engineers As with any language, there are difficulties that arise when attempting to translate to and from it To it (from the informal) - the “requirements problem” From it (to programs) - the “problems” that follow

6. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-6 The Programming Problem Posed SxSx A given formal specification PyPy Any program that satisfies S x A translating program? T Is there a general, effective or systematic method of finding a program that satisfies a given specification?

7. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-7 The Programming Problem Answered No, if the language of specification is expressive enough SxSx A given formal specification PyPy Any program that satisfies S x A translating program? T

8. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-8 The Checking Problem Posed SxSx A given formal specification PyPy A given program A checking program? T Is there a general, effective or systematic method of checking whether a given program satisfies a given specification?

9. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-9 The Checking Problem Answered SxSx A given formal specification PyPy A given program A checking program? T No, if the language of specification is expressive enough

10. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-10 What “expressive enough” means E.g. Those that are able to express basic arithmetic statements about the behaviour of programs That is, able to express statements such as: –Output of program P x is always < 100 –Program P x will not finish before time 1000 –Output of P x is always greater than P y –Output of P x is a approximates that of P y (±10%) (There are other ways such expressiveness can be established: 2-place predicates etc.) Thus in SE terms “expressive enough” is, in fact, the minimum necessary for almost all real systems

11. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-11 Proof Sketch (programming problem answer) Define the “n th limited halting problem”, LH n (x,y), as: Does P x ever halt with input y where both x,y  n? Each LH n (x,y) is computable as a finite lookup table LH n (x,y) is effectively expressible in a language with arithmetic via the construction in (Gödel 1933) as SH m - that is, m here is computable from n Now if there were a translating program, T, then: given x and y; let z=max(x,y); compute SH z ; use T to find a program to compute LH z (x,y) from SH m ; and use this to find whether P x (y) halts; but this is impossible (Turing 1936).

12. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-12 An example: how “simple” a MAS can be and still be beyond formal methods Giving Agent System with Plans (GASP) Fixed number of agents: A 1, A 2, …A n Each agent, A x, has –a single store, S x –a fixed number of plans: P x1, P x2, … Each Plan, P xy, is composed of instructions: –A fixed number of “give one to …” –And one final test instruction: If S i is zero then do plan j next, otherwise plan k next Each time click, all do: get 1 unit; use current plan to: [do giving (while they have); do test instruction; note next plan].

13. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-13 An illustration of a GASP system Plan 1: G 3 G 2 JZ 2,1,3 Plan 2: JZ 1,2,3 Plan 3: G 2 JZ 2,3,3 Agent 1 123 Agent 2 123 Agent 3 Etc. Check if zero 4 27 Store:

14. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-14 Facts about GASP systems They are Turing Complete (see paper for proof outline), hence: …many questions about their behaviour are (in general) undecidable even when given their program In particular, a specification adequate to specifying the behaviour of GASP systems will need basic arithmetic and so… …the previous results about the (difficulties of the) relationship between an adequate specification language and systems hold

15. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-15 Part 2: Software Production Strategies - Specification and Design Strategies and their problems

16. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-16 Goals and Strategies for engineering computational (IT) systems To produce IT systems that work well in practice when working in their operational context One particular strategy to achieve this is the “formal design strategy” (FDS), summarised as: 1.Agree the goals for the system; 2.Write a specification that would meet these goals; 3.Implement a system that meets this specification. Works well for relatively simple, closed, static and analysable cases FDS is obviously a “straw man” but consistent with much rhetoric in MAS and true to the extent that there is an over-emphasis on these stages

17. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-17 Some (well known) causes of problems with the FDS Context of operation is (at least partially) unknown to designers Good in practice operation requires meaningful, complex and abstract goals, Thus, either one has a: High-level specification, in which case you can’t guarantee that the system works according to its specification A Low-level specification in which case you can’t guarantee that the specification achieves the goals Or many staged levels of specification with many chances for errors due to repeated translation

18. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-18 Some kinds of complexity Syntactic Complexity –When the computational ‘distance’ between initial conditions and outcomes is too great to be analytically bridgeable  There are different views of a system Semantic Complexity –When any formal representation is necessarily incomplete  Models are context-dependent  You probably need many of them

19. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-19 Inseparable system embedding When the particular system is embedded into a wider system such that… the wider system can not be separated from the particular system to aid analysis… without changing the behaviour of both particular system and the wider system,… so that off-line analysis and design is difficult and ineffective. When “particular system” = agent (i.e. MAS), such embedding can emerge in a wide variety of kinds of systems and environment making off-line and formal design strategies inappropriate

20. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-20 Engineered Agent Approach  Beliefs, intentions, etc.  Automatic verification  Ontologies, protocols  Agents, groups, teams  Logics  Roles  Deferred decision making  Testing, model checking Adaptive Approach  Actor  Simulation  Agents, groups, societies  Social&biological analogies  Duplication, competition  Social&individual learning  Post hoc exploration/experimentation Some (well known) software production strategies Abstraction Automation Standardisation Modularity Formalisation Transparency Redundancy Adaptivity Testing

21. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-21 Why such over-emphasis on (the rhetoric of) formal design strategies? Some guesses: Illusion that computational systems are deterministic at macro level in practice because they are in theory at micro level Blame is contained to specification  system production stages, and hence substantially defrayed from designer to user Fixed goals suit management, external consultants and academics Used to dealing with simple, closed systems Engineering is seen as “better” than adaptation

22. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-22 Part 3: An Experimental Approach - an alternative for messy systems and situations

23. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-23 What if the properties of most MAS are messy - more like Biology than Logic? Lots of kinds of agents, teams, trust, communication forms, etc. Lots of observation and exploration before any abstraction into theory possible A priori foundationalist studies based on plausibility probably worse than useless Success coming more from: what works in context, reliability, and adaptation to the unexpected And less from: abstraction, modularity and formal analysis

24. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-24 The Classic Experimental Method Theory developed empirically (not a priori) Applied using well-validated processes and tools How the theory can be applied using what approximations is empirically established The conditions under which a theory can be safely applied (and how) developed over time Useful properties can only be deduced after theory has been validated Has worked in messy systems (in science) where there is little that can be completely generalised

25. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-25 What this might give us… Explicit, relevant and testable models/hypotheses concerning the properties of certain MAS With sets of conditions under which it has survived trials/testing (and the extent of success) Also a set of situations where the hypotheses failed, to indicate the limits of its applicability Giving confidence and guidance to those who wish to use these systems From which inferences can be made A scientific basis on which to build sound engineering practices (e.g. systematisation)

26. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-26 E.g. Engineering a bridge Use of well-validated general designs and strategies (e.g. arches, columns, suspension) Multiple approximate calculations (maximum stress, weight, compression) Use of well-validated components or components made using well-validated techniques (e.g. standard girders or cable) Simulations of the set-up (e.g. oscillations) …still the unexpected may occur - no illusion that design proof can be used on whole systems to achieve reliability

27. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-27 A Comparison of Approaches (at present time) Formal/Inferential Precise criteria for success Well-developed methodology/tools Taken from formal sciences Limited applicability For Small components As a check with simplified models Inductive/Experimental Weak criteria of success Methodology/tools need improving From natural sciences Wide applicability On real systems No certainty An eternal task

28. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-28 Strengthening the whole chain of inference about MAS Target MAS Formal or Computational Model Adjustable aspect (AA) Observable outcome (OO) Setting or set-up Inference or results It is the strength of the whole chain from AA to OO that matters Strengthening the inference stage is counterproductive if this weakens the whole chain

29. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-29 Some practical steps… Change the rhetoric and expectations of AOSE Import well-tried testing techniques from other sciences, e.g. [Bryson, Lowe & Stein 2000] Independently replicating simulation experiments [Axtel et. al. 1996, Edmonds & Hales 2003] Extend the open source model to include hypotheses, test results etc. [Edmonds 2004] Develop experimental experience and methodology [MABS, MAMABS, ESOA, JASSS] Tools for analysis of MAS [Barber & Lam 2004] Show how this approach can help achieve robust engineering [Hales & Edmonds 2004]

30. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-30 Conclusion –for messy MAS and MAS environments we am suggesting… a move away from a foundationalist approach towards a more empiricist approach… with less emphasis on prior verification and more on post hoc validation… where reliability is sought from experimentally tested theories of system behaviour… (formalisation entering once there is such theory) which may entail a loss of theoretical certainty… but offers greater potential for adaptation and hence in practice performance, including some robustness in uncertain and complex situations

31. Edmonds & Bryson, The Insufficiency of Formal Design Methods - the necessity of an experimental approach, AAMAS 2004, cfpm.org/papers/ifdm slide-31 The End Bruce Edmonds bruce.edmonds.name Joanna Bryson www.cs.bath.ac.uk/~jjb