1. EMER: Engineering Simulations for scientific research

2. EME : 2 Complex systems simulation Complex systems comprise component systems that interact with each other and their environment Which is itself comprised of complex systems Scientific research in complex systems has many challenges: –Observation interferes with the system –Too many factors and variables for comprehension –Over-reliance on simplistic models

3. EME : 3 Example: immune systems Immunologists need to understand how diseases attack organisms in order to target therapies –Diseases only develop naturally in living organisms –Limitations to how disease development can be observed Kill the subject and work out what has been going on –Disease progress involves killing 1000s of mice Invasive procedures involving nano-cameras or xrays, chemical dyes etc –Expensive, and possibly interferes with disease itself Can simulation provide an alternative experimental subject? –Can a simulation be realistic enough to do real science on?

4. EME : 4 Models of complex systems Simulations are undertaken for different reasons –Developers must be clear about purpose Models that mimic and “predict” or project –e.g. mathematical equations that reproduce observed measurements and statistics differential equations Models that describe –models made up of components and an environment Mathematical individual-based models (IBMs) Agent-based models (ABMs) All inadequate, but in different ways

5. EME : 5 Mathematical models to “predict” Simulating observed behaviour –e.g. population dynamics using differential equations Lotka-Volterra model of predator-prey interaction Logistic equation models of population vs resources –Population-level integrated view –Analytic rigour Good for general trends, but… –Minimal insight to system dynamics –Intractable if too many parameters –Many (vague/unrealistic) assumptions

6. EME : 6 Component-based simulations Design components and environment; initialise lots of components; see what happens –Capability to express heterogeneity –“realistic” interactions Good for exploring, but… –Often poorly engineered –Scalability often limited by inefficient programming or execution –Mapping between reality and simulation often weak

7. EME : 7 Commentary on simulation Many scientists like the analytic strength of mathematical models of behaviour ODEs, PDEs, logistic maps, etc –But parameter-tweaking is rarely validated properly Many scientists distrust computer-based simulations –Because the analogies to real systems are not made clear Concerns about component-based simulation can be reduced by working closely with scientists –Clarity over modelling and design assumptions –Exploration of scientific meaning and assumptions –Challenged assumptions can lead to new experiments In the science and/or in the simulation

8. EME : 8 Complex system simulations: problems Simulations seek to achieve behaviours that are observed in real systems –Emergent behaviours should emerge in simulation Is a simulation valid just because it gives apparently realistic behaviours? e.g. ALife systems often look like the real thing, but the comparison is barely skin deep –Need realistic behaviour from realistic structure and behaviour We need to engineer realistic simulations –Simulations that scientists trust www.cogs.susx.ac.uk/users/ezequiel/wheeler-etal-2002.pdf

9. EME : 9 A PRINCIPLED APPROACH TO SIMULATION The CoSMoS Project, 2007-11 Complex Systems Modelling andSimulation Infrastructure York, Kent, Abertay, UWE

10. EME : 10 CoSMoS www.cs.york.ac.uk/ftpdir/reports/2010/YCS/453/YCS-2010-453.pdf

11. EME : 11 Domain It is not the role of the developer to make decisions about science A simulation is built for a purpose in a domain –Domain information comes from domain experts responsible for scientific scope of the simulation provide disambiguation, answer questions, etc. –Information in the form of papers, expert insight, lab experiments and results, text books… Domain experts and developers negotiate scope and detail level for simulation, and purpose of simulation exercise

12. EME : 12 Some information on domain of lymphocyte capture Andrews et al: www.cosmos-research.org/docs/cosmos2008-understanding.pdf

13. EME : 13 Domain Model Built by developer, checked by domain expert Represents developer’s interpretation of the domain Strict scoping –The domain expert may want more detail, wider scope, etc. –The developer has to focus on the established purpose Different purposes require different domain models See also: Read et al: www.cosmos-research.org/docs/cosmos2009-read.pdf and www.cosmos-research.org/docs/icaris2009-eae.pdfwww.cosmos-research.org/docs/cosmos2009-read.pdf ⁻Focuses on the scientific understanding no implementation details

14. EME : 14 Domain issues Domain experts do not know everything Authoritative scientific papers may be: –Economical with the truth –Following accepted conventions that have no firm basis –Inconsistent with other scientists’ observations Data accuracy may be low –Did all the data come from the same individual? Or even the same (sub)species? –How accurately can a data point be measured? Often only to within several orders of magnitude –Is the datum “real” or an estimate? Some things cannot be measured or understood with current scientific techniques

15. EME : 15 Research context All the assumptions and decisions Record sources and assumptions Achieve deep understanding of the purpose of simulation e.g. Some assumptions from lymphocyte capture study: –The detail provided by domain experts is correct –Lymphocyte migration only takes place in the HEV areas of the lymph node –All lymphocytes are “the same”, and there is no interaction between lymphocytes: they do not collide –Once a suitable chemokine signal has been received by a lymphocyte, it always migrates –Lymphocytes are created, die, and flow through the HEV at a constant rate Andrews et al: www.cosmos-research.org/docs/cosmos2008-understanding.pdf ⁻Separating expert and developer roles encourages developers to challenge domain understanding

16. EME : 16 Platform Model, Simulation Platform The developer’s domain Design and implementation –Must be traceably linked to domain model and research context Must record design decisions, simplifications, computational necessities –In order to understand any results from simulations Simulation Platform encodes the platform model in software and hardware –Simulations are run on the simulation platform (simulator) –Simulations equate to scientific experiments –Simulator may be modified for different simulations

17. EME : 17 Results Model A simulator is not the same as the domain –Results of simulation must be mapped back to the domain In a simple case, each domain concept maps directly to a computational concept In the usual case, computational artefacts do not map cleanly –Results model records mappings –With research context, presents understanding of mappings, assumptions, limitations Put results in context

18. EME : 18 CoSMoS Purposes and responsibilities A simulation is built for a purpose –The purpose relates to the domain –Domain information comes from domain experts responsible for scientific scope of the simulation provide disambiguation, answer questions, etc. Engineering systematically develops –from a model of the domain –to a platform model –to a platform, or simulator used to run a series of experiments or simulations

19. EME : 19 Assurance and documentation Domain modelling includes recording of sources and assumptions Platform modelling includes recording design decisions Assurance uses evidence to support a case that the simulator is adequate to support a specific set of simulation experiments –Justification of assumptions, simplifications, design decisions –Mappings from domain concepts to platform components Assurance also calls on evidence of engineering quality –Testing and other verification activities

20. EME : 20 Engineering computer simulations Two main sources of inspiration for an engineering approach to computer simulation –Simulation approaches devised for conventional systems –Critical systems engineering Key features –Validation and arguments of validity –Modelling approaches that support verification Consider what these mean in relation to complex system simulations that could be used to support scientific research

21. EME : 21 Conventional engineering of simulations Work on simulation by R. G. Sargent over 30 years –Conventional systems A lifecycle model: –Elements of a development Verification and validation citeseerx.ist.psu.edu/viewdoc/sum mary?doi=10.1.1.17.438

22. EME : 22 Validation of computerised models Some of Sargent’s proposals: –Comparison with valid models, with real events –Domain expert appraisal Turing test on the expert using the simulation –Historic or predictive data validity –Combination of rationalisation of assumptions, empirical validity checks and sound theory Most assume that reality is understood or already expressed in validated models –Not realistic for most complex systems simulations We can construct assurance arguments See: Polack, www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS- Workshop-2010.pdfwww.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS- Workshop-2010.pdf Ghetiu et al: www.cosmos-research.org/docs/valid2010-arguments.pdf

23. EME : 23 Verification: software Complex systems have complex interactions –A lot of processes going on in parallel, with periodic synchronisations Simulations need to consider how to check the protocols and structures used –e.g. systematic consideration of assumptions made about time and space Including assumptions made to turn parallel reality into sequential and/or OO code –e.g. formal proof of protocols occam-pi synchronisations can be proved deadlock-free Systematic recording and challenging of design assumptions

24. EME : 24 Validation: Assurance arguments We can adapt techniques used in safety case arguments –We can use Goal Structuring Notation or something very similar Present the structure of an argument –Expose the argument to external review –Identify evidence on which the argument is built Attaining this level of understanding helps establish trust between scientists and developers –Potentially helps convince other scientists See: Polack, www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS- Workshop-2010.pdfwww.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS- Workshop-2010.pdf

25. EME : 25 Recap: Basic Goal Structuring Notation See T.P.Kelly, PhD thesis, www.cs.york.ac.uk/ftpdir/reports/99/YCST/05/YCST-99-05.pdfwww.cs.york.ac.uk/ftpdir/reports/99/YCST/05/YCST-99-05.pdf R. Weaver, PhD these, www.cs.york.ac.uk/ftpdir/reports/2004/YCST/01/YCST-2004-01.pdfwww.cs.york.ac.uk/ftpdir/reports/2004/YCST/01/YCST-2004-01.pdf and papers by Kelly’s group, www-users.cs.york.ac.uk/~tpk/pubs.html

26. EME : 26 GSN: arguing simulation adequacy Polack, CoSMoS 2010: www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS-Workshop-2010.pdf

27. EME : 27 GSN: justification Polack, CoSMoS 2010: www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS-Workshop-2010.pdf

28. EME : 28 Elaborating parts of the argument www.cosmos-research.org/wp/wp-content/uploads/Polack_CoSMoS-Workshop-2010.pdf

29. EME : 29 An alternative argument style Ghetiu et al: Valid 2010: www.cosmos-research.org/docs/valid2010-arguments.pdf

30. EME : 30 Something different: arguing trust! Ghetiu et al: www.cosmos- research.org/docs/valid2010-arguments-slides.pdf

31. EME : 31 Validation: common problems Design steps typically use diagrams and related text or mathematics –To represent state, operations, structures of behaviour We’d like to use these in engineering simulations Existing diagram approaches do not express all that is needed for complex systems –Time –Space –Relative numbers of components Specialist formalisms exist but need experts –eg. process algebras See www-users.cs.york.ac.uk/~susan/bib/ss/nonstd/alife08c.pdf

32. EME : 32 Notes for 2013 The CoSMoS website has been hacked: most material can be found under –www-users.cs.york.ac.uk/~susan/bib/ss/nonstd/www-users.cs.york.ac.uk/~susan/bib/ss/nonstd/ –www-users.cs.york.ac.uk/~fiona/PUBS/recent.htmlwww-users.cs.york.ac.uk/~fiona/PUBS/recent.html We have identified that fitness for purpose is a better term than validation: –Validation in CS tends to imply a Boolean It is valid, it is not valid –In complex systems, validity is contingent Valid in this context for this purpose