Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 1 Context-dependency, risk analysis and policy modelling Bruce Edmonds Centre for Policy Modelling, Manchester Metropolitan University

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 2 the Problem Part 1:

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 3 “A large amount of research is pertinent to public policy. But a seemingly tiny amount of that knowledge makes it through to the policy world. This is a frustration not only for the research community but also for the policy community…” (Chris Tyler 2010)

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 4 The “gap” between the worlds of research and that of policy There is often a mismatch between the wishes of policy advisors/makers and academic researchers in terms of the kind of conclusion desired. Those in the policy world seem to wish for a relatively simple “best guess” (or tools to produce these) possibly accompanied by a supporting narrative Those in the research community might be all-too aware of the difficulties of the issues under consideration and so wish to be much more cautious in their conclusions, and wish to include special cases, caveats and complexities.

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 5 Some Bad Solutions The policy world can try to ‘force’ researchers to produce simple answers which academics are unhappy with or insist they solve impossible problems with inadequate means Researchers can essentially try to ‘con’ those who commission research, claiming that they will answer all the questions that are specified, but re-defining the goals/terms as they go along or ‘discovering’ obstacles to them satisfying the policy makers/advisors as they envisaged The researcher and/or the policy advisor mutually convince themselves and each other that the research gives definitive, simple and possibly general answers and rush to implement the policy indicated

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 6 The “gap” between the worlds of research and that of policy There are many reasons behind these very different views… …including different: kinds of goal, motivations, training, strategies, language, habits, constraints, working environments etc. This talk will focus on a particular difficulty, that caused by the context-dependency of human socio-political behaviour and make some suggestions to improve things

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 7 An Analysis of a Part of the Problem: Context-Dependency Part 2:

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 8 A (simplistic) illustration of context from the point of view of an actor

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 9 Situational Context The situation in which an event takes place This is indefinitely extensive, it could include anything relevant or coincident The time and place specify it, but relevant details might not be retrievable from this It is almost universal to abstract to what is relevant about these to a recognised type when communicating about this Thus the question “What was the context?” often effectively means “What about the situation do I need to know to understand?

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 10 Context-dependency of Cognition “The” context is the situation of an event, but this is indefinitely extensive The brain somehow categorises and recognises different kinds of situation and preferentially gives access to knowledge on this basis, it is context-dependent Many aspects of human cognition seem to be context-dependent, including: memory, visual perception, choice making, reasoning, emotion, and language

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 11 The Context Heuristic The kind of situation is recognised in a rich, fuzzy, complex and unconscious manner Knowledge, habits, norms etc. are learnt for that kind of situation and are retrieved for it Reasoning, learning, interaction happens with respect to the recognised kind of situation Context allows for the world to be dealt with by type of situation, and hence makes reasoning/learning etc. feasible It is a fallible heuristic… …so why do we have this kind of cognition?

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 12 Social Embedding Granovetter (1985) Contrasts with the under- and over-socialised models of behaviour That the particular patterns of social interactions between individuals matter In other words, only looking at individual behaviour or aggregate behaviour misses crucial aspects That the causes of behaviour might be spread throughout a society – “causal spread” Shown clearly in some simulation models

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 13 Illustration of Causal Complexity Lines indicate causal link in behaviour, each box an agent (Edmonds 1999)

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 14 Implications of Social Embedding In many circumstances agents can learn to exploit the computation and knowledge in their society, rather than use or reason with their own knowledge That is perceiving and acting through their network of social relationships Social embedding means that human behaviour can not be understood well separate from the surrounding social context This is not a problem for normal human action as human cognition automatically (and unconsciously) adapts to context However such normal action and cognition is hard to systematically justify or check

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 15 Some Researchers’ Responses to Context-Dependency A number of responses: Only do research within a single context, resisting any generalisation Only use discursive, natural language approaches where context is implicitly dealt with (and thus masked) Try to see what (inevitably weaker knowledge) is general across the various contexts in what is being studied

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 16 Some Policy-World Responses to Context-Dependency Wishing to know what is the “educated guess” as to which policy is best for most Using their detailed and informed intuition as to what the best policy is and looking for a more “scientific” justification for it Wanting a general narrative that “explains” (in comprehensible terms) why a certain policy direction is desirable or which can be used to build a consensus around

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 17 The “Simple is more General” Fallacy If one has a general model one can make it more specific (less general) by adding more processes/aspects… …in which case it can become more complex However, the reverse is no true… If one simplifies/abstracts then you don’t get a more general model (well almost never)! –there may be no simpler model that is good enough for your purpose –But, even if there is, you don’t know which aspects can be safely omitted – if you remove an essential aspect if will be wrong everywhere (no generality)

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 18 Context-Dependency and Randomness Lots of information lost if randomness used to “model” contextual variation

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 19 Context-Dependency and “Being Scientific” If the relevant context can be reliably indentified then… …context-dependency is not the same as subjectivity Generality is nice if you can get it, but its no good pretending to have it if you can’t Science should adapt to what it wishes to understand, not the other way around It does mean (often) an acceptance that general/generic approaches are not appropriate

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 20 What does not work… Inventing a model which only relates macro- level variables, not taking into account the social structures, using randomness to “cover” the variation across contexts (might as well use a straight line projection or random number generator!) Presenting a policy maker/advisor with a “warts and all” model/account that is so complicated it is hard to work out a coherent policy response

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 21 Some Suggestions Part 3:

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 22 Fund Ethnographic Observations of Researchers Interacting with Policy Makers/Advisors To “debug” what goes wrong (or right) with this relationship (randomly selected) projects need to be observed by a third party who… –Watches and makes notes about the interactions –Interviews the participants at various stages –Produces an impartial report reporting on the observations and conclusions (but anonymising the participants/project involved) –Participants get to add their thoughts on reading it as appended commentaries Periodically collections of such reports be publically released

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 23 Encouraging the development of multiple models Getting competing models from contrasting researchers for the same target Allowing/encouraging different models for different aspects of complex issues Encouraging the staging of abstraction from evidence to conclusions Making models publically accessible at an early stage (well before the project ends) to allow comment and/or criticism Models must be accompanied by documentation on: intended use, what is validated, whether replicated, etc.

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 24 Tools that could (used appropriately) help with context-dependency Multi-dimensional data in its original state (or near to), that is different kinds of data about the same set of actors/events Descriptive agent-based simulation where the different contexts agents might face are matched with different behavioural models Local Data-Mining which searches out patterns that hold but also what parts of the data they hold for (indicating context-dependency) Visualisation techniques which allow for researchers and stakeholders to explore, envisiage and check ideas about data Tools to aid the tracking of relations between multiple models and multiple data sets

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 25 Layering Reports on Research Structuring Policy Modelling Research Conclusions: 1.A general summary of the results and recommendations without caveats 2.A short summary of the reasons for the above in narrative form (non-technical language) 3.A risk analysis specifying how the above might fail, i.e. the contexts under which the recommended policies will work less well or not at all 4.A description of the kinds of situation where the policy will be most effective (to aid targeting of a policy) 5.A (perhaps lengthy) appendix with details, maybe broken down into the various cases, documenting assumptions, evaluation criteria etc.

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 26 Towards some Rules for Engagement Establish a set of rules/norms that should hold in policy  research interaction 1.Researchers accepting layered reporting in exchange for ensuring the risk-analysis does not get seperated from the summary advice 1&2 (which is what they fear) 2.An acceptance of context-specific solutions/models/specifications (or that general versions might not be applicable or effective) 3.Not asking for, nor offering, predictions of complex social systems, but rather accepting evaluations of possibilities 4.Not asking for impossible things from researchers (e.g. “general solutions for complexity”) 5.Researchers being honest about their limitations and admitting when they fail (post hoc evaluation)

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 27 Writing “User Manuals” for Policy Advisors and Researchers Two Manuals: 1.A manual written for academics/researchers for how to deal with and communicate with policy adivsors/makers 2.Another written for policy makers/advisors of what to expect from academics/researchers, how to effectively use them and their shortcomings (as might be seen from the policy world)

Context-dependency, risk analysis and policy modelling, Bruce Edmonds, ’2012, Brussels, September 2012, slide 28 The End Bruce Edmonds Centre for Policy Modelling Manchester Metropolitan University