Bayesian networks: opportunities and challenges in the law 27 September 2016 Norman Fenton Queen Mary University of London and Agena Ltd Having already presented two informal seminars and one formal seminar at the FOS programme so far, I decided to wait until the last possible moment to prepare some material I have not already presented before. I do however, make the same restriction I made during my talk at the first workshop, namely that the kind of BN models I am talking about here are those which attempt to model – at a high level – the overall evidence of a case and not specific detailed forensic evidence. So this fits more with the kind of example presented by Phil Dawid, rather than, say that presented yesterday by Julia Mortera or Peter Vergeer. I would regard those as more detailed components of the kind of models I am talking about.

Overview Opportunities (i.e. benefits) for BNs in the law Understanding evidence Defeating objections to use of probability in the law Handling ‘surprise evidence’ Challenges (i.e. difficult to resolve issues) for BN in the law Getting ‘the numbers’ Incorporating motive and opportunity Integrating alternative, conflicting narratives Communicating results to lay people So informed by a lot of very recent discussions I am going to review the opportunities and challenges of using BNs in the law.

Opportunities for BNs and the law Natural formalisation of actual legal practice and reasoning Even without the numbers developing a BN helps us understand a case and its evidence Can help determine whether a case has sufficient evidence to go forward Expose fundamental weaknesses and problems with the over-simplistic LR method Can help lawyers better understand and then explain the impact of evidence when there are complex dependencies Can help ‘defeat’ objections to using probability and Bayes in the law Handles impact of ‘novel’ and ‘surprise’ evidence Stuff not in bold is what I covered previously. Picking out stuff in bold to discuss here.

Understanding evidence H0: Defendant guilty (y/n) H1: Defendant was at scene (y/n) H2: Defendant is source of DNA at scene (y/n) E’: Expert credibility W’: Witness credibility E: Defendant DNA ‘matches’ DNA at scene (y/n) H3: Defendant DNA ‘matches’ DNA at scene (y/n) Yesterday all speakers implicitly or explicitly in the case of Phil Dawid, Paulo Garboloni and Giulio Dagostini made the key point that evidence of a fact did not make the fact true but rather depended on the credibility of the witness asserting the fact. One of the most common errors we have found in people building BNs is to assume that evidence or measure represents the truth about an unknown hypothesis. This is a simplification because credibility needs to be expanded in the way Paolo suggested. W: Witness testifies that defendant was at scene E: Expert testifies defendant DNA ‘matches’ DNA at scene (y/n)

Collins Case When you put it all together. Of course a critical problem raised by Henry Prakken yesterday was how do we arrive at such a model and why should we believe it especially as two different BN modellers would likely come up with completely different models? One of the themes of our research, especially idioms stuff is to try to ensure that there is a structured, repeatable method of building such models.

Handling surprise evidence (1) Problem posed by Anne Ruth Mackor at first FOS Workshop (‘novel facts’) A defendant is accused of murdering his wife. He offers no defence other than to say it was not him he killed her. In classic case of the evidence accuracy idiom we therefore have this model. But there is no other evidence to support his denial. However, at some time later on DNA of an unknown person is found on the wife. How do we interpret this evidence (thanks to Dave L and Bill Thompson for this).

Handling surprise evidence (2) The impact of this new surprise evidence (novel fact) critically depends on whether or not the defendant knew about the possibility such evidence might be found. For example, if the defendant knew that his wife had intercourse with a lover on the morning of her murder, then that provided him with an excuse for his denial.

Handling surprise evidence (3) Initial marginal probabilities Her is the model in it initial state. Note uniform prior assumptions for credible and guilty. But low prior assumption about defendant knowing of surprise evidence.

Handling surprise evidence (4) Defendant denial Defendant denial decreases the probability of guilt but tells us nothing about his credibility.

Handling surprise evidence (5) Later DNA evidence Drastic change when the new evidence is added. Not only are we near convinced of his innocence, but we also believe him likely to be credible.

Handling surprise evidence (6) Very different impact if he knew possibility of such evidence However, everything changes if we have reason to believe he know of the possibility of other evidence. OK he is still less likely to be guilty than the starting point (because we now have another real suspect to consider) but there is a reasonable probability that suspect is unconnected with the murder.

Resolving objections to using probability and Bayes in the law Cohen/Pardo paradox claim Civil case C: conjunction of independent claims A, B with witnesses assumed “70% reliable” So fact finder “determines” P(A)=P(B)=0.7 Individually each fact meets 0.5 threshold for P(C) But the conjunction of the facts does not since P(A)*P(B)= 0.49 “Each witness clearly offered positive support for the plaintiff. How can it be that their combined support undermines his case? Hence ‘Pascalian’ approach fails.” Pardo MS, “The Nature and Purpose of Evidence Theory”, Vanderbilt law Review 66(2) 547-613, 2013 Cohen LJ: “The Probable and the Provable”, 1977 Dawid, A.P., 1987. The Difficulty About Conjunction. J. R. Stat. Soc. Ser. D (The Stat.) 36, 91–97.

‘Paradox’ resolved (1)

‘Paradox’ resolved (2) There is a paper by Pardo that lists dozens of these examples that supposedly demonstrate that Bayesian reasoning is inappropriate for legal reasoning. In every case it is because the necessary components of a BN model have not been articulated.

Challenges for BNs and the law Getting ‘the numbers’ The prior and background information problem Incorporating motive and opportunity Do we model what the lawyers present or what we think they should present Integrating alternative, conflicting narratives Communicating the model and its results to lay people Do we model what lawyers present or what we think they should present inevitably brings us to the issue of integrating alternative conflicting narratives

Getting the numbers H0: Defendant guilty (y/n) H1: Defendant was at scene (y/n) H2: Defendant is source of DNA at scene (y/n) E’: Expert credibility W’: Witness credibility E: Defendant DNA ‘matches’ DNA at scene (y/n) H3: Defendant DNA ‘matches’ DNA at scene (y/n) H3 – the classic LR stuff of DNA experts. E is exactly what paolo garbolino presented yesterday. But the difficult stuff is H1 and H0. W: Witness testifies that defendant was at scene E: Expert testifies defendant DNA ‘matches’ DNA at scene (y/n)

Opportunity and motive

Integrating conflicting narratives (1)

Integrating conflicting narratives (2)

Integrating conflicting narratives (3)

Constraint node (only partial) solution Cn …. auxiliary constraint True False c1 c2 … cn NA Fenton, Neil, Lagnado, Marsh, Yet, Constantinou, “How to model mutually exclusive events based on independent causal pathways in Bayesian network models”, Knowledge Based Systems, Sept 2016, http://dx.doi.org/10.1016/j.knosys.2016.09.012

Constraint node (only partial) solution

Constraint node (only partial) solution

Communicating results: Combining argumentation/scenario approach P(S1)=0.36 P(Guilt|S1)=0.95 P(S2)=0.01 P(Guilt|S2)=0.01

Misplaced optimism? “I assert that we now have a technology that is ready for use, not just by the scholars of evidence, but by trial lawyers.” Edwards, W. (1991). "Influence Diagrams, Bayesian Imperialism, and the Collins case: an appeal to reason." Cardozo Law Review 13: 1025-107

Summary Fact finders ultimately use an ‘internalised BN’ when they arrive at their decision BNs make explicit what is otherwise hidden – in particular assumptions about multiple unknown hypotheses BNs make explicit the difference between the truth of an unknown hypothesis and reports about its truth BNs correctly compute overall results of all assumptions and evidence – can’t be done manually BNs not ready for presentation in court, but commonly cited objections are being addressed I believe that in 50 years time professionals of all types involved in the legal system will look back in total disbelief that they could have ignored these available techniques of reasoning about evidence for so long. 26

Follow up Probability and Statistics in Forensic Science Get free BN software www.AgenaRisk.com 18 July – 21 Dec 2016 www.newton.ac.uk/event/fos www.BayesianRisk.com ERC Project BAYES-KNOWLEDGE http://bayes-knowledge.com/ probabilityandlaw.blogspot.co.uk/