1. Bayesian networks: opportunities and challenges in the law
27 September 2016
Norman Fenton
Queen Mary University of London and Agena Ltd
Having already presented two informal seminars and one formal seminar at the FOS programme so far, I decided to wait until the last possible moment to prepare some material I have not already presented before. I do however, make the same restriction I made during my talk at the first workshop, namely that the kind of BN models I am talking about here are those which attempt to model – at a high level – the overall evidence of a case and not specific detailed forensic evidence. So this fits more with the kind of example presented by Phil Dawid, rather than, say that presented yesterday by Julia Mortera or Peter Vergeer. I would regard those as more detailed components of the kind of models I am talking about.

2. Overview Opportunities (i.e. benefits) for BNs in the law
Understanding evidence
Defeating objections to use of probability in the law
Handling ‘surprise evidence’
Challenges (i.e. difficult to resolve issues) for BN in the law
Getting ‘the numbers’
Incorporating motive and opportunity
Integrating alternative, conflicting narratives
Communicating results to lay people
So informed by a lot of very recent discussions I am going to review the opportunities and challenges of using BNs in the law.

3. Opportunities for BNs and the law
Natural formalisation of actual legal practice and reasoning
Even without the numbers developing a BN helps us understand a case and its evidence
Can help determine whether a case has sufficient evidence to go forward
Expose fundamental weaknesses and problems with the over-simplistic LR method
Can help lawyers better understand and then explain the impact of evidence when there are complex dependencies
Can help ‘defeat’ objections to using probability and Bayes in the law
Handles impact of ‘novel’ and ‘surprise’ evidence
Stuff not in bold is what I covered previously. Picking out stuff in bold to discuss here.

4. Understanding evidence
H0: Defendant guilty (y/n)
H1: Defendant was at scene (y/n)
H2: Defendant is source of DNA at scene (y/n)
E’: Expert credibility
W’: Witness credibility
E: Defendant DNA ‘matches’ DNA at scene (y/n)
H3: Defendant DNA ‘matches’ DNA at scene (y/n)
Yesterday all speakers implicitly or explicitly in the case of Phil Dawid, Paulo Garboloni and Giulio Dagostini made the key point that evidence of a fact did not make the fact true but rather depended on the credibility of the witness asserting the fact. One of the most common errors we have found in people building BNs is to assume that evidence or measure represents the truth about an unknown hypothesis.
This is a simplification because credibility needs to be expanded in the way Paolo suggested.
W: Witness testifies that defendant was at scene
E: Expert testifies defendant DNA ‘matches’ DNA at scene (y/n)

5. Collins Case
When you put it all together. Of course a critical problem raised by Henry Prakken yesterday was how do we arrive at such a model and why should we believe it especially as two different BN modellers would likely come up with completely different models? One of the themes of our research, especially idioms stuff is to try to ensure that there is a structured, repeatable method of building such models.

6. Handling surprise evidence (1)
Problem posed by Anne Ruth Mackor at first FOS Workshop
(‘novel facts’)
A defendant is accused of murdering his wife. He offers no defence other than to say it was not him he killed her. In classic case of the evidence accuracy idiom we therefore have this model. But there is no other evidence to support his denial.
However, at some time later on DNA of an unknown person is found on the wife. How do we interpret this evidence (thanks to Dave L and Bill Thompson for this).

7. Handling surprise evidence (2)
The impact of this new surprise evidence (novel fact) critically depends on whether or not the defendant knew about the possibility such evidence might be found. For example, if the defendant knew that his wife had intercourse with a lover on the morning of her murder, then that provided him with an excuse for his denial.

8. Handling surprise evidence (3)
Initial marginal probabilities
Her is the model in it initial state. Note uniform prior assumptions for credible and guilty. But low prior assumption about defendant knowing of surprise evidence.

9. Handling surprise evidence (4)
Defendant denial
Defendant denial decreases the probability of guilt but tells us nothing about his credibility.

10. Handling surprise evidence (5)
Later DNA evidence
Drastic change when the new evidence is added. Not only are we near convinced of his innocence, but we also believe him likely to be credible.

11. Handling surprise evidence (6)
Very different impact if he knew possibility of such evidence
However, everything changes if we have reason to believe he know of the possibility of other evidence. OK he is still less likely to be guilty than the starting point (because we now have another real suspect to consider) but there is a reasonable probability that suspect is unconnected with the murder.

12. Resolving objections to using probability and Bayes in the law
Cohen/Pardo paradox claim
Civil case C: conjunction of independent claims A, B with witnesses assumed “70% reliable”
So fact finder “determines” P(A)=P(B)=0.7
Individually each fact meets 0.5 threshold for P(C)
But the conjunction of the facts does not since P(A)*P(B)= 0.49
“Each witness clearly offered positive support for the plaintiff. How can it be that their combined support undermines his case? Hence ‘Pascalian’ approach fails.”
Pardo MS, “The Nature and Purpose of Evidence Theory”, Vanderbilt law Review 66(2) , 2013
Cohen LJ: “The Probable and the Provable”, 1977
Dawid, A.P., The Difficulty About Conjunction. J. R. Stat. Soc. Ser. D (The Stat.) 36, 91–97.

13. ‘Paradox’ resolved (1)

14. ‘Paradox’ resolved (2)
There is a paper by Pardo that lists dozens of these examples that supposedly demonstrate that Bayesian reasoning is inappropriate for legal reasoning. In every case it is because the necessary components of a BN model have not been articulated.

15. Challenges for BNs and the law
Getting ‘the numbers’
The prior and background information problem
Incorporating motive and opportunity
Do we model what the lawyers present or what we think they should present
Integrating alternative, conflicting narratives
Communicating the model and its results to lay people
Do we model what lawyers present or what we think they should present inevitably brings us to the issue of integrating alternative conflicting narratives

16. Getting the numbers H0: Defendant guilty (y/n)
H1: Defendant was at scene (y/n)
H2: Defendant is source of DNA at scene (y/n)
E’: Expert credibility
W’: Witness credibility
E: Defendant DNA ‘matches’ DNA at scene (y/n)
H3: Defendant DNA ‘matches’ DNA at scene (y/n)
H3 – the classic LR stuff of DNA experts.
E is exactly what paolo garbolino presented yesterday.
But the difficult stuff is H1 and H0.
W: Witness testifies that defendant was at scene
E: Expert testifies defendant DNA ‘matches’ DNA at scene (y/n)

17. Opportunity and motive

18. Integrating conflicting narratives (1)

19. Integrating conflicting narratives (2)

20. Integrating conflicting narratives (3)

21. Constraint node (only partial) solutionCn ….
auxiliary
constraint
True
False c1 c2 … cn NA
Fenton, Neil, Lagnado, Marsh, Yet, Constantinou, “How to model mutually exclusive events based on independent causal pathways in Bayesian network models”, Knowledge Based Systems,
Sept 2016,

22. Constraint node (only partial) solution

23. Constraint node (only partial) solution

24. Communicating results: Combining argumentation/scenario approach
P(S1)=0.36
P(Guilt|S1)=0.95
P(S2)=0.01
P(Guilt|S2)=0.01

25. Misplaced optimism?
“I assert that we now have a technology that is ready for use, not just by the scholars of evidence, but by trial lawyers.”
Edwards, W. (1991). "Influence Diagrams, Bayesian Imperialism, and the Collins case: an appeal to reason." Cardozo Law Review 13:

26. Summary
Fact finders ultimately use an ‘internalised BN’ when they arrive at their decision
BNs make explicit what is otherwise hidden – in particular assumptions about multiple unknown hypotheses
BNs make explicit the difference between the truth of an unknown hypothesis and reports about its truth
BNs correctly compute overall results of all assumptions and evidence – can’t be done manually
BNs not ready for presentation in court, but commonly cited objections are being addressed
I believe that in 50 years time professionals of all types involved in the legal system will look back in total disbelief that they could have ignored these available techniques of reasoning about evidence for so long. 26

27. Follow up Probability and Statistics in Forensic Science
Get free BN software
18 July – 21 Dec 2016
ERC Project BAYES-KNOWLEDGE
probabilityandlaw.blogspot.co.uk/