1. Henry Prakken Guangzhou (China) 12 April 2018
Spring School on Argumentation in AI & Law Day 3 – lecture 2: Three approaches to rational proof in criminal cases
Henry Prakken
Guangzhou (China)
12 April 2018

2. Uncertainty in legal proof in criminal cases
Legal proof of facts is never completely certain
Eye witnesses can be unreliable
Expert witnesses sometimes disagree
Documents can be manipulated
Generalisations can have exceptions
DNA tests have an error margin
Confessions might be false …

3. Models of proof that leave room for uncertainty
Bayesian models
Determine the prior probability of guilt
Determine how probable the evidence is given guilt and given innocence
Then apply Bayes’ theorem to determine how probable guilt is given the evidence
Argumentation-based models
Construct arguments for and against guilt, starting with the available evidence
Determine which argument survives the competition between the conflicting arguments.
Story-based models
Construct stories about what might have happened
Choose the best
The most coherent, the best explanation of the evidence
All approaches have their strengths and weaknesses, in many cases they should be combined
Floris Bex studied the combination of story- and argumentation-based models
Charlotte Vlek studied the combination of Bayesian with story-based models,
And Sjoerd Timmer studied the combination of Bayesian with argumentation-based models

4. Problem We want a model of rational proof that is:
rationally well-founded
cognitively feasible

5. Rational legal proof with arguments
F.J. Bex, H. Prakken, C. Reed & D. Walton (2003), Towards a formal account
of reasoning about evidence: argumentation schemes and generalisations.
Artificial Intelligence and Law 11: (2003)

6. Wigmore Charts
This goes back to Wigmore, then the Neo-Wigmoreans. And independently AI & Law.

7. Reasoning with generalisations
Involved
Defeasible Modus Ponens: P
If P then normally Q
So (presumably), Q
Critical questions:
Is there an exception?
Are there conflicting generalisations?
Flees
If flees then normally involved
People who flee from a crime scene are normally involved in the crime

8. Implicit generalisation
“X entered the office at 4:30 since witness W says that he saw X entering the office at 4:30”
X entered the office at 4:30
A key notion in argumentation models of evidential reasoning is that of a generalisation. They are the reason why the other premises support the conclusion.
W says that he saw X entering the office at 4:30

9. Implicit generalisation made explicit
“X entered the office at 4:30 since witness W says that he saw X entering the office at 4:30”
Unless W’s memory is flawed
Unless W could not observe well
X entered the office at 4:30
Unless W has a reason to lie
W says that he saw X entering the office at 4:30
What witnesses say is usually true

10. Witness testimony W says P W was in the position to observe P
Critical questions:
Is W sincere?
Does W’s memory function properly?
Did W’s senses function properly?
W says P
W was in the position to observe P
Therefore (presumably), P
P is usually of the form
“I remember that I
observed that ...”

11. Implicit generalisation (2)
“X was in the office at 4:45 since he entered the office at 4:30”
X was in the office at 4:45
X entered the office at 4:30

12. Implicit generalisation made explicit
Unless X has meanwhile left the office
“X was in the office at 4:45 since he entered the office at 4:30”
X was in the office at 4:45
X entered the office at 4:30
People who enter a building are usually still inside a short while later

13. Temporal persistence (Forward)
Critical questions:
Was P known to be false between T1 and T2? …
P is true at T1
Therefore (presumably), P is
still true at later time T2

14. Temporal persistence (Backward)
Critical questions:
Was P known to be false between T1 and T2? …
P is true at T1
Therefore (presumably), P was already true at earlier time T2

15. If V is killed at L at time T, and S was there then, then S killed V
X killed Y
Generalisation
If V is killed at L at time T, and S was there then, then S killed V
Y was killed in the office at 4:45
X was in the office at 4:45
Forward temporal persistence
Backward temporal persistence
X entered the office at 4:30
X left the office at 5:00
This is an example from William Twining, meant to explain the Neo-Wigmorean argumentation approach and the importance of exposing generalisations as a source of doubt. The argument is clearly not deductive, but it is still rational.
testimony
testimony
W1: “X entered the office at 4:30”
W2: “X entered the office at 4:30”
W3: “X left the office at 5:00”

16. How are generalisations justified?
Scientific research (induction)
Experts
Commonsense
Individual opinions
Prejudice?
Very reliable
Very unreliable
Example “young male lovers tend to dominate / be dominated by their older female lover”. 16

17. Inducing generalisations
Almost all observed P’s were Q’s
Therefore (presumably), If P then usually Q
Critical questions:
Is the size of the sample large enough?
was the sample selection biased?
A ballpoint shot with this type of bow will usually cause this type of eye injury
In 16 of 17 tests the ballpoint shot with this bow caused this type of eye injury 17

18. Expert testimony E is expert on D E says that P P is within D
Therefore (presumably), P is the case
Critical questions:
Is E biased?
Is P consistent with what other experts say?
Is P consistent with known evidence?

19. Supporting and using generalisations
scheme
V’s injury was caused by a fall
This type of eye injury is usually caused by a fall
V has this type of injury
E says that his type of injury is usually caused by a fall
E is an expert on this type of injury
Expert testimony scheme
Suppose there is no expert evidence: then the generalisation is unsupported. An analysis like this can expose this.
This example shows why generalisations must be premises.

20. Rational legal proof with stories/scenarios
Now I briefly introduce the story-based approach, in order to compare how it would model the same example.

21. Pennington & Hastie (1993)
Judges and jurors can only understand evidential reasoning in the form of stories about what may have happened.
A good story:
Explains the evidence
Is coherent:
Is internally consistent
Is plausible (conforms to our general knowledge about the world)
Is complete: contains initiating events which cause the main actor to have intentions, which give rise to actions, which have consequences

22. Argumentation vs. scenario approach
Argumentation: reasons from evidence to hypotheses by applying evidential generalisations / argument schemes
Critical testing by searching for counterarguments based on exceptions
Scenario approach: assumes the hypotheses and then tests whether the evidence (likely) follows by applying causal generalisations
Critical testing by checking under which hypothesis the evidence most likely follows

23. Evidential vs. causal generalisations
Evidential: P is evidence of Q (smoke means fire)
Causal: P causes Q (fire causes smoke)
This is a key distinction in contrasting the argumentation approach with the story-based (and also the Bayesian) approach.

24. Y was killed in the office at 4:45
X entered the office at 4:30
X met Y at 4:45
X killed Y at 4:45
X left the office at 5:00
X wanted to kill Y
First scenario: X killed Y.
time 24

25. W1: “X entered the office at 4:30” W2: “X entered the office at 4:30”
Y was killed in the office at 4:45
W3: “X left the office at 5:00”
X entered the office at 4:30
X met Y at 4:45
X killed Y at 4:45
X left the office at 5:00
X wanted to kill Y
First scenario: X kiled Y. This scenario explains all evidence.
time 25

26. W1: “X entered the office at 4:30” W2: “X entered the office at 4:30”
Y was killed in the office at 4:45
W3: “X left the office at 5:00”
X met Z at 4:45
X entered the office at 4:30
X met Y at 4:45
X killed Y at 4:45
X left the office at 5:00
X wanted to kill Y
Second scenario: X wanted to visit a friend. This scenario only explains the witness testimonies.
We can never convict on the basis of just one scenario, since logically speaking alternative explanations of the evidence are always possible.
Either choose now or find more evidence: (for a motive to kill or for X meeting Z in the office. (prediction!)).
X wanted to visit his friend Z
time 26

27. Example: Adams v Regina
A rape took place near London (UK) in In 1993 Denis John Adams was arrested for another offence and a routine check showed that his DNA matched with that of a sample of semen obtained from the victim of the 1991 rape.
The prosecution’s forensic expert estimated the random match probability as 1 in 200 million; the defence thought that 1 in 2 million was a better estimate.
A line up took place but the victim did not recognize Adams, and she said he did not resemble her attacker. Adams was 37 and looked older, while the victim claimed the rapist was in his early twenties.
Adams’ girlfriend testified that he had spent the night of the attack with her.

28. Arguments and counterarguments
John is the rapist
John’s DNA matches with DNA found with the victim
Back to the argumentation approach, to illustrate how it compares alternatives and draws conclusions. The slogan is: an argument on its owndoes not prove anything.
For simplicity I leave the generalisations implicit.
The forensic scientist’s report says so
Regina v. Adams 1995 28

29. John’s DNA matches with DNA found with the victim
John is the rapist
John is not the rapist
John’s DNA matches with DNA found with the victim
John does not look like the rapist
The forensic scientist’s report says so
Victim: “John does not look like the rapist” 29 29

30. The a priori prob that John is the rapist is not too low
John is not the rapist
The a priori prob that John is the rapist is not too low
John was elsewhere
John does not look like the rapist
Mary has reason to lie
Argumentation logics deal with this kind of indirect interactions of attack and defence. There is a huge formal-logical literature on this, but for present purposes this basic pattern of ‘reinstatement’ suffices.
The forensic scientist’s report says so
Victim: “John does not look like the rapist”
Mary: “John was with me”
Mary is John’s girlfriend 30 30

31. Legal proof with arguments: critical questions
Which evidential generalisations are the glue in the arguments?
Do they hold in general?
If so: are there exceptions in this case?
Critical questions of argument schemes can help
Are there counterarguments on other grounds?
Can counterarguments (if any) be refuted?
This is a summary of the argumentation approach. An argument on its own does not prove anything.
Aben (2014) argued that this is by-and-large how Dutch judges decide cases. He proposed to replace it with a model of ‘scientific proof’, which would be the rational approach.
But is legal proof always like scientific proof? Often it is commonsense reasoning.

32. Scenario construction again
John’s DNA matches with DNA found with the victim
John is the rapist
Mary is John’s girlfriend
Mary: “John was with me”
Victim: “John does not look like the rapist”
How does the scenario-based approach model the Adams case? 32 32

33. Scenario construction
John’s DNA matches with DNA found with the victim
Mary: “John was with me”
Someone else with the same DNA profile raped the victim
Victim: “John does not look like the rapist”
John does not look like the rapist
John was with Mary 33 33

34. Scenario construction
John’s DNA matches with DNA found with the victim
John is the rapist
Mary is John’s girlfriend
Mary: “John was with me”
Someone else with the same DNA profile raped the victim
Victim: “John does not look like the rapist”
Van Koppen in fact asks what is the likelihood ratio of the evidence?
John does not look like the rapist
Van Koppen (2011):
Under which scenario is the evidence the most likely?
John was with Mary 34 34

35. Legal proof with scenarios: critical questions
Is the scenario plausible, consistent and complete?
Does it explain the evidence?
Are there alternative scenarios that explain the evidence?
If so:
How plausible are the various scenarios?
how likely is the evidence given the various scenarios?
A scenario can also be criticised if there are no alternative scneario’s.
But convicting without even considering alternative scenario’s is irrational (fallacy of affirming the consequent).
A weakness of the scenario approach is that it does not tell us how we should combine relative plausibility and likelihood. Here is where probability theory becomes relevant.

36. Rational legal proof with Bayesian probability theory
This this approach like the story-based approach reasons indirectly from the hypotheses to the evidence (and then back).

37. Probability theory In logic statements are true or false
Uncertainty expressed as defeasibility
In probability theory statements have a probability between 0 and 1 (or between 0% and 100%)
‘The probability of P is 1 (100%)’ means: P is certainly true
‘The probability of P is 0 (0%)’ means: P is certainly false
The probabilities of P and not-P add up to 1 (to 100%).
So the probability of not-P is 1 minus the probability of P (or 100% minus the probability of P)
Notation: ‘Prob’ = probability.
Probabilities can be conditional: the probability of Q given P

38. The main idea Determine prior probability of guilt
Determine how probable the evidence is given guilt and given innocence
Then apply Bayes’ theorem to determine how probable guilt is given the evidence

39. Bayes’ theorem (odds version)
If greater than 1, then E is incriminating evidence,
If less than 1, then E is exculpating evidence, otherwise E is irrelevant.
Bayes’ theorem (odds version)
The prob of G given E
The prob of not-G given E
The prob of E given G
The prob of E given not-G
The prob of G
The prob of not-G = x
Then the computer computes the posterior prob of G given E
Determine or ask an expert to determine the likelihood ratio of E wrt G and not-G
Determine the prior prob of G
Posterior
odds
Likelihood
ratio
Prior
odds = x
G = guilty
Not-G = innocent

40. Example: Adams v Regina
Evidence pro guilt:
The DNA match
Evidence con guilt:
Victim did not recognize Adams during a line up
Alibi of Adam’s girlfiend

41. ‘Prosecutor fallacy’: inverting conditional probabilities
“The probability that Adams’ DNA matches the rapist’s DNA given that Adams is not rapist is very small”
Is NOT the same as
“The probability that Adams is not the rapist given that Adams’ DNA matches the rapist’s DNA is very small”
Compare
“The probability that a person is a man given that the person is a rapist is very high”
“The probability that a persons is a rapist given that the person is a man is NOT very high”

42. Bayesian Reasoning = x = x
E: match of Adams DNA with DNA found at the crime scene
G: John is the rapist
Bayesian Reasoning
So the prob that Adams is the donor given the DNA match is 50%
The prob of G given E
The prob of not-G given E
The prob of E given G
The prob of E given not-G = x
The prob of G
The prob of not-G 1 =
2,000,000 x
1:2,000,000
Random match probability = 1:2,
2,000,000 potential
donors
Dawid, Philip (2005). Probability and proof. Online appendix to T.J. Anderson, D.A. Schum and W.L. Twining: Analysis of Evidence,

43. Bayesian Reasoning = x = x
E: match of Adams DNA with DNA found at the crime scene
G: John is the rapist
Bayesian Reasoning
So the prob that Adams is the rapist given the DNA match is 0.91
The prob of G given E
The prob of not-G given E
The prob of E given G
The prob of E given not-G = x
The prob of G
The prob of not-G 10 =
2,000,000 x
1:200,000
Random match probability = 1:2,
200,000 potential
donors
Dawid, Philip (2005). Probability and proof. Online appendix to T.J. Anderson, D.A. Schum and W.L. Twining: Analysis of Evidence,

44. Bayes with multiple pieces of evidence (1)
Repeat the calculation with the old posterior as the new prior
= successively multiply the prior with the likelihood ratio of every piece of evidence

45. Bayes with multiple pieces of evidence (2)=
Posterior odds
Prior odds x
Likelihood ratio
evidence 1 x
Likelihood ratio
evidence 2 x
Likelihood ratio
evidence 3

46. Bayes with multiple pieces of evidence in Adams (1)
Prob of Adams’ guilt given E1&E2&E3
Prob of Adams’ innocence given E1&E2&E3 =
The prior odds of Adams’ guilt was 1 in x
The likelihood ratio of the DNA match for Adams’ guilt was x
The likelihood ratio of the non recognition for Adams’ guilt was ??
E1 = DNA match,
E2 = non recognition,
E3 = girlfriend’s alibi x
The likelihood ratio of the girlfriend’s alibi for Adams’ guilt was ??

47. Bayes with multiple pieces of evidence in Adams (2)
Prob of Adams’ guilt given E1&E2&E3
Prob of Adams’ innocence given E1&E2&E3 =
The prior odds of Adams’ guilt was 1 in 1/
91% x
The likelihood ratio of the DNA match for Adams’ guilt was 10 x
53%
The likelihood ratio of the non recognition for Adams’ guilt was 1/9
10/9
E1 = DNA match,
E2 = non recognition,
E3 = girlfriend’s alibi
36% x
The likelihood ratio of the girlfriend’s alibi for Adams’ guilt was 1/2
5/9

48. Limitations of ‘naive’ Bayes
Repeated updating (or multiplying likelihood ratios) is only justified if the evidence is statistically independent.
Otherwise Bayesian networks are needed
Graphically display statistical dependencies
Can reduce the number of probabilities to be estimated
Their graphical structure can perhaps capture scenarios
Indepedence in this context means that learning another piece of evidence does not change the likelihood ratios of the already considered pieces of evidence.

49. BN met klikken

50. Legal proof with Bayes: critical questions
To which extent is the evidence statistically independent?
Can the prior probabilities be reasonably estimated?
Are the conditional probabilities well-founded?
Are the considered alternatives jointly exhaustive? …
All these questions also arise in some way in the other aproaches.

51. The three approaches: commonalities
All three approaches can be given mathematical foundations
All approaches acknowledge that legal proof can never give full certainty
All approaches compare alternatives
In all approaches new evidence can change the outcome
In all approaches generalisations are important

52. The three approaches: differences (1)
Argumentation and scenario construction are qualitative, probability theory quantifies uncertainty
Do numbers have excessive persuasive force?
Or does quantifying uncertainty make judges too cautious?

53. The three approaches: differences (2)
Argumentation reasons directly from evidence to hypotheses, the other two reason indirectly from hypotheses to evidence (and then back to the hypotheses)
But why does the story-based approach not take priors into account?

54. The three approaches: differences (3)
Argumentation and scenario approach(?) can handle inconsistency, Bayes cannot.

55. My favourite approach
There is no single general model of rational legal proof
A toolbox or hybrid approach is needed
Start with constructing alternative scenario’s
Then zoom in on details with argumentation or Bayes
Rationally well-founded? Bayes is the best? But the others also have foundations.
Cognitively feasible? Stories is the best? But judges handle certain types of evidence, e.g. witnesses, in an argumentative fashion.

56. Combinations Argumentation + stories Argumentation + Bayes
Floris Bex (2011), Arguments, Stories and Criminal Evidence: A Formal Hybrid Theory. Springer, Dordrecht.
Argumentation + Bayes
Sjoerd Timmer et al., A two-phase method for extracting explanatory arguments from Bayesian networks. International Journal of Approximate Reasoning 80 (2017):
Stories + Bayes
Charlotte Vlek et al., A method for explaining Bayesian networks for legal evidence with scenarios. Artificial Intelligence and Law 24 (2016):