Basic Concepts of Evidentiary Reasoning

1
Basic Concepts of Evidentiary Reasoning

• Robert J. Mislevy
• University of Maryland
• February 5, 2001

2

• We live in an age when we are still more adept
  at gathering, transmitting, storing, and
  retrieving information than we are at putting
  this information to use in drawing conclusions
  from it.
• Kadane Schum, 1996, p. xiv

3
Data versus evidence

• A datum becomes evidence in some analytic problem
  when its relevance to conjectures being
  considered is established. (Schum)
  
• Evidence is relevant on some conjecture if it
  either increases or decreases the likeliness of
  the hypothesis.
  
• Conjectures, and the understanding of what
  constitutes evidence about them, emanate from the
  variables, concepts, and relationships of the
  domain.

4
What inference is (1)

• Inference is reasoning from what we know and what
  we observe to explanations, conclusions, or
  predictions.
  
• Troubleshooting
• Medical diagnosis
• History
• Law
• Probability and Statistics

5
What inference is (2)

• We always reason in the presence of uncertainty.
  
  
• Evidence is almost always
• Incomplete
• Inconclusive
• Amenable to multiple explanations

6
Credentials of Evidence

• Evidence has three major properties that must be
  established
  
• relevance
• credibility
• inferential force
• (Kadane Schum, 1996)

7
An Example concerning the Gettysburg Address

• Gary Wills (1992) Lincoln at Gettysburg is
  mainly an analysis of what Lincoln meant when he
  presented the Gettysburg Address, but its
  Appendix I explores what he actually said. Five
  versions in Lincolns hand and four newspaper
  transcriptions survive. Unanimity about a phrase
  suggests he spoke it as such, but for
  discrepancies Wills must consider such clues as
  these The draft Lincolns secretary claimed he
  saw Lincoln speak from appears on Executive
  Mansion letterhead, corroborating eyewitness
  accounts, but it omits key phrases all newspapers
  report and garbles the transition between pages.
  (Mislevy, 1994)

8
Examples concerning the Kennedy Assassination

• How many shots were fired at Dealey Plaza?
  Estimates at the scene ranged from one to eight.
  However, on this issue, there was more agreement
  than on any other post-assassination matter. Of
  the nearly two hundred witnesses over 88
  percent heard three shots. Although almost
  every conspiracy theory proposes that more than
  one assassin relies on there having been four or
  more shots, the writers seldom disclose that
  fewer than one in twenty witnesses heard that
  many.
• Gerald Posner (1993, p. 236)

9
Examples concerning the Kennedy Assassination

• Failure Analysis Associates probability cones
  (Figure 1) for sources of shots in the Kennedy
  assassination extend uncertainties in positions
  and angles backwards from the points of impact
  (Posner, 1993, p. 476).

10
Concepts from Stephen Toulmin

• Claim
• Conjecture, hypothesis, facta probanda
• Data
• Observation evidence facta probans
• Warrant
• Why observation should change our belief
• Qualifiers
• Caveats, exception conditions
• Rebuttal data
• Clues an exception condition might hold

11
Claim Data
C
so
D
12
Claim Data
Harry is a British subject.
so
Harry was born in Bermuda.
Example from Toulmin, 1958
13
Claim, Data, Warrant
C
since
W
so
D
14
Claim, Data, Warrant
Harry is a British subject.
A person born in Bermuda is a British subject.
since
so
Harry was born in Bermuda.
Example from Toulmin, 1958
15
Backing the Warrant
C
since
W
on
account
so
of
B
D
16
Backing the Warrant
Harry is a British subject.
A person born in Bermuda is a British subject.
since
on
account
so
of
Harry was born in Bermuda.
The British Nationalities Act, which states...
Example from Toulmin, 1958
17
Qualifiers
C
unless
A
since
W
on
account
so
of
B
D
18
Qualifiers
Harry is a British subject.
unless
both parents are not British subjects.
A person born in Bermuda is a British subject.
since
on
account
so
of
Harry was born in Bermuda.
The British Nationalities Act, which states...
Example from Toulmin, 1958
19
Qualifiers
Harry is a British subject.
unless
both parents are aliens
unless
A person born in Bermuda is a British subject.
since
One becomes a naturalized citizen of another
country
on
account
so
of
Harry was born in Bermuda.
The British Nationalities Act, which states...
Example from Toulmin, 1958
20
Rebuttal
C
unless
A
since
W
on
account
so
of
supports
R
B
D
21
Rebuttal
Harry is a British subject.
unless
both parents are aliens
A person born in Bermuda is a British subject.
since
on
account
so
supports
of
Harry was born in Bermuda.
The British Nationalities Act, which states...
Harrys father is an American citizen.
Example from Toulmin, 1958
22
Three Kinds of Inference

• Deductive inference
• Inductive inference
• Abductive inference (Charles Peirce)

23
Deductive Inference

• In logic if A then B
• A
• B
• In probability, reasoning from model parameters
  to probability of observations
  
• In Toulmin diagram, reasoning forward through the
  warrant--from data like the data at hand, to
  claims like the claim at hand.

24
Inductive Inference

• In logic if A then B if A then B
• not B
  B
  
• not A
  A?
  
• In statistics, reasoning from observations to
  model parameters
  
• In Toulmin diagram, reasoning backwards through
  the warrant, from the data at hand to the claim
  at hand.

25
Abductive Inference

• In logic New syllogisms
• In statistics,
• propose models for real-world situations
• revise models in light of anomalous data
• In Toulmin diagrams,
• propose diagram structure
• venture qualifiers

26
Multiple Diverse Pieces of Evidence

• Eg, physical evidence testimony, maybe about
  different but related parts of an argument.
  
• Overlap (double counting)
• Relationships among items of evidence
• Disagreement, contradiction.
• Determining relative value
• Different warrants, backing, alternative
  explanations

27
Recurring Patterns of Evidence

• Wigmores The science of judicial proof
• John Henry Wigmore, Dean at Northwestern Law
  School, in Evidence, 1900.
  
• Concern that admissibility rules got attention,
  over how to marshal argue from evidence.
  
• Sought recurring forms principals.
• Examples include fingerprints.

28
Recurring Patterns of Evidence

• Each of these notions raises difficult questions
  about what is involved in determining the overall
  probative force or weight of evidence.
  
• conflict
• corroboration
• convergence
• compound inferences (and, or, average)
• catenated inferences (chains)
• (Twining, 1985, p. 182)

29
Issue Did Y die of poison?
One line of Prosecutions argument
1b
7
Defenses rebuttal
1a
8
9
10
2
3
4

• 1a. Y died, apparently in health, within three
  hours after the drink of whisky.
  
• 1b. The cause of Ys death was poisoning.
• 2-4. Ys wife and the Northingtons witness to
  1a.
  
• 5. Y might have died from colic / 6. Witness
  testimony to Ys previous colic attacks
  
• 7. Colic would not have produced leg cramps and
  teeth-clenching strychnine would have.
  
• 8. Ys wife and the Northingtons witness to
  cramps and teeth-clenching.
  

Example based on Schum, 1994 adapted from
Wigmore, 1937
30
Issue Did Y die of poison?
One line of Prosecutions argument
A compound proposition
conflict
1b
7
Defenses rebuttal
1a
8
9
10
corroboration
2
3
4

• 1a. Y died, apparently in health, within three
  hours after the drink of whisky.
  
• 1b. The cause of Ys death was poisoning.
• 2-4. Ys wife and the Northingtons witness to
  1a.
  
• 5. Y might have died from colic / 6. Witness
  testimony to Ys previous colic attacks
  
• 7. Colic would not have produced leg cramps and
  teeth-clenching strychnine would have.
  
• 8. Ys wife and the Northingtons witness to
  cramps and teeth-clenching.
  

Example based on Schum, 1994 adapted from
Wigmore, 1937
31
Comments on Wigmores charting methods (1)

• Historical and current use
• In law Anderson Twining, Tillers
• AI/stat Schum, Bayes nets
• No single right diagram
• Perspective
• Grain-size
• Where to stop, how many links?

32
Comments on Wigmores charting methods (2)

• Benefits of using Wigmore charts
• Makes reasoning explicit sharable
• Deal with problems too big to hold in your head
  at once--can switch magnifying powers (Kadane
  Schums on Sacco-Vanzetti)
  
• Benefits of making Wigmore charts
• Forces you to think clearly and carefully about
  your arguments

33
Comments on Wigmores charting methods (3)

• How far do Wigmore charts go?
• They show structure of evidence claims
• No claim about how to integrate information
• Only qualitative information about strength of
  evidence
  
• Looking ahead to Bayes nets