Data as Evidence

1
Data as Evidence

• By
• Kevin T. Kilty
• LCCC Star Club
• November 20, 2003

2
Specific Topics

• Invalid reasoning by way of circular logic
  provides no reliable evidence at all.
• What does a person do about missing, censored,
  and suspended data?
• The ordinary way of testing hypotheses is not a
  measure of evidence. Likelihood ratio is.

3
Circular Reasoning

• Presuming in advance what you intend to prove
  through an experiment.
• In other words, you are guaranteeing the
  correctness of a hypothesis through axiomatic
  elements of the experiment. That is the
  hypothesis under test is actually an axiom.

4
The unsmiling Newton
Considered by many (physicists) to be the
greatest scientist of all time. The Principia
presented a system of the world, but not
everything was known in 1689, was it? His system
worked perfectly for celestial mechanics, but not
well for poorly understood things like the speed
of sound.
5
Example Isaac Newton and fudging

• Speed of sound in a gas Velocity squared
• Newton could not reach the correct value through
  the isothermal compressibility, so he invented
  correction factors.
• Ref Westfall. Newton and the fudge factor,
  Science, 179, 751-8, 1973.

Is easy to measure but this is correct
6
The Baltimore Case

• Teresa Imanishi-Kari accused of scientific
  misconduct.
• Forensic experts establish the anomalosity of
  her lab notebook by comparing it only to other
  notebooks that did not appear the same.

7
A little misdiagnosis of JE
Outbreak of a flu-like disease in Malasia in
1998. The cause is presumed to be Japanese
Encephalitis (JE). Hypothesis confirmed by
finding antibodies for JE in the blood of many
victims. As authorities respond to JE another 100
people die. The actual cause is a new viral agent.
8
Toxic Oil Syndrome

• Spain 1981. Mysterious deaths presumed to be the
  result of an infectious agent.
• This hypothesis confirmed by finding a structure
  resembling a pneumonia-producing bacterium.
• Actual cause found by careful examination of an
  outlier. The cause is some chemical agent in rape
  seed cooking oil imported from France.

9
Hormone Replacement Therapy

• Womens Health Initiative (WHI) hormone therapy
  study halted in summer 2002 because it appeared
  to represent more risks than benefits.
• 7 or 8 more strokes, cardiovascular events,
  breast cancers per 10,000.
• 5 or 6 fewer colorectal cancers and broken hips.
• Women not on HRT do not have lower quality of
  life than those on the therapy.
• Caveat women most likely to benefit from HRT
  were excluded during study screening.

10
Small World Studies

• A problem that Milgram first suggested. What are
  social connections like?
• Milgrams experiment used the post office. Ref
  Psychology Today, 1, 61-67.
• A recent study used the internet.
• Ref Dodds, et al., Science, 301, 827-829, 2003.
• A skeptic (Ref Kleinfeld, Society, 39, 61, 2002.)

11
Missing data
12
Censored and suspended data

• Censored data have magnitudes too small to
  measure. We know there is a value, but cant
  enumerate it. An example chemical analyses that
  are BDL.
• Suspended data come from experiments terminated
  early for some reason. Example In failure
  studies some of the samples will not have failed
  by the time we need to stop.

13
Example of suspended data

• Of 26 pumps installed and tested on trucks from
  03/97 To 11/01, only 18 failed. Whadya do about
  the others?

14
The typical statistical analysis of data

• Presume a test statistic, set rejection region.

15
Likelihood

• Knowing what PDF applies to an event, a person
  can assign a likelihood or probability to its
  occurrencei.e. likelihood of data .
• We can reverse the conditional probability, so to
  speak, to write the likelihood of circumstances
  given particular data.

16
An example of Likelihood

• Wall Street Journal (Wednesday August 3, 2003
  section D3) carried a story about risks of debris
  at ground-zero for low-weight birth.
• Exposed group 15 LWB in 187.
• Control group 4 LWB in 89.
• Binomial PDF.

17
Likelihood calculations

• Pooled probability (p) of LBW is 6.9
• Which is about 0.0114
• Otherwise p is 8.2 in the exposed group and 3.8
  in the control.
• Which is about 0.0201
• The Ratio is less than 2. This is not strong
  evidence.

18
Examining data as it comes in.

• Typically forbidden to do in typical clinical
  trial that is double blind and designed for a
  particular power.
• Makes perfect sense to do when the protocol is
  winner continues and the consequences of a bad
  alternative therapy are horrific.
• Royalls example of an egregious trial.

19
Conclusions

• Protect your study from circular axiomatic
  influences.
• Return to original sources
• Never defer to authority maintain skepticism
• View missing and censored data carefully because
  it represents a bias. A fair test cannot contain
  a bias.
• Think about using Likelihood in place of, or as
  supplement to, usual statistical methods.