Multiple Comparison

1
Multiple Comparison
Type I Error Rates
error rate of any comparison
I. Per Comparison
of comparisons
II. Per Experiment (frequency)
if 10 and 0.05 per experiment
0.5
(frequency)
III. Familywise (for independent comparisons)
probability of at least one Type I error.
Per Comparison Familywise Per Experiment
2
Complete vs. Restricted H0
Independent Samples
Treatments
1 2 3
4 16 4 16 7 49
10 100 9 81 14 196
7 49 8 64 12 144
13 169 14 196 18 324
7 49 6 36 8 64
41 41 59
8.2 8.2 11.8
383 393 377
141
MSTreat
MSerror
Complete H0
3
Restricted H0
e.g.,
A Priori Comparisons Post Hoc Comparisons The
role of overall F
- might NOT pick up
- changes Familywise
4
A Priori Comparisons
Multiple t tests (compare two conditions)
- replace individual or with MSerror
- test t with dferror
Comparing Treatments 1 and 3
n.s.
5
Linear Contrasts
Compare ? two conditions ? a set of conditions
and a condition ? two sets of conditions
Let
for equal ns
6
(1) Contrasting Treatments 1 3 again




SScontrast

SSerror

MScontrast

always 1
MSerror

MScontrast
SScontrast


MSerror

(Look at t- test)
F(1,12)

7
(2) Contrasting Treatments 1 2 with 3







SScontrast

SSerror

MScontrast

MScontrast

F(1,12)

8
(3) Contrasting Treatments 1 3 with 2






SScontrast

SSerror

MScontrast

MScontrast

F(1,12)

9
Orthogonal Contrasts


if ns are equal
of comparisons dfTreat
dfTreat 2 in our example
contrasts

1 and 2



2 and 3



1 and 3


SScontrast1

SScontrast3

SSTreat

10
Bonferronis
Control for FW error rate (?)
use ? 0.01
Bonferroni Inequality
e.g. if, per comparison ? 0.05 and if, 4
comparisons are made then, the FW? CANNOT exceed
p 0.02
EW? or FW? ? c(PC?)
c of comparisons
Thus, we can set the FW? or the per experiment ?
to a desired level (e.g., 0.05) and adjust the PC?
If we desire a FW? 0.05 then
0.05 PC? (4)
0.0125 PC?
11
Bonferronis (comparing 2 means)
using t2 F and moving terms
This allows us to contrast groups of means.
(linear contrasts)
if
12
Multistage Procedures
Bonferroni divides ? into equal parts
Multistage (Holm) divides ? into different size
portions
if
or heterogenous S2s
13
Multistage
1.
calculate all
s
2.
arrange in order of magnitude
compare largest s to critical
3.
value (Dunns Tables)
C total of contrasts to be made
ONLY if significant
compare next largest to critical value C
C-1
4.
5.
and so on
FW? is kept at 0.05 (?)
14
Linear Contrast
Subject X Treatment Design
I
Weight each observation by its assigned condition
weight
II
Compute Di for each subject
III
Sum Di across subjects
IV
Compute SScontrast
Compute SScontrast
V
Compute SSSsXC(error)
15
Treatments
Subjects 1 2 3
1 4 4 7
2 10 9 14
3 7 8 12
4 13 14 18
5 7 6 8
Contrast 1 with 3
Di
Di2
4 0 -7 -3 9
10 0 -14 -4 16
7 0 -12 -5 25
13 0 -18 -5 25
7 0 -8 -1 1
18 76
MScon
SScon
MSSsXcon
SSSsXcon
df 4
F(1,4) 23.14
16
Treatments
Subjects 1 2 3
1 4 4 7
2 10 9 14
3 7 8 12
4 13 14 18
5 7 6 8
Contrast 1 3with 2
Di
Di2
-4 8 -7 -3 9
-10 18 -14 -6 36
-7 16 -12 -3 9
-13 28 -18 -3 9
-7 12 -8 -3 9
18 72
MScon
SScon
MSSsXcon
SSSsXcon
F(1,4) 36
17
SScon1
SScon2
SSTreat
Total
Orthogonal Contrasts
Error term could be
SSres or error
SSSsXcon1 SSSsXcon2 ?
5.6
1.2
6.8 ??


dfcon1 dfcon2 dfTreat

1

2
dfSsXcon1 dfSsXcon2 df ?
4
4


8