CEP 933: Planned and Post hoc Contrasts

1
CEP 933 Planned and Post hoc Contrasts

• To make more explicit statements about the means
  that we have analyzed in ANOVA we must use
  contrasts, also known as planned and post hoc
  comparisons of means.
• From the names you can see that the difference
  here is whether you know what means you want to
  compare ahead of time (i.e., you have planned
  them), or whether you go hunting for mean
  differences after youve found a significant F
  test in ANOVA (then you are doing post hoc tests).

2
CEP 933 Planned and Post hoc Contrasts

• Well see below that planned (a priori) tests are
  more powerful than post hoc tests, because post
  hoc tests penalize you for just hunting around to
  figure out which means differ.
• In statistics it is always better to know what to
  expect than to just look around for something
  potentially interesting.

3
CEP 933 Planned and Post hoc Contrasts

• One other point is important here.
• If you have planned some comparisons of means
  ahead of time because you expect specific means
  to differ, then you do not really need to do the
  F test to see if you can reject the omnibus null
  hypothesis that all means are equal.
• Our book says that we dont even really need to
  do the F test for post hoc tests, but that is an
  unusual opinion.

4
CEP 933 Planned and Post hoc Contrasts

• Planned comparisons are more specific than the
  omnibus test, so they can be done whether or not
  the overall test is significant.
• Also usually they are more targeted we may only
  make a few of all possible comparisons when doing
  planned comparisons.
• Often researchers who have planned some
  comparisons do the omnibus test anyway, but it is
  not necessary.

5
CEP 933 Planned and Post hoc Contrasts

• When we use contrasts, the tests we use are built
  to protect us from having overly large chances of
  making a Type I error (i.e., we are protected
  from saying H0 is false when it is actually
  true).
• To do this the tests consider Type I error rates
  in two different ways, by examining the rate per
  comparison (PC or per-contrast) or the
  so-called familywise (FW) rate, which pertains
  to a set of comparisons.

6
CEP 933 Planned and Post hoc Contrasts

• If we are making several comparisons, the
  familywise rate may apply. The FW rate tells us
  what the chance is of making at least one Type I
  error in the set of comparisons.
• The comparison procedures differ because some of
  them limit the per-contrast error rate and the
  others limit the familywise rate. Suppose ?? is
  the error rate of one comparison. Then
• Per-contrast (PC) error rate ??
• Familywise (FW) error rate 1 - (1 - ??)c for c
  independent comparisons

7
CEP 933 Planned and Post hoc Contrasts

• Suppose ?? .05 is the error rate of one
  comparison. Also lets say we are making 3
  comparisons. Here are the values
• Per-contrast (PC) error rate ?? .05
• Familywise (FW) error rate
• 1 - (1 - ??)c 1- (.95)3 1-.857 .142
• If we just add the rates of the three comparisons
  we get
• c ?? 3 ?? 3 .05 .15
• In general, PC lt FW lt c ?? FW is usually close
  to c ??

8
CEP 933 Planned and Post hoc Contrasts

• Also here there is one additional concern when
  we look at the data before doing comparisons we
  are increasing our chances of making a Type I
  error, because we may decide to only test the
  differences among means that look big.
• This is why most post-hoc comparisons examine all
  possible comparisons, and also why post-hoc tests
  are not as powerful as planned tests.

9
CEP 933 Whats a contrast anyway?

• A contrast is just a linear combination of means.
  Usually such a combination takes the form of a
  difference between two means, or a difference
  between averages of two sets of means. For
  instance,
• L1 -
• is a contrast. Also another contrast is
• L2 ( )/2 -

10
CEP 933 Whats a contrast?

• The example
• L1 -
• is a pairwise comparison. Here we have taken the
  difference between two means.
• Some of the tests well encounter examine ALL
  pairwise comparisons (i.e., they look at all
  k(k-1)/2 differences for all possible pairs in a
  set of k means).

11
CEP 933 Whats a contrast?

• The contrast
• L2 ( )/2 -
• is a more complex contrast.
• For the example L2, we have averaged the means
  for groups 1 and 2 and we compare them to the
  mean for group 3. This is the same as writing
• 1/2 1/2 (-1)

12
CEP 933 Whats a contrast?

• All comparisons are tests of specific hypotheses.
  
• For instance if we are comparing all pairs of
  means we are testing
• H0 mj mj? for j and j? 1, 2, k where j
  ?j?
• Or suppose we are testing L2 above. We will be
  testing
• H0 (m1 m2 )/2 m3 or
• H0 (m1 m2 )/2 - m3 0

13
CEP 933 Contrasts

• The contrast in the population is often named
  using the Greek letter psi
• Y S cj mj
• So for L1 above we have
• Y1 S cj mj (m1 - m2 )
• and for L2 we have
• Y2 S cj mj (m1 m2 )/2 - m3

14
CEP 933 Contrasts

• Our book calls the estimate of the contrast L,
  for linear combination, and L S cj .
  (The book also uses aj for the coefficients, but
  cj is more common notation.)
• The values of the "coefficients" cj are
  determined by your ideas about the subsets of
  means you want to compare. Above the cj values
  were 1/2, 1/2 and -1. The examples below should
  help to clarify the nature of the coefficients.
• Later we will learn how to use contrasts, but for
  now well see a list of the many possible
  comparison procedures that use contrasts.

15
CEP 933 List of comparison tests

• TEST USE ERROR, POWER
• --------------------------------------------------
  ---------------------------
• POC k-1 planned per-contrast a
• Planned independent (Most folks use a as
• orthogonal contrasts rate for each test, but
  we
• comparisons can use the Bonferroni
  approach to reduce error)
• most powerful contrast
• tests available
• Trend k-1 independent same as POC
• trend tests

16
CEP 933 List of comparison tests

• TEST USE ERROR, POWER
• --------------------------------------------------
  ---------------------------
• Dunn or any (c) of planned familywise a
• Bonferroni contrasts (use per-contrast level of
• (use if contrasts a/c if you desire
• are not orthogonal) familywise rate of a a)
• Dunnett paired contrasts of 1 familywise a
• mean with (k-1) other
• means (e.g., one control
• vs other treatments)

17
CEP 933 List of comparison tests

• TEST USE ERROR, POWER
• --------------------------------------------------
  --------------------------
• Fishers LSD post-hoc, familywise a
• all pairs of means
• Tukeys HSD post-hoc, familywise a
• all pairs of means
• same CV is used for all pairs
• Newman-Keuls post-hoc, mystery a
• all pairs of means (fw rate ka/2
• CV varies as means for even k)
• get closer power is higher
• than Tukey test

18
CEP 933 List of comparison tests

• TEST USE ERROR, POWER
• --------------------------------------------------
  ---------------------------
• Ryan post-hoc, controls rate using
• all pairs of means different levels for each
• pair of means
• Scheffe any of post hoc familywise a
• contrasts based on large set of
• contrasts, low power

19
CEP 933 Rules for coefficients

• There are not too many rules for figuring out
  what the coefficients in a contrast should be.
  The coefficients are the numbers we multiply
  times our means. Here are a few rules
• 1. The coefficients MUST sum to zero within a
  contrast.
• 2. The coefficients should make sense they
  often produce averages that are subtracted from
  each other.
• 3. Its better (easier) to use integers (whole
  numbers).

20
CEP 933 Rules for coefficients

• 1. The coefficients MUST sum to zero within a
  contrast.
• Consider our contrast above
• L2 ( )/2
• We said this is the same as The
  s show
• L2 ½ ½ 1 the
  coefficients.
• The sum of the coefficients is ½ ½ -1 0

21
CEP 933 Rules for coefficients

• 2. The coefficients should make sense they
  often produce averages that are subtracted from
  each other.
• Suppose we have 5 means. If we want to compare 3
  means to 2 others we can compute
• This is like computing two new means
• and
• Then we compute

22
CEP 933 Rules for coefficients

• 3. Its better (easier) to use integers (whole
  numbers).
• This is true because eventually we need to get
  standard errors for the contrasts, and the SEs
  involve squaring the coefficients. So for the
  contrast
• well need to square all the fractions in this

23
CEP 933 Rules for coefficients

• 3. Its better (easier) to use integers (whole
  numbers).
• Squaring all the fractions produces
• This will be a mess when we try to compute the
  SEs!!
• So we can use integer coefficients instead.

24
CEP 933 Rules for coefficients

• 3. Its better (easier) to use integers (whole
  numbers).
• One way to find the integer coefficients is to
  get the least common denominator (LCD) for the
  fractions.
• The LCD is 6 for 1/3 and ½ so we can multiply all
  the coefficients by 6 and use
• Also as our book points out, this is the same as
  using as a coefficient the number of means in the
  OTHER set.

25
CEP 933 Rules for coefficients

• There are also some special kinds of coefficients
  for special contrasts.
• In particular we will learn about trend tests
  for those we use weights that represent linear,
  quadratic, cubic trends, etc. The coefficients
  are made up especially for testing trends and are
  on page 742 in our book. For instance for k5
  means (for a 5-level quantitative factor), the
  linear trend contrast is

26
CEP 933 Data for examples

• Suppose we want to compare five modes of
  presentation of course materials, with an outcome
  representing student learning of the material.
  We could consider the lecture group to be a
  baseline or control group all the others seem
  to use video or technology-rich modes.
• Group n Id (j) Mean
• Interactive video 10 1 48.7
• CAI 10 2 43.4
• Standard video 10 3 47.2
• Slide tape 10 4 36.7
• Lecture 10 5 40.3

27
CEP 933 Data for examples

• Source df SS MS F
• Between 4 969.32 242.33 8.41
• Within 45 1296 28.8
• Total 49 2265.32
• Note that because Fc(4,45) 3.83, the overall
  (omnibus) F test is significant, indicating that
  we can proceed with either planned or post hoc
  tests.
• Also E2 969.32/2265.32 .428 or about 43 of
  score variance is explained by mode of
  presentation.

28
CEP 933 Planned orthogonal comparisons

• Planned orthogonal comparisons (POC) are
  contrasts of a certain type. For any one-way
  anova with k groups, there are k-1 POCs. POCs
  are simply contrasts that are orthogonal or
  independent of one another and we can determine
  this by looking at each pair of contrasts and
  seeing if they are independent.
• Each set of k-1 POCs provides tests of all of the
  unique information in the k means. Also, there
  may be more than one set of POCs for any set of k
  means.
• To tell whether two contrasts are orthogonal, we
  multiply together the weights (the cj values)
  from the contrasts.

29
CEP 933 Planned orthogonal comparisons

• Consider the following two possible contrasts for
  the means in the data above.
• A Interactive versus standard video 1 vs. 3
• LA -
• B Video (I or S) versus all others 1 and 3
  vs. rest
• LB ( )/2 - (
  )/3

30
CEP 933 Planned orthogonal comparisons

• The following weights are being applied to the 5
  group means
• Group
• 1 2 3 4 5
• Contrast A 1 0 -1 0 0
• weights
• Contrast B .5 -.33 .5
  -.33 -.33
• weights
• Note first that the weights within each contrast
  must sum to zero. Then we check for orthogonality.

31
CEP 933 Planned orthogonal comparisons

• We compute the products of the pairs of weights
  for each group
• Group 1 2 3 4 5
• Products 1 .5 0 -.33 -1.5 0 -.33
  0 -.33
• of weights .5 0 -.5 0
  0
• If we add up the products and they sum to zero,
  the two contrasts are orthogonal or independent.
  So here contrasts A and B are orthogonal ( .5 0
  -.5 0 0 0).
• There are always k-1 independent contrasts, and
  there may be several different sets of
  independent contrasts for any set of k means.

32
CEP 933 Planned orthogonal comparisons

• Lets look at another set of weights. Well
  compare the first four trts to the last (standard
  lecture) this will be contrast C and we will
  keep contrast B
• Group
• 1 2 3 4 5
• Contrast C 1 1 1 1 -4
• weights
• Contrast B .5 -.33 .5
  -.33 -.33
• weights
• The weights within each contrast do still sum to
  zero. Then we check for orthogonality.

33
CEP 933 Planned orthogonal comparisons

• We compute the products of the pairs of weights
  for each group
• Group 1 2 3 4 5
• Products 1 .5 1 -.33 1.5 1 -.33
  -4 -.33
• of weights .5 -.33 .5 -.33
  1.33
• We add up the products and here contrasts A and B
  are NOT orthogonal (.5 -.33 .5 -.33 1.33
  1.67).
• So B and C are not orthogonal contrasts.

34
CEP 933 Planned orthogonal comparisons

• How about contrasts A and C?
• Group 1 2 3 4 5
• Contrast A 1 0 -1 0 0
• weights
• Contrast C 1 1 1 1 -4
• weights
• Products 1 1 01 -11 01 0-4
• of weights 1 0 -1 0 0
• Are A and C orthogonal?

35
CEP 933 Planned orthogonal comparisons

• Finally note that although trend tests are always
  orthogonal, trend tests dont make sense for this
  example. The treatment factor has groups that are
  qualitatively different the levels represent
  different kinds of instruction.
• So we would not use trend tests on these data
  even though we could compute them. If the factor
  were duration of instruction (e.g., 10 minutes,
  20 minutes, 30 minutes, etc. through 50 minutes)
  we could do trend tests.