2-Group Multivariate Research

1
2-Group Multivariate Research Analyses

• Research Designs
• Research hypotheses
• Outcome Research Hypotheses
• Outcomes Truth
• Significance Tests Effect Sizes
• Multivariate designs
• Increased effects
• Increased specificity
• Considering confounds

2

• True Experiment
• random assignment of individual participants by
  researcher before IV manip (helps eliminate
  confounds)
• treatment/manipulation performed by researcher
  (helps eliminate confounds)
• good control of procedural variables during task
  completion DV measurement (helps eliminate
  confounds)

Research Designs
True Experiments If well-done, can be used to
test causal RH -- alternative hyp. are ruled out
because there are no confounds !!!

• Quasi-Experiment
• no random assignment of individuals (no confound
  control)
• treatment/manipulation performed by researcher
• poor or no control of procedural variables during
  task, etc. (no confound control)

Non-Experiments No version can be used to test
causal RH -- cant rule out alternative hyp.
Because there are confounds !!

• Natural Groups Design also called Concomitant
  Measures or Correlational Design
• no random assignment of ind. (no confound
  control)
• no treatment manipulation performed by researcher
  (all variables are measured) -- a comparison
  among participants already in groups (no confound
  control)
• no control of procedural variables during task,
  etc. (no confound control)

What designs go with what types of RH ????
3

• Basic Statistical Designs - BG vs. WG
• Between Subjects (Between Groups)
• each subject completes 1 of the IV conditions
• different groups each complete 1 of the IV
  conditions
• Within-subjects (Within-groups, Repeated
  Measures)
• each subject completes all of the IV conditions
  
• one group of subjects completes all of the IV
  conditions

• Design Language
• For both Between Within designs, we refer to
  the IV and the DV
• Typically the IV (causal variable) is
  qualitative
• Typically the DV (effect variable) is
  quantitative

• SPSS Language
• Between Groups Designs
• the IV is the grouping variable -- which IV
  condition each subject was in
• the DV is the response variable and tells each
  participants score on the DV
• Within-groups Designs
• there is no IV -- each variable is referred to
  as a DV
• there is one DV score for each IV condition
• each DV score tells the participants score in
  that IV condition

4

• ANOVA
• Between Groups (Independent Samples, etc.)
• H0 Populations represented by the IV conditions
  have the same mean DV.
• degrees of freedom (df) numerator 1,
  denominator N - 2
• Range of values 0 to ?
• Reject Ho If Fobtained gt Fcritical or
  If p lt .05
• Within-groups (Dependent Samples, etc.)
• H0 Populations represented by the IV conditions
  have the same mean DV.
• degrees of freedom (df) numerator 1,
  denominator N - 1
• Range of values 0 to ?
• Reject Ho If Fobtained gt Fcritical or If
  p lt .05

5

• Types of Research Hypotheses
• Attributive Hypothesis -- a construct
  (phenomena, behavior, etc.) exists
• an operational definition of the construct
• a system to measure the construct
• demonstration that the construct can be
  differentiated from other (related) constructs

• Associative Hypothesis -- two constructs are
  related (i.e., knowing the value of one
  provides information about the value of the
  other)
• demonstration of a statistical relationship
  between the variables used to measure the
  constructs
• specific statistical analysis is not important,
  as long as it is appropriate to the data and
  the expression of the research hypothesis

6

• Causal Hypothesis -- the value of one construct
  influences (causes, produces,
  etc.) the value of the other construct
• temporal precedence -- operation of IV comes
  before measurement of DV
• no alternative hypotheses (no design flaws,
  confounds, alternative explanations of the
  results, etc.)
• statistical relationship between IV and DV

• The types of RH are hierarchically arranged !!
• Posing a causal hypothesis assumes the
  associative hypothesis that the IV and DV are
  related -- if two things arent related then
  one can cause the other
• Posing an associative hypothesis assumes support
  for the attributive hypothesis of each
  construct/variable-- unmeasureable things
  cant be statistically analyzed

7
2-group RH and outcomes BG WG...
and three possible statistical outcomes
There are only three possible Research Hypotheses
Research Hypotheses G1 lt G2 G1 G2 G1
gt G2
Outcomes G1 lt G2 G1 G2 G1 gt G2
? ?
? ?
? ?
? ?
So, there are only 9 possible combinations of
RH Outcomes of 3 types effect as
expected unexpected null/effect
backward effect
? ?
? ?
8

• Keep in mind that rejecting H0 does not
  guarantee support for the research
  hypothesis?
• Why not ???
• The direction of the mean difference might be
  opposite that of the RH
• The RH might be thats theres no difference
  (RH H0)
• Also replication of findings is important,
  even when you get what you expect !!

? ?
? ?
9
2-group outcomes truth ... In the population
there are only three possibilities...
and three possible statistical decisions
In the Population G1 lt G2 G1 G2 G1 gt
G2
Decisions G1 lt G2 G1 G2 G1 gt G2
Type I error
Correctly rejected H0
Type III error
Type II error
Type II error
Correctly retained H0
Type I error
Correctly rejected H0
Type III error
Please note that this is a different question
than whether the results match the RH This is
about whether the results from the sample are
correct whether the results are right. This
is about statistical conclusion validity
10
The 9 outcomes come in 5 types Type I error --
false alarm - finding a significant mean
difference between the conditions in the
study when there really isnt a difference
between the populations Type II error -- miss -
finding no difference between the
conditions of the study when there really is a
difference between the populations Type III
error -- misspecification - finding a
difference between the conditions of the
study that is different from the the
difference between the populations Correctly
retained H0 -- finding no difference between the
conditions of the study when there
really is no difference between the
populations Correctly rejected H0 -- finding a
difference between the conditions of
the study that is the same as the the
difference between the populations
11
Practice with statistical decision errors
evaluated by comparing our finding with other
research
We found that those in the Treatment group
performed the same as those in the Control group.
However, the other 10 studies in the field found
the Treatment group performed better,
Type II
We found that those in the Treatment group
performed better than those in the Control group.
This is the same thing the other 10 studies in
the field have found.
Correct Reject
We found that those in the Treatment group
performed poorer than those in the Control group.
But all of the other 10 studies in the field
found the opposite effect.
Type III
We found that those in the Treatment group
performed better than those in the Control group.
But none of the other 10 studies in the field
found any difference.
Type I
We found that those in the Treatment group
performed the same as those in the Control group.
This is the same thing the other 10 studies in
the field have found.
Correct retain
12
Information from p-values vs. Effect Sizes

• The p-value (value range 1.0 0) tells the
  probability of making a Type I error if you
  reject the H0 based on the data from this sample
• e.g., p .10 means if we reject H0 based on
  these data there is a 10 chance that there
  really is no relationship between the variables
  in the population represented by the sample
• The usual acceptable risk is less than 5 or p
  lt .05
• Effect size estimates (value range 0 1.0) tell
  how much of the variability in the DV is
  accounted for (predicted from or caused by)
  the IV
• e.g., r .30 means we estimate that .302 or 9
  of the variability in the DV is accounted for by
  the IV
• large enough to be interesting effect sizes
  vary with research topics and design types, but a
  common guideline is .1 small, .3 medium and
  .5 large

13
Calculating Using Effect Sizes

• For 2-group ANOVA (BG or WG) r
  ? F / (F dferror)

Effect Size large enough
too small to be interesting
to be interesting
Significance Test p lt .05 p gt .05
Be careful about dismissing these many small
effects have turned out to be important
Best case big enough probably really there
Which to believe? Rem - w/ small N comes
lowered confidence in the replicability of
r Easier to believe r if it replicates earlier
research then the large p-value is probably
small N
Best case too small to care about probably
not really there
14
Where we go from here ...
2-group designs with a single DV

• multiple-group designs
• single DV
• multiple DVs

2-group designs with a multiple DVs

• Factorial designs (2 IVs)
• single DV
• multiple DVs

Knowing the design statistical analyses to
directly test any research hypothesis involving
treatment mean comparisons!!!
15
Multivariate Research -- when there are multiple
DVs

• Advantages of Multivariate Research
• Increasing the Number of Effects in
  the Research
• by including measures of multiple possible
  effects, we have a greater chance of finding an
  effect -- something that is influenced by or
  related to the IV
• e.g., If the IV were some sort of clinical
  treatment, using the Beck Depression Inventory
  State Anxiety Measure Somatic Complaint Scale
  gives us a better chance of detecting some type
  of improvement than would using just one of
  these
• research is costly (time ) -- multiple
  measures typically add little to the cost but
  increase the chances of finding something

16

• Advantages of Multivariate Research, cont.
• Increasing the specificity of the effects we
  find
• there is no one measure that is the perfect
  representation of the effect we are studying
  -- different measures of the same thing often
  are only moderately correlated (r .3-.5)
• using multiple related DVs allows us to more
  precisely define what is the effect
• e.g., If the construct DV under study were
  anxiety, we might want to have measures of
  anxiety physiological measures, self-report
  measures, observational measures
• that we we can better specify what we mean when
  we say the treatment decreases anxiety because
  we can say what types of anxiety showed the
  effect and which didnt

17

• Advantages of Multivariate Research, cont.
• Combining the Two Approaches in a Single Study
• multiple indices of multiple constructs -
  give the most precise and dependable results -
  greater chance of finding something influenced by
  IV - greater specificity about what is ( isnt)
  influenced by IV - replication is still
  important
• by using Beck Depression Inventory, MMPI
  Depression Scale, MCMI Depression Scale, State
  Anxiety Measure, Trait Anxiety Measure, Somatic
  Complaint Scale, MMPI Hypochondriasis Scale would
  allow us to determine if the treatment is
  specific to depression ( what kind), or
  includes anxiety and/or somatic complaints (
  what kinds)

18

• Using Multiple DVs in Quasi-Experimental
  and Natural Groups Designs
• Remember that confounds come in two kinds
• subject variable confounds
• IV groups start with different means, on
  something like age, education, personality
  attributes or motivation
• procedural variable confounds
• during IV manipulation or DV measurement,
  something besides the IV is done differently
  between the IV conditions, like instructions,
  amount of stimulus exposure or practice
• The presence of either type of confound
  interferes with the causal interpretation that
  mean differences on the DV indicate an effect of
  the IV
• confounds provide an alternative hypothesis
  about what caused the DV differences for the IV
  conditions

19

• Using Multiple DVs in Quasi-Experimental
  and Natural Groups Designs, cont.
• Measuring subject variables that you fear may be
  subject variable confounds can help
• any subject variable that does have a mean
  difference between IV conditions is a subject
  variable confound -- cant causally interpret the
  results of the study !!!
• that subject variable is an alternative
  hypothesis
• any subject variable that does not have a mean
  difference between the IV conditions cant
  possibly be a confounding subject variable
• remember that a subject variable working
  against the IV is a confound (technically), but
  does not refute that the IV may be causing the
  effect!
• you cant give a causal interpretation to the
  study, but you can establish whether or not a
  particular subject variable is a likely
  alternative hypothesis

20
Multivariate approach to confound evaluation
Design is a quasi-experiment w/o random
assignment of participants 2 different
kinds of exam prep Intended DV correct on the
exam grade on last exam GPA prior to this
class Exam prep time (hrs) Credit card
interest
rate
Tx control p r
89 78 .02 .38 86 77 .02
.37 2.87 2.86 .95 .001 2.22 2.78
.03 .29 14.3 17.1 .04 .12
-- looks pretty good !
-- effect in same direction a likely
confound
-- no effect cant be con-found of the IV-DV
relationship
-- a confound (even though not inflating the
IV/DV relationship)
a statistical confound relationship
to DV is either complicated or spurious