Cultural Nuances, Assumptions and the Butterfly Effect: A Prelude

1
Cultural Nuances, Assumptions and the Butterfly
EffectA Prelude

• A Brief Introduction
• to
• Generalizability Theory
• for the
• Uninitiated

2
What is Generalizability Theory?

• Generalizability Theorythe theory that deals
  with the degree to which one can say that
  conclusions drawn on the basis of data collected
  from sampling different domains can be applied
  with confidence to those domains.

3
Still Lost?

• What were talking about you best know as
  instrument reliability. Commonly known examples
• inter-rater reliability,
• test-retest reliability,
• split-halves reliability,
• Kuder-Richardson reliabilities,
• Hoyt reliability, and
• Cronbachs alpha

4
Test (True) Score Theory

• Observed scores are composed of true scores and
  error.
• The variance of the observed scores is
  partitioned.
• Then estimates are combined to produce a
  coefficient.

• Xo Xt Xe Xscore,
• ttrue, oobserved, eerror
• So2 St2 Se2
• S2 variance
• r11 St2/So2

5
True Score Theory and Generalizability

• Generalizabilty is an extension of true-score
  theory.
• Sources of Variation are viewed from different
  perspectives.
• S2O S2LG S2TG S2LS S2TS
• Llong term, Ttemporary Ggeneral,
  Sspecific

6
So what does this have to do with Multivariate
Analysis?

• All techniques we have studied deals with the
  degree of overlap of variance (information) for
  sets of variables (Tabachnick Fidell, 2001). In
  the present case the overlap of interest is
  between observed variance for evaluators
  opinions about the spheres involved and the
  actual underlying dimensions influencing
  interpersonal interactions perceived by the
  evaluators--true-score variance.
• S2TOT S2E S2V S2Sc S2e
• Eevaluator, Vvignette,Scscale, eerror

7
Relationship of Gerneralizability to Multivariance

• To produce the variance estimates needed ANOVA
  is used. In this situation a 6 x 7 x 46
  Evaluator (E) x Sphere Primary Category (S) x
  Vignette (V) Mixed Effects Repeated Measures
  ANOVA is used. Evaluator and Vignette are
  considered random, Sphere fixed.

8
ANOVA DESIGNDoubly Repeated MeasuresScale and
Vignette WithinEvaluator Between
9
Table 1 Variance Components Derived from the SPSS
GLM repeated Measures Calculation for Overall
Evaluations   Source of Variation SS
df MS   Vignette (V) 29.220
45 .649   Sphere (S) 639.865
6 106.644   Evaluator (E) 61.972
5 12.394   V x S
185.516 270 .687 V x E
96.528 225 .429 S x E
340.184 30
11.339 V x S x E   Error
(e)   Residual
648.149 1350
.480
10

• Table 2
• Variance Components and Expected Mean Squares for
  Overall Evaluations
•  
• Source of Variation Means Square (MS) Expected
  Mean Square (EMS) Estimated Variance
•  
• Vignette (V) .649 ?2e
  ?2VE ?2V .220
•  
• Sphere (S) 106.644 ?2e
  ?2VS ?2SE ?2S 94.618
•  
• Evaluator (E) 12.394 ?2e
  ?2VE ?2E 11.965
•  
• V x S .687 ?2e ?2VS
  .207
•  
• V x E .429 ?2e ?2VE
  -.059
•  
• S x E 11.339 ?2e ?2SE
  10.859
•  
• V x S x E ?2e ?2VSE .480
•  

11
Table 3 Variance Components Derived from the SPSS
GLM repeated Measures Calculation for
Spheres   Source of Variation SS df
MS   Vignette (V) 12.426 45
.276   Sphere (S) 22.575 5
4.515   Evaluator (E) 16.996
5 3.399   V x S 73.730 225
.328 V x E 27.643 225 .123   S x
E 24.769 25 .991   V x S x E
  Error (e)
  Residual 168.092
1125 .149
12

• Table 4
• Variance Components and Expected Mean Squares for
  Spheres
•  
• Source of Variation Means Square (MS) Expected
  Mean Square (EMS) Estimated Variance
•  
• Vignette (V) .276 ?2e
  ?2VE ?2V .153
•  
• Sphere (S) 4.515 ?2e
  ?2VS ?2SE ?2S 3.345
•  
• Evaluator (E) 3.399 ?2e
  ?2VE ?2E 3.276
•  
• V x S .328 ?2e
  ?2VS .179
•  
• V x E .123 ?2e
  ?2VE -.026
•  
• S x E .991 ?2e
  ?2SE .842
•  
• V x S x E ?2e ?2VSE .149
•  

13
A Sample Calculation
As an interesting example of how the ?s are
calculated here is one (see Table 2 for the
data)   Sphere (MS) 106.644 ?2e
?2VS ?2SE ?2S -V x S (MS)
.687 ?2e ?2VS -S x E (MS)
11.339 ?2e ?2SE Error (MS)
.480 ?2e Sphere (Variance)
95.094 ?2S -Error (MS)
.480 ?2e Sphere (Variance)
94.618   Total (TOT) 107.763 ?2e
?2S ?2V ?2E 107.283 ? S2S /
S2TOT 94.618 / 107.763 0.879 95.094 /
107.283 0.886   0.879 ? 0.882
14
The Conclusion

• What we get is three little numbers that are not
  as simple as they look
• Primary Influence
• ? .966
• Spheres
• ? .473
• Overall
• ? .879