Maximal Reliability of UnitWeighted Composites - PowerPoint PPT Presentation

Title: Maximal Reliability of UnitWeighted Composites


1
Maximal Reliability of Unit-Weighted Composites

• Peter M. Bentler
• University of California, Los Angeles
• SAMSI, NOV 05
• In S. Y. Lee (Ed.) (in press). Handbook of
  Structural Equation Models.

2
Reliability Internal Consistency of the
composite X based on p components

3

4

• Since uniqueness specificity error, i.e.,

5

• Under mild assumptions consistent with the above,
  Novick Lewis (1967) showed that

6

7

• This is not the only condition for

8

• This Hoyt-Guttman-Cronbach a is by far the
• most widely used measure of the internal con-
• sistency reliability of a composite X. In prac-
• tice, one substitutes the sample covariance
• matrix S and its diagonal DS to get

9
Some Recent References

• Becker, G. (2000). Coefficient alpha Some
  terminological ambiguities and related
  misconceptions. Psychological Reports, 86,
  365-372.
• Bonett, D. G. (2003). Sample size requirement for
  testing and estimating coefficient alpha. Journal
  of Educational and Behavioral Statistics, 27,
  335-340.
• Enders, C. K., Bandalos, D. L. (1999). The
  effects of heterogeneous item distributions on
  reliability. Applied Measurement in Education,
  12, 133-150.
• Green, S. B., Hershberger, S. L. (2000).
  Correlated errors in true score models and their
  effect on coefficient alpha. Structural Equation
  Modeling, 7, 251-270.
• Hakstian, A. R., Barchard, K. A. (2000). Toward
  more robust inferential procedures for
  coefficient alpha under sampling of both subjects
  and conditions. Multivariate Behavioral Research,
  35, 427-456.
• Kano, Y., Azuma, Y. (2003). Use of SEM programs
  to precisely measure scale reliability. In H.
  Yanai, A. Okada, K. Shigemasu, Y. Kano, J. J.
  Meulman (Eds.), New developments in psychometrics
  (pp. 141-148). Tokyo Springer-Verlag.
• Komaroff, E. (1997). Effect of simultaneous
  violations of essential tau-equivalence and
  uncorrelated error on coefficient alpha. Applied
  Psychological Measurement, 21, 337-348.

10

• Miller, M. B. (1995). Coefficient alpha A basic
  introduction from the perspectives of classical
  test theory and structural equation modeling.
  Structural Equation Modeling, 2, 255-273.
• Raykov, T. (1998). Coefficient alpha and
  composite reliability with interrelated
  nonhomogeneous items. Applied Psychological
  Measurement, 22, 375-385.
• Raykov, T. (2001). Bias of coefficient a for
  fixed congeneric measures with correlated errors.
  Applied Psychological Measurement, 25, 69-76.
• Schmitt, N. (1996). Uses and abuses of
  coefficient alpha. Psychological Assessment, 8,
  350-353.
• Shevlin, M., Miles, J. N. V., Davies, M. N. O.,
  Walker, S. (2000). Coefficient alpha A useful
  indicator of reliability? Personality
  Individual Differences, 28, 229-237.
• Schmitt, N. (1996). Uses and abuses of
  coefficient alpha. Psychological Assessment, 8,
  350-353.
• Yuan, K.-H., Guarnaccia, C. A., Hayslip, B. J.
  (2003). A study of the distribution of sample
  coefficient alpha with the Hopkins Symptom
  Checklist Bootstrap versus asymptotics.
  Educational Psychological Measurement, 63,
  5-23.

11
Some Advantages of ?

• Widely taught and known
• Simple to compute and explain
• Available in most computer packages
• ? is a lower bound to population internal
  consistency under reasonable conditions (not true
  of sample )
• Not reliant on researcher judgments (i.e., same
  data, same result for everyone)

12
Note Independence of researcher judgment is not
totally correct

• Different covariance matrices, or scalings of the
  variables, yield different
• and hence different
• Sometimes alpha is computed from the correlation
  matrix, and not the covariance matrix, implying
  the sum X is a sum of standardized variables

13
Some Disadvantages of ?

• a is not a measure of homogeneity or
  unidimensionality
• It may underestimate or overestimate uni-
• dimensional reliability
• Since
• it will overestimate reliability if ,
• the average covariance, is spuriously high
  or underestimate it if the average is spuriously
  low

14
This illustrates the advantages and disadvantages
of alpha

• The plus The average covariance is what it is,
  period. No judgment can change it. Hence, no
  judgment will change the reported .
• The minus A researchers model of sources of
  variance does not influence the reported
  reliability (alpha) when, perhaps, it should.
• Some examples from Kano/Azuma (2003) illustrate
  this.

15
If for a one factor
model and possibly correlated residuals (
not necessarily diagonal)

16
Three examples from Kano Azuma (2003) showing
alpha as accurate, and as an overestimate.
1-factor reliability (rho omega) ignoring
correlated errors can overestimate.

17
Alpha as a unidimensional lower-bound the
comparison of in a 1-factor model

• If , and is px1,
  is diagonal,
• . Example McDonald (1999) showed
• with equality when

18

19

• Thus,
• a is a good indicator of association
• but it is not clear what interesting sets of
  models it represents in the general case
• A model based coefficient such as
• provides a clear partitioning of variance,
• but it also can be (1) misleading (e.g., Kano
• -Azuma), (2) not relevant

20

• To give up a for a model-based coefficient, the
  model must be correct for S. That is, it should
  fit the data (say, sample covariance matrix S).
  I would conclude
• If is computed, the researcher must also
  provide evidence of acceptability of the 1-factor
  model.
• If a modified estimator
• is computed, the researcher should provide an
  argument for the variance partition.

21
Should the correlated error be part of the
residual covariance ? (on left), or part of the
common variance Sc (on right)? Substantive
reasoning should determine the variance
partitioning. This is more than just model fit.
22
What other model-based coefficients could be used
instead?

• Arbitrary latent variable model (Raykov Shrout,
  2002 EQS 6)
• Dimension-free lower bound (Bentler, 1972 bias
  correction Shapiro ten Berge, 2000)
• Greatest lower bound (Woodhouse Jackson, 1977
  Bentler Woodward, 1980 etc. ten Berge,
  Snijders, Zegers, 1981 bias correction Li
  Bentler, 2001)

23
Arbitrary LV Model

24
Dimension-free Lower Bound

25
Greatest Lower Bound

• This is a constrained version of the
  dimension-free coefficient. In addition to

26

27
Maximal Unidimensional Reliability

• The problem with a and the multi-dimensional
  coefficients seems to be that they do not
  represent unidimensional reliability
• Although not obvious, unidimensional reliability
  can be defined for multi-
• dimensional latent variable models. That is
  the main new result in this talk.

28
Repeating the Basic Setup Xi Ti Ei,
X T E ,
29
Can we have something like 1-Factor Based
Reliability when the latent variables are
multidimensional?
30
Maximal Unit-weighted Reliability
(p x k) for some k (small k lt
or large k)
for some acceptable k-factor model
is (px1) and
is (px(k-1))
, where
contains unrestricted free parameters.
, that is, the k-1 columns of
sum to zero.
contains free parameters subject to (k-1)(k-2)/2
restrictions (usual EFA identification
conditions)
31
Reliability under this Parameterization
X is based on 1 factor!
32
This is Maximal Unit-Weighted Reliability
and let t be a normal vector
Then the factor loading vector
that maximizes
and the residual factors
is given by
where
have zero column sums
33
Applications to Arbitrary Structural Models
Any structural model with additive errors
Linear structural model with additive errors
Let
with
Greatest lower bound
34
EFA Example (Its all in EQS)
/TITLE Maximum Reliability EFA Model Setup Nine
Psychological Variables /SPECIFICATIONS VARIABLES
9 CASES101 MATRIXCOVARIANCE METHODML
/EQUATIONS V1F1F20F3E1 V2F1F2F3E2 V
3F1F2F3E3 V4F1F2F3E4 V5F1F2F
3E5 V6F1F2F3E6 V7F1F2F3E7 V8F1
F2F3E8 V9F1F2F3E9
35
/VARIANCES F1 TO F3 1.0 E1 TO E9
.5 /CONSTRAINTS (V1,F2)(V2,F2)(V3,F2)(V4,F2)
(V5,F2) (V6,F2)(V7,F2)(V8,F2)(V9,F2)0 (V2,F3
)(V3,F3)(V4,F3)(V5,F3)(V6,F3) (V7,F3)(V8,F3)
(V9,F3)0
/MATRIX 1.00 .75 1.00 .78 .72 1.00 .44 .52 .47
1.00 .45 .53 .48 .82 1.00 .51 .58 .54 .82 .74
1.00 .21 .23 .28 .33 .37 .35 1.00 .30 .32 .37
.33 .36 .38 .45 1.00 .31 .30 .37 .31 .36 .38 .52
.67 1.00 /END
36
for 9 Psychological Variables