Measuring Group-Level Psychological Properties (A Tribute to Larry James)

1
Measuring Group-Level Psychological Properties(A
Tribute to Larry James)

• Daniel A. Newman
• University of Illinois

Daniel A. Newman, Ph.D.
2
Overview

• Group-Level Psychological Properties?
• Psychological Climate
• Group-Level vs. Individual-Level Constructs
• Aggregation Bias
• Why we need rWG (Within-group agreement)
• Justifying Aggregation
• rWG(J) for multi-item scales
• Agreement vs. Reliability

2
3
Overview

• Group-Level Psychological Properties?
• Psychological Climate
• James Jones (1974), Jones James (1979), James
  Sells (1981), James (1982), James et al.
  (1988), James James (1989)
• Aggregation Bias James (1982), James et al.
  (1980)
• Why we need rWG (Within-group agreement)
• James (1982), James, Demaree, Wolf (1984
  1993), George James (1993)
• rWG(J) for multi-item scales
• James, Demaree, Wolf (1984), LeBreton, James,
  Lindell (2005)

3
3
4
Overview

• Group-Level Psychological Properties?
• Psychological Climate
• James Jones (1974), Jones James (1979), James
  Sells (1981), James (1982), James et al.
  (1988), James James (1989)
• Aggregation Bias James (1982), James et al.
  (1980)
• Why we need rWG (Within-group agreement)
• James (1982), James, Demaree, Wolf (1984
  1993), George James (1993)
• rWG(J) for multi-item scales
• James, Demaree, Wolf (1984), LeBreton, James,
  Lindell (2005)

4
4
4
5
Quotes Equations

• In summarizing Larry Jamess contributions to
  Multilevel Theory, Ill use a two-pronged
  approach
• Quotes
• Equations

5
5
6
Quotes Equations

• In summarizing Larry Jamess contributions to
  Multilevel Theory, Ill use a two-pronged
  approach
• Quotes
• Equations

6
7
Levels of Analysis

• In social science, hypothetical constructs reside
  at multiple levels of analysis (or levels of
  aggregation)
• National Level Culture
• Organizational Level Organizational Climate, CEO
  personality, Strategy
• Team Level Team efficacy, Norms, Leader style
• Individual Level Attitude, Personality, Job
  Performance, Psychological Climate

8
Levels of Analysis
Organizational
Group
Individual
9
Levels of Analysis

• Individuals are nested within Groups
• Groups are nested within Organizations
• One level can influence another
• Group norms influence individual behavior
• Individual behaviors aggregate to produce
  group/team performance

10
Psychological Climate

• Psychological Climate the meaning an
  individual attaches to a work environment
• Organizational Climate the aggregated meaning
  i.e., the typical, average, or usual way people
  in a setting work environment describe it
• Schneider (1981, pp. 4-5), as cited by James
  (1982)

10
11
Psychological Climate

• Psychological Climate individual level
  construct
• Organizational Climate group level construct

11
11
12
Psychological Climate

• perceptual agreement implies a shared
  assignment of psychological meaning, from which
  it follows that an aggregate (mean) climate score
  provides the opportunity to describe an
  environment in psychological terms.
• Furthermore, given perceptual agreement, I
  submit that a climate construct at the aggregate
  level is defined in precisely the same manner as
  it is at the individual level.
• James (1982, p. 221)

12
12
13
Psychological Climate

• Relationship between organizational climate and
  psychological climate
• PC psychological climate perception of person
  in a group
• OC organizational climate of the group

13
13
13
13
14
Psychological Climate

• Relationship between organizational climate and
  psychological climate
• PCpg psychological climate perception of person
  p in group g
• OC0g organizational climate in group g

14
14
15
Psychological Climate

• Relationship between organizational climate and
  psychological climate
• PCpg psychological climate perception of
  person p in group g
• OC0g organizational climate in group g
• upg deviation of person ps individual psych.
  climate perception from group gs org. climate

15
16
Psychological Climate

• James Jones (1974), reviewed 3 approaches to
  conceptualize measure org. climate
• Org.-Level Attribute, Multiple Measures
• Org.-Level Attribute, Perceptual Measures
• Indiv.-Level Attribute, Perceptual Measures
• Introduced the term, Psychological Climate

16
16
17
James Jones (1974)

• Returning to the perceptual definition of
  organizational climate, it would seem that the
  reliance on perceptual measurement may be
  interpreted as meaning that organizational
  climate includes not only descriptions of
  situational characteristics, but also individual
  differences in perceptions and attitudes. This is
  somewhat confusing if one wishes to employ
  organizational climate as an organizational
  attribute or main effect, since the use of
  perceptual measurement introduces variance which
  is a function of differences between individuals
  and is not necessarily descriptive of
  organizations or situations. Therefore, the
  accuracy and/or consensus of perception must be
  verified if accumulated perceptual organizational
  climate measures are used to describe
  organizational attributes (Guion, 1973). (p.
  1103)

17
17
18
Jones James (1979)

• The conceptual argument for aggregating
  perceptually based climate scores (i.e.,
  psychological climate scores) appears to rest
  heavily on three basic assumptions first, that
  psychological climate scores describe perceived
  situations second, that individuals exposed to
  the same set of situational conditions will
  describe these conditions in similar ways and
  third, that aggregation will emphasize perceptual
  similarities and minimize individual differences.
  Based on this logic, it is generally presumed
  that empirically demonstrated agreement among
  different perceivers implies that these
  perceivers have experienced common situational
  conditions (Guion, 1973 Insel Moos, 1974
  James Jones, 1974 Schneider, 1975a),
• (p. 206).

18
18
18
19
James Jones (1974)

• Although this school of thought from Schneider
  and others assumes that situational and
  individual characteristics interact to produce a
  third set of perceptual, intervening variables,
  such an assumption does not mean that perceived
  climate is different from an individual
  attribute. Rather, the intervening variables are
  individual attributes which provide a bridge
  between the situation and behavior.
• (p. 1107)
• So Psychological Climate is born!

19
19
19
20
James (1982)

• current thinking in climate suggests that the
  unit of theory for climate, including
  organizational climate, is the individual, and
  the appropriate unit to select for observation is
  the individual. This thinking is based on the
  view that climate involves a set of macro
  perceptions that reflect how environments are
  cognitively represented in terms of their
  psychological meaning and significance to the
  individual.
• (p. 219)
• So measuring organizational climate (an
  org.-level attribute) involves an
  individual-level true score (i.e., psychological
  climate).

20
20
20
20
21
James et al. (1988)

• Shared assignment of meaning justifies
  aggregation to a higher level of analysis (e.g.,
  groups, subsystems, organizations) because it
  furnishes a way of relating a construct (PC) that
  is defined and operationalized at one level of
  analysis (the individual) to another form of the
  construct at a different level of analysis (e.g.,
  group climate, subsystem climate, OC). Although
  the unit of analysis for the aggregate
  psychological variable is the situation (e.g.,
  group, subsystem, organization), the definition
  and basic unit of theory remains psychological.
• (p. 130, from Organizations Do Not Cognize)

21
21
21
21
21
22
James James (1989)
General PC
.85
.81
.77
.86
Group Warmth Cooperat.
Leader Support
Role Stress, Conflict, Ambiguity
Job Autonomy, Challenge

• PC Cognitive evaluation of work environment
• See James Sells (1981), Jones James (1979)

22
22
22
22
22
23
Psychological Climate
Psych. Climate
Job Satisfaction/Affect

• Reciprocal relationship between PC and Job
  Satis./Affect
• James Tetrick (1986), James James (1992)

23
23
23
23
23
23
24
Psychological Climate

• Summary
• There is a group-level organizational reality
  (the situation)
• That reality is reflected in individual-level,
  psychological perceptions
• The individual-level psychological climate
  perceptions are a meaningful locus of theory
• The individual perceptions can be aggregated to
  represent a group-level, psychological property
  if perceptions are shared

24
24
25
Aggregation Bias

• Aggregation combining micro-level data so it
  can represent the macro-level (typically, by
  taking an average of micro-level responses)
• The aggregate of individuals scores represents
  the group-level construct

26
Levels of Analysis
Organizational
Group
Individual
26
27
Aggregation

• Ecological fallacy generalizing group-level
  (aggregate) results to the individual level
• Because we know group collectivism is related to
  group-level cooperation, we inaccurately assume
  individual collectivism is related to individual
  cooperativeness.
• Atomistic fallacy generalizing individual-level
  results to the group (aggregate) level
• Because we know indiv. IQ is strongly related to
  indiv.-level job performance, we inaccurately
  assume group IQ is strongly related to group
  performance.

28
Aggregation

• The Truth about Aggregates
• If the individual-level correlation between X and
  Y is rindiv. .3, this does not imply that the
  group-level correlation between X and Y is rgroup
  .3.
• Likewise, if the group-level correlation between
  X and Y is rgroup .3, this does not imply that
  the individual-level correlation between X and Y
  is rindiv. .3.

29
Aggregation
Direction of a correlation ( or -) can change
when we move from the individual level to the
group level.
Within-Group Correlation Between-Group Correlatio
n
Y
X
30
Aggregation
Example) Foreign birth Illiteracy (Robinson,
1950). rindiv. .12 rgroup(states) -.53
Within-Group Correlation Between-Group Correlatio
n
Y
X
31
Aggregation
Total correlation is a combination of the
individual-level correlation and the group-level
correlation.
Within-Group Correlation Between-Group Correlatio
n
rtotal
rwithin
Y
rbetween
Total Correlation
X
32
Aggregation

• Total correlation is a combination of the
  individual-level (within) correlation and the
  group-level (between) correlation.

33
Aggregation

• Specifically,
• rtotal overall X-Y correlation, ignoring
• group membership
• rbetween between-groups X-Y correlation
• rwithin within-groups X-Y correlation
• (from ANOVA DV X,
  IV group)
• like R2 variance in X accounted for by group
  membership, then inflated by the unreliability of
  group means i.e.,
  .

34
Aggregation

• For example, suppose
• rbetween -.45 between-groups X-Y correlation
• rwithin .20 within-groups X-Y correlation
• .64 (from ANOVA DV X, IV
  group)
• .81 (from ANOVA DV Y, IV
  group)
• Then

35
Aggregation

• For example, suppose
• rbetween -.45 between-groups X-Y correlation
• rwithin .20 within-groups X-Y correlation
• .64 (from ANOVA DV X, IV
  group)
• .81 (from ANOVA DV Y, IV
  group)
• Then

36
Aggregation
Total correlation is a combination of the
individual-level correlation and the group-level
correlation.
Within-Group Correlation Between-Group Correlatio
n
rtotal
rwithin
Y
rbetween
Total Correlation
X
37
Aggregation

• Implications
• Even if total correlation between X and Y
  (rtotal) is statistically significant,
• rwithin might not be
• rbetween might not be
• Many studies in top journals report total
  relationships between variables, while ignoring
  nesting/ nonindependence (e.g., different groups,
  different jobs, different supervisors).
  Considering levels of analysis could potentially
  change the results!

38
Aggregation

• Implications
• So-called aggregation bias when rbetween is
  larger than rtotal
• Only occurs if rbetween happens to be larger than
  rwithin

38
39
Aggregation Bias

• Implications
• Dont look at rtotal to draw inferences about
  rwithin!
• Dont look at rtotal to draw inferences about
  rbetween!
• See James (1982) and James, Demaree, Hater
  (1980), who applied similar formulae to estimate
  bias in both h2 and corr.s between aggregated
  situational (OC) and individual difference
  variables.

40
Aggregation Bias

• Summary
• When we aggregate individual-level measures
  (e.g., psychological climate) to represent
  organizational attributes (e.g., organizational
  climate), then all the theoretical and empirical
  relationships can change.
• Aggregation of the same measures can create a
  different construct!

40
40
40
41
Why We Need rWG

• Justifying Aggregation
• organizational climate is the overall meaning
  derived from the aggregation of individual
  perceptions of a work environment (i.e., the
  typical or average way people in an organization
  ascribe meaning to that organization) (James,
  1982 Schneider, 1981). Thus, organizational
  climate can be viewed as the outcome of
  aggregating individuals psychological climates.
  The important caveat is that these psychological
  climates are shared in order to make the
  inference that an organizational climate exists.
• James et al. (2008, pp. 15-16)

41
41
42
Why We Need rWG

• Group-Level Consensus Constructs
• In measuring group consensus constructs,
  agreement and reliability are tools used to
  justify aggregation of individual-level responses
  to the group level
• Agreement and reliability help us gauge how well
  the average across individual responses
  represents the group.

43
Why We Need rWG
Group-Level Consensus Constructs
Organizational Climate (average)
Psych. Climate, Person 1
Psych. Climate, Person 2
Psych. Climate, Person 3
44
Why We Need rWG

• Overview
• Aggregation/Composition Models
• Chan (1998)
• Kozlowski Klein (2000)
• Agreement
• rWG family of indices
• Reliability
• ICC(1)
• ICC(2)

See Bliese, 2000
45
Why We Need rWG

• Aggregation/Composition Models
• Chan (1998)
• Kozlowski Klein (2000)
• Both typologies include consensus models
• Use the mean of individual responses to represent
  the group-level construct
• Assume isomorphism (James, 1982)
• Require high within-group agreement

46
Why We Need rWG

• Within-Group Agreement degree to which ratings
  from individuals are interchangeable
• Agreement-based tests reflect degree to which
  raters provide essentially the same rating
• Three dominant indices designed to assess
  within-group agreement
• James et al.s (1984) rWG(J)
• Lindell et al.s (1999)
• Burke, Finkelstein, Dusigs (1999) AD index

47
George James (1993)

• The key statistical test of the appropriateness
  of aggregation to the group level of analysis is
  that there is within-group agreement on the
  variable in question. If there is agreement
  within groups on the theorized group-level
  variable, then the aggregate may be used in
  subsequent analyses.
• agreement within a group is not conditional on
  between-groups differences. For example, in a
  scenario that Yammarino and Markham portray, in
  which all members in each group have the same
  moderately high score, both agreement and
  aggregation may be justified provided that
  aggregation to the group level was theoretically
  based. However, there would be no group effect
  inasmuch as the group means do not vary under
  these conditions.
• (p. 799)

47
47
47
47
47
47
47
48
Why We Need rWG

• Within-Group Agreement
• For single items
• observed variance of single item
• theoretical null variance (represents
  zero agreement)
• rWG 1 - observed variance over expected
  variance

49
Why We Need rWG

• Summary
• Under consensus composition models (with
  isomorphism across levels), within-group
  agreement is needed to justify aggregation.
• Within-group agreement is even more essential
  than ICC(1) and ICC(2), both of which depend upon
  between-group variance.
• Within-group agreement shared psychological
  meaning!
• rWG is the key to measuring group-level
  psychological properties!

49
49
49
49
50
rWG(J) for Multi-Item Scales

• rWG(J) is NOT the same as rWG!
• rWG for single items
• rWG(J) for multiple-item climate scale

50
50
51
rWG(J) for Multi-Item Scales

• Within-Group Agreement (James et al., 1984)
• For multiple
• items
• J number of items
• mean of observed item-level variances
• theoretical null variance (represents
  zero agreement)
• Can be derived without Spearman-Brown (LeBreton
  et al., 2005)

52
rWG(J) for Multi-Item Scales

• Three Issues with James et al.s (1984) rWG(J)
• J number of items
• (is rWG(J) an index of agreement, reliability, or
  both?)
• mean of observed item-level variances
• theoretical null variance (represents
  zero agreement)
• (addressed by LeBreton Senter, 2008)

53
James et al. (1993)

• Describing whether rWG(J) is an index of
  agreement vs. reliability
• Kozlowski and Hattrup are also correct in
  stating that our intention was to suggest a
  measure of agreement, and not consistency
  reliability, and that rWG is an estimator of
  agreement. However, what cannot be done, at least
  not the way things are presently set up, is to
  follow Kozlowski and Hattrup's recommendation to
  sever all ties between interrater reliability and
  rWG and to treat rWG as strictly a measure of
  agreement with, in effect, no ties to classic
  measurement theory. It is not possible to follow
  this recommendation because rWG is currently
  derived in terms of classic measurement theory as
  an interchangeability (agreement) index of
  interrater reliability.
• (p. 306)

53
53
53
53
53
53
53
53
54
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)
• J number of items
• What happens to rWG(J) as number of items (J)
  increases?

54
55
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)
• J number of items
• What happens to rWG(J) as number of items (J)
  increases?

55
55
56
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)

56
57
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)

rWG(J) .7
J
57
58
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)
• To get a large rWG(J) (James et al., 1984),
  simply add more items to your scale!!
• Even under near-maximal within-group variance,
• 1.8 rWG(J) .7 when the scale has J
  20 items!

58
59
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)
• mean of observed item-level variances
• What is it?
• First calculate the within-group variance of each
  item,
• Then average these variances across items,

60
rWG(J) for Multi-Item Scales

• mean of observed item-level variances
• Compare vs. (scale score
  variance)
• Scale score variance gt
• First calculate mean across items (i.e., scale
  score),
• Then take the within-group variance of scale
  score,
• is almost always larger than

61
rWG(J) for Multi-Item Scales

• Why is almost always larger than scale score
  variance ?

PC Item 1
d1
Psych. Climate
PC Item 2
d2
PC Item 3
d3
PC Item 4
d4
True Score Variance
Item Unique Variance
62
rWG(J) for Multi-Item Scales

• Why is almost always larger than scale score
  variance ?

PC Item 1
d1
Psych. Climate
PC Item 2
d2
PC Item 3
d3
PC Item 4
d4
True Score Variance
Item Unique Variance
63
rWG(J) for Multi-Item Scales

• mean of observed item-level variances
• Compare vs. (scale score
  variance)
• , Scale score variance gt zooms in on true,
  construct-level
• variance within-groups
• vs.
• , Mean of observed item-level variances gt
  includes true
• construct-level variance item-specific
  variance

64
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)
• mean of observed item-level variances
• It would be much clearer to just base
  within-group agreement on the within-group
  variance in scale scores, rather than on
  the average of item-level within-group variances,
  .

65
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)
• theoretical null variance (represents
  zero agreement)
• E.g., Uniform null distribution
• A number of response options (e.g., A 5 for a
  5-point Likert scale)

66
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)
  
• theoretical null variance
• Can alternatively use a non-uniform expected null
  variance for rWG(J) (see James et al., 1984
  LeBreton Senter, 2008)
• Normal null dist.
• Skewed null dist.
• Maximum null dist. (Brown Hauenstein, 2005)

67
rWG(J) for Multi-Item Scales

• Issues with James et al.s (1984) rWG(J)
  
• theoretical null variance
• Can alternatively use an Average Deviation index
  (AD average absolute value deviation from mean
  or median Burke et al., 1999).
• Less vulnerable to outliers
• Still compared against arbitrary cutoff, AD lt A/6
• Still includes item-specific variance (like
  )

68
rWG(J) for Multi-Item Scales

• Summary
• Whereas rWG is a great index of standardized
  within-group agreement,
• rWG(J) reflects 3 sources of variance
• within-group variance in psych. climate/latent
  construct true scores (shared meaning), plus
• item-specific variance (in ), and
• number of items (J).
• It would be better to use an agreement index that
  homes in on (a) within-group variance in psych.
  climate/latent construct true scores (shared
  psychological meaning).

68
68
68
68
69
Within-Group Agreement

• So what is the alternative?

69
70
Within-Group Agreement

• What if we still want to assess within-group
  agreement (shared psychological meaning) with a
  multi-item climate scale?
• First, conceptualize the degree of shared
  psychological meaning at the latent theoretical
  level (James, 1982 James et al., 1988), but use
  a format similar to rWG

71
Within-Group Agreement

• yWG does not increase as you add items to the
  climate scale (i.e., it is a pure parameter of
  within-group agreement, not reliability)

71
72
Within-Group Agreement

• How well does each of the following within-group
  agreement indices estimate yWG? (shared
  psychological meaning)
• James et al. (1984)
• Lindell et al. (1999)
• Simple index

72
72
73
Within-Group Agreement

• Comparison of rWG(J), rWG(J), and rWG(a)

J 5 items, aWG .90
Newman Sin, 2008
73
73
74
Within-Group Agreement

• Conclusions
• All within-group agreement indices are very
  strongly correlated.
• rWG(J) can notably overestimate within group
  agreement, especially when rWG(J) gt .7.
• rWG(a) seems to offer a closer estimate of within
  group agreement (slight underestimate)
• One could also directly estimate yWG .

74
74
74
75
Within-Group Agreement

• How well does each of the following within-group
  agreement indices estimate yWG? (shared
  psychological meaning)
• When yWG .60
• rWG(J) .75 rWG(J) .38, rWG(a) .56
• When yWG .65
• rWG(J) .81 rWG(J) .46, rWG(a) .61
• When yWG .70
• rWG(J) .85 rWG(J) .53, rWG(a) .67

J 5 items, aWG .90
75
75
75
76
Overview

• Group-Level Psychological Properties?
• Psychological Climate
• Group-Level vs. Individual-Level Constructs
• Aggregation Bias
• Why we need rWG (Within-group agreement)
• Justifying Aggregation
• rWG(J) for multi-item scales
• Agreement vs. Reliability

76
76
77
Overview

• Group-Level Psychological Properties?
• Psychological Climate
• Group-Level vs. Individual-Level Constructs
• Aggregation Bias
• Why we need rWG (Within-group agreement)
• Justifying Aggregation
• rWG(J) for multi-item scales
• Agreement vs. Reliability

77
77
78
Overview

• Group-Level Psychological Properties?
• Psychological Climate
• Group-Level vs. Individual-Level Constructs
• Aggregation Bias
• Why we need rWG (Within-group agreement)
• Justifying Aggregation
• rWG(J) for multi-item scales
• Agreement vs. Reliability

78
78
79
Overview

• Group-Level Psychological Properties?
• Psychological Climate
• Group-Level vs. Individual-Level Constructs
• Aggregation Bias
• Why we need rWG (Within-group agreement)
• Justifying Aggregation
• rWG(J) for multi-item scales
• Agreement vs. Reliability

79
79
80
Overview

• Group-Level Psychological Properties?
• Psychological Climate
• Group-Level vs. Individual-Level Constructs
• Aggregation Bias
• Why we need rWG (Within-group agreement)
• Justifying Aggregation
• rWG(J) for multi-item scales
• Agreement vs. Reliability

80
80
81
Thank You Larry!
81