GroupLevel Measurement

1
Group-Level Measurement

• Katherine Klein
• University of Pennsylvania
• kleink_at_wharton.upenn.edu
• CARMA Presentation
• February 2007

2
Why Group-Level Measurement?

• Burgeoning of multilevel theory and research in
  last 25 years
• Great progress in conceptualizing and measuring
  group-level constructs
• Especially shared constructs
• Continuing challenges and opportunities
• Especially regarding configural constructs

3
A Few Terms and Assumptions

• Ill refer to groups but much or all of what I
  say will apply as well to organizations,
  departments, stores, etc.
• Ill focus on the creation and use of original
  survey measures to assess group constructs.
• Ill address statistical issues in passing only.
• But see past CARMA presenters including James
  LeBreton, Gilad Chen, Paul Bliese, Dan Brass,
  Steve Borgatti, and others

4
Roadmap

• Fundamentals Theory First
• Construct Types Global, Shared, and Configural
  Constructs
• Practicalities and Technicalities
• Survey Wording
• Sampling
• Qualitative Groundwork
• Single-source Bias
• Justifying Aggregation
• Opportunities and Challenges
• The Configuration of Diversity
• Social Network Analysis

5
Fundamentals Theory First

• Constructs are our building blocks in developing
  and in testing theory.
• High quality measures are construct valid.
• The development of construct valid measures thus
  begins with careful construct definition.
• Group-level constructs describe the group as a
  whole and are of three types (Kozlowski Klein,
  2000)
• Global, shared, or configural.

6
Global Constructs

• Relatively objective, easily observable,
  descriptive group characteristics.
• Originate and are manifest at the group level.
• Examples
• Group function, size, or location.
• No meaningful within-group variability.
• Measurement is generally straightforward.

7
Shared Constructs

• Group characteristics that are common to group
  members
• Originate in group members attitudes,
  perceptions, cognitions, or behaviors
• Which converge as a function of attraction,
  selection, socialization, leadership, shared
  experience, and interaction.
• Within-group variability predicted to be low.
• Examples
• Group climate, norms, leader style.
• Measurement challenges are well understood.

8
Configural Group-Level Constructs

• Group characteristics that describe the array,
  pattern, dispersion, or variability within a
  group.
• Originate in group member characteristics (e.g.,
  demographics, behaviors, personality, attitudes)
• But no assumption or prediction of convergence.
• Examples
• Rates, diversity, fault-lines, social networks,
  team mental models, team star or weakest member.
• Measurement challenges are less well understood.

9
A Related Framework Chans (1988) Composition
Typology

• Shared Constructs
• Direct consensus models (e.g., group norms)
• Referent shift models (e.g., team efficacy)
• Configural Constructs
• Dispersion model (e.g., climate strength)
• Additive models (e.g., mean group member IQ)
• Multilevel, Homologous Models
• Process model (e.g., efficacy-performance
  relationship)

10
Construct Definition ComplexitiesAn Example
Shared Leadership

• Shared leadership
• A dynamic, interactive influence process among
  individuals in work groups in which the objective
  is to lead one another to the achievement of
  group goals It involves peer, or lateral,
  influence and at other times involves upward or
  downward hierarchical influence
• Conger Pearce, 2003, p. 286
• Is this a shared construct, or a configural
  construct, or ?

11
Construct Definition ComplexitiesAn Example
Shared Leadership

• Well, how would you measure it?
• Shared team leadership as a shared construct
• Team members share in the leadership of this
  team.
• Many team members provide guidance and direction
  for other team members.
• Shared team leadership as a configural construct
  (network density)
• To what extent do you consider _____ an informal
  leader of the team?

12
Construct Definition ComplexitiesAn Example
Shared Leadership

• Calling it a referent shift construct is not
  the answer.
• Referent shift is a measurement strategy, not a
  construct type
• Shifting the referent in an unthinking manner can
  be quite problematic
• The members of my team
• Express confidence that we will achieve our
  goals
• Will recommend that I am compensated more if I
  perform well
• Are friendly and approachable
• Rule with an iron hand

13
A Quick Recap

• Theory first Define and explain the nature of
  your group-level constructs.
• Is it a clearly objective description of the
  group?
• If yes, a global construct.
• Do you expect within-group agreement?
• If yes, a shared construct.
• Does it describe the group in terms of the
  pattern or array of group members on a common
  attribute?
• If yes, a configural construct.

14
Now What?

• Having defined your constructs, the goal is to
  create measures that
• Are construct valid
• Show homogeneity within (shared constructs)
• Show variability between (all group-level
  constructs)
• Practicalities and technicalities
• Survey wording
• Sampling
• Qualitative groundwork
• Minimizing single-source bias
• Testing for aggregation

15
Survey WordingGlobal Constructs

• Draw attention to objective descriptions of each
  group.
• Gather data from experts and observers (SMEs) who
  can provide valid information about the groups in
  question.
• No need to gather data from individual
  respondents within groups
• Use language that fits your sample.

16
Survey Wording Shared Constructs

• Draw attention to shared group characteristics
• Use a group referent rather than individual
  referent to enhance
• Within group agreement
• Between group variability
• Predictive validity
• Gather data from individual respondents so
  within-group agreement can be assessed.
• Actual consensus methods (discussion prior to
  group survey completion) work well but are
  labor-intensive.

17
Survey WordingConfigural Constructs

• Draw attention to individual group member
  characteristics by using an individual referent.
• Gather data from experts and observers (SMEs) who
  can provide valid information regarding
  individual group members, or gather data from
  individual respondents within groups.
• The challenge is perhaps less in the survey
  wording than in operationalizing the array or
  pattern of interest.

18
Sampling

• Substantial between-group variability is
  essential. Seek samples in which groups vary
  considerably on the constructs of interest
• Whether they are global, shared, or configural.
• Statistical power reflects both
• Group sample size (n of groups)
• Within-group sample size
• When group size is large (number of respondents
  per group), measures of shared constructs are
  more reliable.
• More research needed on power in multilevel
  analyses.

19
Qualitative Groundwork

• The survey wording and sampling guidelines seem
  fairly obvious and easy, but
• Check your assumptions in the field prior to
  survey data collection.
• Are you measuring the right groups?
• Example Grocery stores or departments?
• Is there meaningful between-group variability?
• Example Fast food chain
• Are you measuring the right variables, and not
  too many of them?
• Beware the blob.

20
Single-Source Bias

• Group-level correlations between measures of
  shared group constructs may be disturbingly high.
• Examples
• Transformational and transactional leadership
• Task, emotional, and procedural conflict
• Aggregation does not average away response
  biases.
• Rather, group members may share response biases
• Halo, logical consistency, social desirability
• Response bias may be particularly influential
  when respondents must make subtle distinctions
  among constructs.

21
Single-Source BiasBeating the Blob

• Survey measures
• Choose and measure truly distinct constructs
• Use different survey response formats
• Survey design
• Keep survey items measuring distinct constructs
  separate.
• Help respondents recognize the distinction
  between leadership types, or conflict types, for
  example.

22
Single-Source BiasBeating the Blob

• Survey analysis
• Randomly split the within-group sample of
  respondents during data analysis.
• All receive the same survey, but half provide IV
  and the other half provide the DV for analyses
• Survey administration
• Randomly split the within-group sample of
  respondents during data administration.
• Respondents receive distinctive surveys. Half
  receive the IV survey and the other half receive
  the DV survey.

23
A Quick Recap

• Having
• Defined our constructs
• Written our survey items
• Conducted qualitative groundwork
• Sampled appropriately
• Taken steps to reduce single source bias
• Were almost ready for hypothesis testing
• But first We need to justify aggregation

24
Justifying Aggregation

• Why is this essential?
• In the case of shared constructs, our very
  construct definitions rest on assumptions
  regarding within- and between-group variability.
• If our assumptions are wrong, our construct
  theories, our measures, and/or our sample are
  flawed and so are our conclusions.
• So, test both
• Within group agreement
• The construct is supposed to be shared, but is it
  really?
• Between group variability (reliability)
• Groups are expected to differ significantly, but
  do they really?

25
Justifying Aggregation rwg(j)

• Developed by James. Demaree, Wolf (1984)
• Assesses agreement in one group at a time.
• Compares actual to expected variance.
• Answers the question
• How much do members of each group agree in their
  responses to this item (or this scale)?
• Highly negatively correlated with the within
  group standard deviation
• Valid values range from 0 to 1
• Rule of thumb rwg(j) of .70 or higher is
  acceptable

26
Justifying Aggregation rwg

• Common to report average or median rwg(j) for
  each group for each variable
• If rwg(j) is below .70 for one or more groups,
  check
• Does the group have low rwg(j) values on several
  variables?
• Do many groups have low rwg(j) values on this
  variable?
• Remember rwg(j) indicates within-group
  agreement, not between-group variability.
• Beware When variance in a group exceeds
  expected variance, out of range rwg(j) result.

27
Justifying Aggregation h2

• Assesses between-group variance relative to total
  variance, across the entire sample.
• Based on a one-way ANOVA
• Answers the question
• To what extent is variability in the measure
  predictable from group membership?
• The F-test provides a test of significance
• The larger the sample of individuals, the more
  likely eta2 is to be significant.
• Beware h2 may be inflated when group sizes are
  small (under 25 individuals per group)
• But, this is an easy way to begin tests of
  aggregation

28
Justifying Aggregation ICC(1)

• Assesses between-group variance relative to total
  variance
• Based on a one-way ANOVA
• Answers the question
• To what extent is variability in the measure
  predictable from group membership?
• The F-test provides a test of significance
• Based on h2 but controls for the number of
  predictors relative to the total sample size, so
  ICC(1) is not biased by group size.

29
Justifying Aggregation ICC(2)

• Assesses the reliability of the group means
  (i.e., between-group variance) in a sample, based
  on ICC (1) and group size.
• Answers the question
• How reliable are between-group differences on the
  measure?
• Reflects ICC(1) and within-group sample size
• Example If ICC(1) .20 and
• Mean group size is 5, expected ICC(2) .56
• Mean group size is 20, expected ICC(2) .71

30
Justifying Aggregation An Example
31
A Quick Recap

• The hope is that we have successfully
• Defined our constructs.
• Written our survey items.
• Conducted qualitative groundwork.
• Collected data from a large sample of groups.
• Taken steps to reduce single source bias.
• Justified aggregation.
• And moved on to test our hypotheses.
• So, what remains?

32
Opportunities and ChallengesThe Configuration
of Diversity

• Configural constructs describe the array,
  pattern, dispersion, or variability within a
  group.
• The easy example is diversity
• Demographic diversity
• Climate strength
• But even the easy example isnt so easy What is
  the definition of diversity? And how should it
  be measured?

33
The Configuration of Diversity

• A starting definition of diversity
• The distribution of differences among the members
  of a group with respect to an attribute, X, such
  as age, ethnicity, conscientiousness, positive
  affect or pay.
• Okay, but whats maximum diversity?
• Which team has maximum age diversity?
• 20, 20, 20, 70, 70, 70
• 20, 30, 40, 50, 60, 70
• 20, 20, 20, 20, 20, 70
• 20, 70, 70, 70, 70, 70

34
The Configuration of Diversity

• Diversity isnt one thing.
• Its three things Separation, Variety, or
  Disparity
• The three types differ in
• Meaning or substance
• Pattern or shape
• Likely consequences
• Appropriate operationalization
• Blurring across these distinctions leads to fuzzy
  theory, misguided operationalizations, and
  potentially invalid research conclusions

35
The Configuration of DiversityExample Three
Research Teams

• Team S
• Members differ in their view of qualitative
  research.
• Half of the team members respect it, half dont.
• Team V
• Members differ in their discipline.
• 1 psychologist, 1 sociologist, 1 anthropologist,
  etc.
• Team D
• Members differ in their rank
• 1 senior professor, others are incoming graduate
  students.

36
Diversity as Separation

• Differences in group members position, attitude,
  or opinion along a continuum
• Min Every member has the same opinion
• Max Two polarized extreme factions
• Theory Similarity-attraction
• Operationalization Standard deviation

37
Diversity as Variety

• Differences in kind or category
• Min Every member is the same type
• Max Each group member is a different type
• Theory Requisite variety, cognitive resource
  heterogeneity
• Operationalization Blaus index of categorical
  differences

38
Diversity as Disparity

• Differences in concentration or proportion of
  valued assets or resources
• Min Every member has an equal portion of the
  resource
• Max One member is rich and all others are
  impoverished
• Note Disparity is asymmetric
• Theory Inequality, relative deprivation,
  tournament compensation
• Operationalization Coefficient of variation
  (SD/Mean)

39
The Configuration of DiversityA Recap

• Theory first
• Separation is about position, attitude, or
  opinion
• At maximum Polarized factions
• Variety is about knowledge or information.
• At maximum One of a kind
• Disparity is about resources or power.
• At maximum One towers over others
• Operationalize accordingly
• The coefficient of variation is not a default or
  catch-all

40
Opportunities and Challenges Social Network
Analysis

• Multilevel analysis and social network analysis
  have developed along separate paths.
• Rich opportunities for cross-fertilization.
• Social network analysis provides a means to
  conceptualize and operationalize configural
  constructs.
• Illuminating the pattern or array of
  interpersonal ties within a group

41
Opportunities and Challenges Social Network
Analysis

• Many of our shared constructs appear to rest on
  tacit, often fuzzy, assumptions about
  interpersonal ties with groups.
• Examples Cohesion, communication, coordination,
  knowledge sharing, shared leadership, conflict
• But we know little about the configuration of
  interpersonal ties the structures that
  underlie our shared constructs and measures.

42
An Example Social Network Analysis and Shared
Team Conflict

• When teams report high task or emotional
  conflict, what is the structure of interpersonal
  ties within the team?
• As a starting point
• How dense are positive (advice) ties?
• How dense are negative (difficulty) ties?

43
An Example Social Network Analysis and Shared
Team Conflict

• Task and emotional conflict The blob
• r .83
• Advice density and negative tie density More
  weakly correlated
• r -.36
• Task conflict (mean task and emotional conflict),
  advice density, and negative tie density
• Team Conflict and Advice Density r -.47
• Team Conflict and Difficulty Density r .40

44
Negative Tiesin a Low Conflict Team
45
Negative Ties in a High Conflict Team
46
Advice Ties in a High Conflict Team
47
Advice Ties in a Low Conflict Team
48
Social Network AnalysisA Recap

• Social network analysis illuminates the
  configuration of interpersonal ties in groups.
• What network structures underlie our shared
  constructs and measures?
• Do network measures provide incremental validity?
• Not just density, but centralization, cliques,
  and more.
• What explains between-group differences in
  network structures?

49
In Conclusion

• Theory first. Define your constructs.
• Are they global, shared, or configural?
• Measure constructs and collect data with care
• Match item wording to the construct
• Conduct qualitative groundwork
• Sample appropriately
• Take steps to reduce single source bias
• Test for aggregation
• Studying configural constructs remains a
  challenge and an opportunity
• Conceptualizing and measuring diversity
• Integrating social network analysis within our
  arsenal

50
Some Helpful References

• Bliese, P. D. (2000). Within-group agreement,
  non-independence, and reliability Implications
  for data aggregation and analysis. In K. J. Klein
  S. W. J. Kozlowski (Eds.), Multilevel theory,
  research and methods in organizations (pp.
  349-381). San Francisco Jossey-Bass.
• Borgatti, S. P. (2003). The network paradigm in
  organizational research A review and typology.
  Journal of Management, 29, 991-1013.
• Chan, D. (1998). Functional relations among
  constructs in the same content domain at
  different levels of analysis A typology of
  composition models. Journal of Applied
  Psychology, 83, 234-246.
• Harrison, D. A. Klein, K. J. (2007). Whats
  the difference? Diversity as separation,
  variety, or disparity in organizations. Academy
  of Management Review.
• Harrison, D. A. McLaughlin, M. E. (1996).
  Structural properties and psychometric qualities
  of organizational self-reports Field tests of
  connections predicted by cognitive theory.
  Journal of Management, 22, 313-338.
• James, Demaree, Wolf, G. (1984). Estimating
  within-group interrater reliability with and
  without response bias. Journal of Applied
  Psychology, 69, 85-98.

51
Some Helpful References

• Klein, K. J., Conn, A. B., Smith, B., Sorra, J.
  S. (2001). Is everyone in agreement? An
  exploration of within-group agreement in employee
  perceptions of the work environment. Journal of
  Applied Psychology, 86, 3-16.
• Klein, K. J., Conn, A. B. Sorra, J. S. (2001).
  Implementing computerized technology An
  organizational analysis. Journal of Applied
  Psychology, 86, 3-16.
• Kozlowski, S. W. J. Klein, K. J. (2000). A
  multilevel approach to theory and research in
  organizations. In Klein, K. J. Kozlowski, S. W.
  J. (Eds.), Multilevel theory, research, and
  methods in organizations (pp. 3-90). San
  Francisco Jossey-Bass.
• Morgeson, F. P. Hofmann, D. A. (1999). The
  structure and function of collective constructs
  Implications for multilevel research and theory
  development. Academy of Management Review, 24,
  249-265.
• Ostroff, C., Kinicki, A. J., Clark, M. A.
  (2002). Substantive and operational issues of
  response bias across levels of analysis An
  example of climate-satisfactoin relationships.
  Journal of Applied Psychology, 87, 355-368.