Perspectival Diversity and Consensus Analysis

1
Perspectival Diversity and Consensus Analysis

• John Gatewood . . . . . . Lehigh University
• John Lowe . . . . . . . Cultural Analysis Group
• AAA Meetings, Philadelphia, Dec 5, 2009

2
Preview

• INTRODUCTORY REMARKS
• Problem of culture-sharing (and non-sharing)
• Basic patterns of inter-informant agreement
  addingperspectival diversity to the list
• OUR CURRENT STUDY
• Assessing effects of different distributional
  patterns on consensus analysiss key indicators
• Some initial findings
• CONCLUSIONS
• Methodological lessons for researchers
• Future directions

3
INTRODUCTORY REMARKS
4
Problem of Culture-Sharing

• By definition, culture is socially transmitted
  knowledge hence, it must be shared but
  sharing is always a matter of degree
• Hence, two related issues for any given cultural
  domain
• How much knowledge is shared? (the AVERAGE
  cultural competence)
• How is the knowledge socially distributed? (the
  DISTRIBUTIONAL PATTERN)
• KEY INSIGHT assess degree of culture-sharing by
  examining patterning of inter-informant agreement

5
Basic Patterns of Agreement

• Boster (1980, 1985) four basic patterns of
  agreement (paraphrasing expanding)
• UNIFORM agreement traditional view of culture
• RANDOM agreement free variation ? no culture
• EXPERTISE gradient experts tend to agree with
  one another whereas non-experts deviate
  randomly
• SUBCULTURAL variation more than one school
  of thought
• Competing answer sets different groups,
  different truths
• Complementary knowledge different groups
  systematically know different things
• Romney, Weller Batchelder (1986) cultural
  consensus theory consensus analysis

6
Perspectival Diversity a 5th Pattern

• Pilot study of credit union employees (Gatewood
  Lowe 2006)
• No consensus in sample
• No identifiable subcultural groups
• gt Fish-scale overlappings of partial
  knowledge perspectival diversity
• i.e., social interaction and knowledge among the
  employees was rather departmentalized their
  understandings of credit unions reflected what
  they needed to know to perform their own jobs,
  not necessarily what might be relevant to other
  people
• _______________________
• Pilot studys conclusion about no
  consensus turned out to be an artifact of our
  failure to counter-balance items in the
  questionnaire form (see Gatewood Lowe 2008)
  but thats another story

7

• To generalize, perspectival diversity occurs
  when
• All individuals have limited knowledge with
  respect to a given domain and to approximately
  the same degree
• Each individuals range of knowledge only
  partially overlaps with the ranges known by
  others
• And, consistent with this definition, different
  geometries of perspectival diversity are
  possible e.g., circular pattern, linear
  pattern, taxonomic-hierarchical, overlapping
  polygons on a surface, etc.

8
OUR CURRENT STUDY
9

• RESEARCH QUESTION
• Ceteris paribus, do the different distributional
  patterns affect the key indicators of consensus
  analysis?
• As the average knowledge in a sample varies, do
  different distributional patterns show
  consensus more readily than other patterns?
• Do some distributional patterns mask cultural
  consensus when other patterns reveal it?
• If NO ? nothing to worry about yippee!
  
• If YES ? distributional patterning has an
  independent effect that needs to be taken into
  account when interpreting results of consensus
  analyses

10

• ITEM FORMAT
• Counter-balanced Likert-style questions, i.e.,
  6-point strongly agree to strongly disagree
  response scale because these are so common in
  survey research
• ANALYSES
• Such data can be analyzed two ways
• INFORMAL MODEL of consensus analysis i.e.,
  input to factor analysis is a Resp x Resp
  correlation matrix (data treated as
  interval-scale)
• FORMAL MODEL of consensus analysis i.e., input
  to factor analysis is a chance-corrected
  agreement matrix (data treated as nominal-scale,
  e.g., dichotomized responses)

11
Research Design with average knowledge and
distributional pattern as manipulated variables
Distributional Patterns Key Indicators Key Indicators Key Indicators variety of other measures
Distributional Patterns Ratio of eigenvalues Mean 1st factor loading Number of negative loadings variety of other measures
Uniform-to-Random ? ? ? ?
Expertise ? ? ? ?
Subcultures ? ? ? ?
Perspectival ? ? ? ?
12
Theoretical Predictions
Distributional Pattern Prediction
Uniform-to-Random ? None serves as benchmark for other patterns
Expertise Gradient ? INCREASE consensus indicators
Subcultures ? DECREASE consensus indicators
Perspectival ? ???
13
Implementation

• How to experimentally manipulate key parameters
  for different distributional models while holding
  others constant ?? computer simulation to the
  rescue !
• See Excel data-generating file Excel
  findings file

14
SOME INITIAL FINDINGS
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
Key Findings

• Distributional pattern has an independent effect
  with respect to consensus indicators
• w/r/to RATIO OF EIGENVALUES
• ( compared to the Uniform-to-Random model )
• Expertise patterns INCREASE the ratio
• Subcultural patterns DECREASE the ratio
• Perspectival patterns DECREASE the ratio
• w/r/to MEAN 1st FACTOR LOADING
• Distributional patterns have little effect on
  this indicator, AND consensus analysis estimates
  actual competence very well with one exception
• Expertise (triangular) pattern INFLATES mean
  competence as well as the ratio of eigenvalues
  (because it violates the homogeneity of items
  assumption?)

27

• Expertise (rectangular) pattern
• The range of expertise about the same average
  competence also makes a difference greater
  range ? larger ratio of eigenvalues
• Subcultural patterns
• As expected, systematic differences in sub-group
  knowledge undermine consensus
• By question sub-groups may still show consensus
  overall, with the groups showing up on the 2nd
  factor
• Different answer keys just destroy consensus
• Formal consensus model (on dichotomized data)
  and informal consensus model yield very similar
  results

28
CONCLUSIONS
29
Lessons for Researchers

• Since the ratio of eigenvalues is particularly
  sensitive to the distributional pattern of
  knowledge, REPORT MORE than just the ratio
• Minimally, include
• Ratio of 1st to 2nd eigenvalues
• Mean 1st factor loading (and st.dev. of those
  loadings)
• Number of negative loadings
• And, comparable guidelines should be
  established for evaluating these additional
  measures e.g., 0.500 for mean loading fewer
  than 5 negative loadings in sample
• These output statistics are necessary for more
  meaningful interpretations of ones data

30

• IF your data show a hefty mean 1st factor loading
  but a low ratio of eigenvalues DO NOT leap to
  the conclusion that either (a) subcultures exist
  or (b) there is free variation in the domain
• You may be dealing with a case of PERSPECTIVAL
  DIVERSITY which would warrant further
  investigation, such as examining the inter-person
  correlation matrix and the response-profiles of
  individuals one at a time to see if you can
  detect a subtle social patterning to
  who-knows-what

31

• Try to formulate questions that are EQUALLY
  DIFFICULT ( and ask lots of questions )
• Violations of Assumption 3 will inflate both the
  obtained ratio of eigenvalues the mean 1st
  factor loading
• e.g., Expertise (triangular) pattern INFLATES
  both indicators
• So ex post factoif you notice that some
  questions were much easier than others, then
  either (a) use higher threshold criteria before
  claiming the data conform to the cultural
  consensus model, and/or (b) remove the very
  easy questions and re-analyze

32
Future Directions

• Developing additional geometries of
  perspectival overlapping
• Analyzing relations between a variety of measures
  describing the initial Resp x Resp correlation
  matrix and the key indicators from consensus
  analysis
• Exploring different instantiations of guessing
  (binomial, truncated-normal, beta distributions)
• Exploring other possible measures from the factor
  analysis as predictors of culture-sharing, e.g.,
  1st eigenvalue divided by sample size

33
Thank you

• and we would be happy to continuetalking with
  interested folksafter the session

34
Scalar data analyzed via Informal Method
(vertical axis)VSDichotomized data analyzed via
Formal Method (horizontal axis)
Mean 1st factor loadings( r .969 )
Ratios of eigenvalues( r .933 )
35
Real Survey Item
Real Survey Item
Simulated Item
POTENTIAL PROBLEM Frequency distributions of
itemsfrom real surveys (top panels)are more
graded than oursimulated data (lower
right) something were trying to resolve,but
not there yet