Survey Design and Cross cultural Issues

1
Survey Design and Cross cultural Issues

• Avinish Chaturvedi
• Carlos Torelli

2
Agenda

• Brief Review of the readings
• Methods and Data Analysis for Cross-Cultural
  Research
• Discussion

3
Heine et al., (2002)

• Reference group effect people from different
  cultural groups use different referents in their
  self-reported values (i.e., compare with
  different others)

• Low Individualism -------------------- High
  Individualism ?
• ------------- Japanese High (7) ------ American
  High (7) ?
• An apparent no difference is actually a
  significant difference

4
Wong et al., (2003)

• Problems with Likert scales that contain a
  mixture of positive-worded items (PWI) and
  reverse-worded items (RWI) in cross-cultural
  research.
• Why use RWI?

• Reduce acquiescence bias.
• General problems with RWI?

• May negatively impact internal consistency.
• May disrupt dimensionality (RWI loading in a
  separate factor).

5
Brislin chapter

• Why do we need to change/modify the survey
  instrument in a different culture
• Linking it with previous discussion
• What factors change across cultures
• Why not to use existing scales

6
Types of cross cultural studies Van de Vijver
chapter

• Generalizability studies
• The design of a generalizability study is
  usually a replication of the original study.
• - When the goals of cross cultural studies are
  defined as both delineating and explaining cross
  cultural differences, generalizability studies
  capitalize on first and often ignore the second.

7
Types of cross cultural studies Van de Vijver
chapter

• Studies of psychological differences
• These are often based on a less elaborate
  theoretical framework than are generalizability
  studies

8
Types of cross cultural studies Van de Vijver
chapter

• Theory driven studies
• - These studies test a theory of cross
  cultural differences. Theory driven studies are
  more systematic. Such studies are designed and
  carried out to critically test a hypothesis.
  These studies provide powerful tets of theories
  of cross cultural differences, which is one of
  the main goals of cross cultural research.

9
Types of cross cultural studies Van de Vijver
chapter

• External validation studies
• - These studies take observed cross cultural
  level or structural differences as their starting
  point and scrutinize these differences either by
  exploring their antecedents or by testing
  interpretations of these differences.

10
Cross cultural issues

• Researchers run the risk of missing aspects of a
  phenomenon as viewed by people in other cultures.
  
• They risk imposing conclusions based on concepts
  which exist in their cultures but which are
  foreign, or at least partially incorrect, when
  used in other culture.

11
Emic and Etic distinction

• How to determine what attributes of a construct
  are static, i.e., unchangeable across cultures
• What a priori measures can be employed in this
  regard
• Example from Brislin chapter Authoritarian and
  conservatism.
• What kind of questions are more sensitive to
  cultural variations?

12
Method and Data analysis

• In cross cultural studies what kind of sampling
  measures should be taken
• Convenience Sampling
• Studies adopting this sampling scheme often
  fall into the category of psychological
  differences studies

13
Method and Data analysis

• Systematic Sampling
• Cultures are selected in a systematic,
  theory guided fashion. These studies usually fall
  into the categories of theory driven or
  generalizability studies. Cultures are selected
  in this procedure because they represent
  different values on a theoretical continuum
• Random Sampling
• It involves the sampling of a large number
  of cultures randomly. This strategy is preferable
  for generalizability studies, in which a
  universal structure or a pan cultural theory is
  evaluated.

14
Summary
15
Method and Data analysis

• Choosing Cultures
• How does choice of cultures in a research
  design changes when
• a. we are looking for similarities
• b. we are looking for differences.

16
Method and Data analysis

• Administration of instruments
• tester/interviewer
• testee/interviewee
• interaction between two
• response procedures
• stimulus materials

17
Method and Data analysis

• Remedies
• A priori and post priori techniques.
• Similarly prior and post hoc remedies can be
  used to alleviate problems of sample
  incomparability.
• Application of a monotrait multimethod
  matrix in order to examine the influence of
  response procedures is useful
• What are the other ways through which this
  effect can be minimized??

18
Method and Data analysis

• Changing the survey instrument
• Application
• Adaptation
• Assembly
• In which scenario, each of above three technique
  becomes useful?

19
Method and Data analysis

• Validity Enhancement
• The major criterion in the choice of
  application, adaptation and assembly is the type
  of bias expected. If there are serious concerns
  that construct bias could play a role, adaptation
  or assembly should be chosen.
• Indigenization which aims at maximizing the
  appropriateness of psychological theories and
  instruments to local cultures, will often amount
  to the assembly of new instruments

20
Method and Data analysis

• Obtaining linguistically equivalent instruments
• Translation and back translation
• Committee approach

21
Method and Data analysis

• Translation
• Translation is more than producing text in
  another language. Translators should know or be
  made aware of the linguistic and cultural
  differences that could influence responses to
  translated or adapted instruments.

22
Translation

• Guidelines for translating and adapting
  psychological and educational instruments
• What are the other translation issues involved

23
Study 1 Experts Ratings

• Sample Japanese specialists living in North
  America
• What were the questions?

• How much items in Singelis (1994) scale
  characterize Japanese or Americans

• Findings
• Experts were in agreement with the common view
  for all 23 items (high face validity).

• These findings suggest that the use of different
  reference groups can obscure cross-cultural
  differences (Americans use other Americans, and
  East Asians other East Asians).

24
Study 2 Manipulation of Reference Groups

• Sample Canadian biculturals with knowledge about
  Japan and Canada.
• Instruments

• Singelis (1994) in standard form.
• Answers in comparison with most Japanese.
• Answers in comparison with most Americans.
• Findings

• Weak support for the common view using standard
  format.
• When comparing with people from the other
  culture strong support for the common view.
• When comparing with people from their own
  culture evidence from acculturation.

25
Study 3 Within-culture

• Sample Asian Canadians vs. European Canadians,
  and returnee Japanese vs. Japan-bound Japanese
• Common referent for each group.

• Findings

• European Canadians scored higher in independence
  and lower on interdependence than Asian Canadians
• Returnee Japanese scored higher on independence
  and lower on interdependence than Japan-bound
  Japanese

26
Discussion

• Likert scales capture ones feelings relative to
  a comparison group or shared norm?
• Is it always the case? The case of self-esteem?
• When can Likert scales assess ones feelings
  without a referent? When do people use less
  social comparison for answering (i.e.,
  introspection)?

• Implications from the reference-group effect

• Use of subjective Likert scales most valid for
  identifying differences within rather than
  between groups ? What do you think?

• Remmedial measures
• Measure at the cultural level
• Prompt comparisons with an standard (average
  peer)
• Responses to information that is
  consistent/inconsistent with implicit theories
  (new information)
• Use of behavioral measures
• Items with concrete, objective response options

27
The MVS

• The MVS in cross-cultural settings
• Good reliability and validity in the U. S.
• Questionable reliability and validity abroad.
• Why the problems outside the U. S.?

• Negations and contradictions can be confusing
• Languages can employ different ways of making
  negation or contradiction.
• Differences in how PWI and RWI are interpreted
  (cultural norms regarding agreeableness?)

28
MVS across cultures Study 1

• Countries the U. S., Singapore, Japan, Korea,
  and Thailand.
• Prediction
• Negative correlation with life satisfaction in
  wealthy countries only.
• Positive correlation with individualism across
  all countries
• Findings

• MVS specified as a 2nd. Order factor with 3
  dimensions (happiness, centrality, and success)
  showed a poor fit ? systematic error?
• PWI and RWI correlated in the U.S., but weakly or
  uncorrelated elsewhere ?RWI being responded
  differently?
• Divergent correlations for materialism and life
  satisfaction

• Revised Model include two method factors ? fit
  comparable to those obtained in the U.S., but
  construct validity still an issue
• Support for the notion that RWI might have led to
  the poor initial performance

29
MVS across cultures Study 2

• Objectives

• Assess conceptual meaning of materialism.
• Show that other scales that use PWI and RWI have
  the same problems.
• Evaluate alternative scales ? interrogative format

• Predictions
• Show conceptual equivalence by using adjectives
• Other scales that include RWI should show
  two-factors structures
• Inquiring about a respondents position on an
  issue would minimize agreement for the sake of
  being polite

• What do you think about the way conceptual
  equivalence was assessed?

• Does it assess the 3-dimensions?

• What does the evidence that other scales show two
  factors mean?

• Does it prove that agreeableness is the
  underlying issue? What about the findings in the
  general discussion?

30
Discussion

• Acquiescence vs. substantive cultural
  differences?
• Are the results driven by acquiescence or not?
• Are the meanings of the MVS items different
  across cultures due to religion and beliefs? If
  so, what is the explanation for the results with
  the other 4 scales?
• Can there be any referent group effect?

• Recommendations
• Use only PWI
• Use interrogative formats
• Which one is better?