Adapting Credentialing Exams for Use in Multiple Languages and Cultures

1
Adapting Credentialing Exams for Use in Multiple
Languages and Cultures

• Ronald K. Hambleton
• University of Massachusetts

2
Three Reasons for Translating and Adapting Exams

• 1. Knowledge and skills of interest are often
  the same across language groups--exam adaptation
  ensures consistency of content and structure.
• 2. Often more efficient to translate and adapt
  an exam than to build a new exam.
• 3. Easier to insure exam equivalence and
  fairness (than construct new exam in a second
  language).

3
Goals of the Presentation

• 1. To describe several myths about test
  adaptation.
• 2. To review steps in the test adaptation
  process.
• 3. To present the ITC guidelines for test
  translation/adaptation.

4
Test Translation vs. Test Adaptation?

• Test Adaptation is more descriptive of the
  process that usually takes placedirections,
  formats, contexts, etc.
• Test Translation is sometimes only a small part
  of the process.

5
Examples of Test Adaptations

• Intelligence, Aptitude, and Personality Tests
• Quality of Life Surveys
• TIMSS, OECD/PISA, and More
• SAT, GED, NAEP, State Assessments
• Credentialing Exams

6

• Example 1
• Out of sight, out of mind
• (Back translated from French)
• invisible, insane

7
Example 2 (IEA Study in Reading)

• Are these words similar in meaning?
• Pessimistic -- Sanguine

8

• Pessimistic -- Sanguine
• Adapted to
• Pessimistic -- Optimistic

9
Example 3 (1995 TIMMS Pilot)

• Alex reads his book for 1 hour and then used a
  book mark to keep his place. How much longer will
  it take him to finish the book?
• A. ½ hour
• B. 2 hours
• C. 5 hours
• D. 10 hours

10
Example 4

• Parker Pen in Mexico It wont leak in your
  pocket and embarrass.
• Became
• It wont leak in your pocket and make you
  pregnant.

11
Four Common Myths About Adapting Credentialing
Exams

• 1. Know two languages and you can be a
  translator.

12
Selection and Training of Translators

• Knowledgeable in the languages
• Knowledgeable in the cultures
• Knowledgeable in the subject matter
• Knowledgeable in the principles of test
  development, item writing, and scoring rubrics,
  etc.

13
Four Common Myths About Adapting Credentialing
Exams

• 2. A good translation guarantees validity.

14
Four Common Myths About Adapting Credentialing
Exams

• 3. Judgmental reviews are sufficient to identify
  problems in a exam translation/adaptation.

15
Four Common Myths About Adapting Credentialing
Exams

• 4. The common strategy of a back-translations
  design and the use of a bilingual design to
  compile empirical data is sufficient to justify
  exam use.

16
8 Steps for Adapting Exams

• 1. Checking exam content and format equivalence.
• -content may not be generalizable
• -item formats are not universally used

17
8 Steps for Adapting Exams

• 2. Deciding on the desirability of a
  translation/adaptation.
• -unsuitability of content and/or the exam
  format?
• -technical expertise to produce and validate a
  new exam?

18
8 Steps for Adapting Exams

• 3. Choosing translators.
• -Four qualifications given earlier are
  important.
• -Some technical expertise may be especially
  helpful.

19
8 Steps for Adapting Exams

• 4. Translating and adapting the credentialing
  exam.
• -Should often be more than a single translator.
• -Several designs and variations available.

20
Judgmental Designs for Assessing Suitability
of a Translation/Adaptation

• 1. Backward Adaptations
• 2. Forward Adaptations

21
Backward Translation Design

• Advantages
• 1. Some problems can be identified.
• 2. Researchers do not need to be bilingual.
• Disadvantages
• 1. No review of target language version!

22
Forward Translation Design

• Advantages
• 1. Judgments made directly about language
  versions of an exam.
• Disadvantages
• 1. Bilingual judges make assessments of an exam
  for monolingual examinees.

23
8 Steps for Adapting Exams

• 5. Reviewing the adapted version of the exam.
• -Multiple reviewers are often very valuable.
• -Smoothing of language is possible at this
  step.

24
8 Steps for Adapting Exams

• 6. Conducting a small tryout of the adapted exam
• -item analysis, reliability analysis, candidate
  review, etc.

25
8 Steps for Adapting Exams

• 7. Carrying out a more ambitious study of the
  adapted exam (that is, checking for exam
  validity).
• -construct equivalence, method equivalence, item
  equivalence

26
Statistical Designs

• 1. Bilingual Designs
• 2. Monolingual Designs
• A. Source Language Examinees Take Original
  and Backward Adapted Versions
• B. Monolingual Examinees Take Versions in
  Their Own Languages

27
Checking for Construct Equivalence
28
(No Transcript)
29
Checking for Item Equivalence Delta Plots
30
Checking for Item Equivalence IRT Methods
(change in difficulty)
31
IRT Methods (change in discrimination)
32
(No Transcript)
33
Checking for Method Equivalence

• 1. Any biases due to directions, exam format,
  scoring?
• 2. Preparation of exam administrators?
• 3. Equivalence of exam taking skills?

34
8 Steps for Adapting Exams

• 8. Documenting the process.
• -Steps in exam adaptation, details, evidence for
  validity.

35
International Test Commission Guidelines for
Adapting Educational and Psychological
InstrumentsPrepared ByInternational Test
Commissionand Seven International
Organizations(EAPA, ETP, IACCP, IAAP, IEA, ILTA,
IUPsyS)
36
Instrument Adaptation Guideline

• A practice which is judged as important for
  conducting and evaluating the adaptation or
  parallel development of psychological and
  education instruments for use in different
  populations.

37
Organization of the 22 ITC Guidelines

• Context (2)
• Instrument Development and Adaptation (10)
• Administration (6)
• Documentation/Score Interpretations (4)

38
Organization of Each of the 22
ITC Guidelines

• Description of the Guideline
• Rationale/Explanation
• Steps to Meet the Guideline
• Common Errors
• References for Additional Study

39
D.3

• Test developers/publishers should provide
  evidence that the choice of testing techniques,
  item formats, test conventions, and procedures
  are familiar to all intended populations.

40
D.8

• Test developers/publishers should provide
  information on the evaluation of validity in all
  target populations for whom the adapted versions
  are intended.

41
D.9

• Test developers/publishers should provide
  statistical evidence of the equivalence of
  questions for all intended populations.

42
Three Main Conclusions

• 1. Has progress been made in test adaptation
  methodology?
• -Definitely!
• -Moved from single unqualified translator and
  bilingual design to considerably more
  sophisticated methodologies.

43
Conclusions, cont.

• -Emergence of ITC guidelines methodological
  advances in judgmental designs and statistical
  methods (e.g., DIF, SEM)more resources and time
  being allocated.

44
Conclusions, Cont.

• 2. What lies ahead for the field of test
  adaptations?
• -Im very optimistic about the future.
• -Needed is commitment of resources and time, and
  implementation of existing methods.

45
Conclusions, Cont.

• 3. What are the most important areas for
  improvement?
• -choose multiple qualified translators and give
  them sufficient time
• -aggressively implement empirical methods
• -build on past experiences and knowledge base

46

• For more information about exam translation and
  adaptation, contact Ron Hambleton at
• Rkh_at_educ.umass.edu