University of Michigan

1
14th International GALA conference, Thessaloniki,
14-16 December 2007
Behavioural scales of language proficiency
insights from the use of the Common European
Framework of Reference Spiros Papageorgiou

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
2
Outline

• Background
• Aims
• Data collection
• Data analysis
• Results
• Implications

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
3
Background

• Advent of the CEFR increased interest in
  behavioural scales of language proficiency
• Using the CEFR scales Problems
• Designing test specifications (Alderson et al.,
  2006)
• Measuring progression in grammar (Keddle, 2004)
• Describing the construct of vocabulary (Huhta
  Figueras, 2004)
• Designing proficiency scales (Generalitat de
  Catalunya, 2006)

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
4
Background (2)

• Using the CEFR scales Criticism
• Equivalence of tests constructed for different
  purposes (Fulcher, 2004bWeir, 2005)
• Danger of viewing a test as non valid because of
  not claiming relevance to the CEFR (Fulcher,
  2004a)
• Progression in language proficiency not based on
  SLA research but on judgements by teachers (cf.
  North 2000 North Schneider 1998)

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
5
Aims of the study

• Investigation of three research questions
• Can users of the CEFR rank-order the scaled
  descriptors in the way the appear in the 2001
  volume?
• If differences in scaling exist between the users
  of the CEFR and the 2001 volume, why does this
  happen?
• Can training contribute to more successful
  scaling?

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
6
Data collection

• 12 users of the scales acting as judges in
  relating two language examinations to the CEFR
• Data collected during Familiarisation sessions
  described in the Manual for relating examinations
  to the CEFR
• Part of a doctoral thesis at Lancaster University
  (Papageorgiou, 2007) and a research project at
  Trinity College London
• Task sort descriptors into the six levels

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
7
Data collection (2)
University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
8
Data analysis

• Analysis FACETS Rasch computer program
• 3 facets descriptors-raters-occasions
• Rank-ordering of elements of facets on a common
  scale
• Fit statistics (Bond and Fox, 2001 McNamara,
  1996)
• Overfit too predictable pattern
• Misfit more than expected variance
• Acceptable range of fit statistics
• Descriptors .4-1.2 (Linacre Wright, 1994)
• Raters .5-1.5 (Weigle, 1998)

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
9
Results Writing Levels A1-B1
University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
10
Results Writing Levels B2-C2
University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
11
Results Raters
University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
12
Results Occassions
University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
13
Results Correlations
Correlations of scaling between the judges and
the CEFR volume
University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
14
Summary of results

• Trained judges perceived language ability as
  intended in the CEFR
• Almost identical scaling
• Cut-offs between B2-C1 and C1-C2 unclear
• Competences other than linguistic misfitting
  descriptors
• Unclear and inconsistent wording resulted in
  level misplacement by the judges
• Mixed effect of training

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
15
Implications of findings

• Common understanding of the construct in the CEFR
  scales can be achieved, but
• How valid is it to claim that a test is linked to
  B2 instead of C1 and C1 instead of C2?
• How can sociolinguistic and strategic competences
  be tested in relation to the CEFR?
• Can SLA research help better understand these
  issues?

University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli
16
Contact details
Spiros Papageorgiou University of Michigan
English Language Institute 500 East Washington
Street Ann Arbor, MI 48104-2028 USA spapag_at_umich.e
du
University of Michigan English Language
Institute Testing and Certification
Division www.lsa.umich.edu/eli