The CEFR and common standards for international language examinations a QMS approach Copenhagen 2006 Nick Saville University of Cambridge ESOL Examinations

1
The CEFR and common standards for international
language examinations a QMS approach
Copenhagen 2006Nick SavilleUniversity of
Cambridge ESOL Examinations

2
Aims of ALTE from 1990 onwards

• To establish common levels of proficiency in
  order to promote the transnational recognition of
  certification in Europe
• To establish common standards for all stages of
  the language-testing process
• To collaborate on joint projects and in the
  exchange of ideas and know-how
• Attaining standards sustaining diversity

3
1 Common Levels of Proficiency - ALTE

• 1990 onwards - the ALTE 5-level framework
• .to establish common levels of proficiency in
  order to promote the transnational recognition of
  certification in Europe

4
Common Levels of Proficiency - ALTE

• The process of co-locating ALTE members exams,
  on a framework through
• extensive content analysis
• guidelines for writing test items
• empirically validated performance indicators
  Can Do statements
• glossary of testing terms
• These projects were supported by EU funding

5
Common Levels of Proficiency ALTE/CEFR

• 1991-1996 the Common European Framework of
  Reference (CEFR) project
• Aims transparency and coherence
• 1997 first draft of CEFR sent out for
  consultation
• 1998-2000 the ALTE Can Do Project
• 2001 publication of CEFR
• ALTE adopted 6-level CEFR framework A1 C2

6
John Trim on the CEFR

• What we were aiming at was something which will
  be a common reference point that people working
  in different fields and people using it for
  entirely different things and in different ways
  could refer to in order to feel that they were
  part of a common universe.
• Interview with John Trim, Language Assessment
  Quarterly, 2005

7
2 Common (professional) standards

• 1994 - ALTE published its first Code of Practice
• It was a broad statement of what the users of the
  examinations should expect
• Focused on the roles and responsibilities of
  stakeholders in striving for fairness

8
Developing the ALTE Code of Practice

• Striving for fairness for stakeholders of the
  examinations
• The Code of Practice identifies the roles of
  three groups of stakeholder in the testing
  process
• the examination developers - e.g. members of
  ALTE
• the examination takers - primary users - who take
  the examinations by choice, direction or
  necessity
• the examination users secondary users - who
  require the examination for some decision-making
  or other purpose

9
Developing the ALTE Code of Practice

• The Code of Practice lays down four broad areas
  of responsibility
• developing examinations
• interpreting examination results
• striving for fairness
• informing examination takers
• Striving for fairness is a shared responsibility
  involving all stakeholders

10
ALTE Standards and Principles of Good Practice

• A supplementary document ALTE Standards -
  Principles of Good Practice for ALTE Examinations
  -1993, updated 2001
• This document sets out in more detail the
  principles which ALTE members should adopt in
  order to achieve high professional standards
• (cf. Standards of American Psychological
  Association, etc.)

11
The ALTE Principles of Good Practice

• Based on VRIP features
• Validity
• Reliability
• Impact
• Practicality
• Plus additional feature
• Quality of service
• These features must be balanced to produce useful
  tests which are fit for purpose

12
A Quality Management System for ALTE

• Since 2000
• COP Working Group putting the principles into
  practice
• Revised documents COP as self-evaluation
  checklist

13
Applying the ALTE QMS 2001 to 2006

• Establish desired outcomes and impacts within
  each member organisation
• Move towards good practice
• Aim at continuous improvement
• Discuss and agree on minimum standards but
  establish best practice models as long-term
  target
• Monitor quality through self-assessment
• Seek confirmation that standards are being met
• e.g. through professional peer review within ALTE

14
ALTE QMS

• Revise the Code of Practice as checklists
• Apply the checklists to the 4 aspects of the Test
  Development and Administration Cycle
• Test Design and Construction
• Administration
• Processing - marking, grading, issue of results
• Analysis and Review

15
ALTE QA Checklists - Units 1 to 4
16
A Quality Profile

• Based on the Code of Practice and QMS checklists,
  17 minimum standards have been agreed to
  establish a Quality Profile for an exam or suite
  of exams
• Members are required to explain whether the
  examination meets these standards, and if so, in
  what way
• Evidence to back up the argument is also required
• Evidence-based test validation

17
17 Minimum Standards examples 1 to 5
18
The Auditing Process

• Members are required to build up an argument that
  the quality profile is sufficient and appropriate
  for a particular test or suite of tests
• Different tests are used in different contexts by
  different groups of test users. There is no
  intention to impose a single set of quality
  profiles across the ALTE group
• The audit has both a quality control and
  consultancy function
• It aims to establish that minimum standards are
  met in a way that is appropriate to the context
  of the test and to offer recommendations towards
  best practice where there is room for improvement

19
Implementing the Auditing Process 2006 to 2007

• A guide has been developed for auditors and the
  first draft is available to Members (presented
  here in Copenhagen)
• The auditing process has recently been tried out
  for the Danish Consortium, the Goethe Institut,
  and WBT
• All Members are now committed to auditing - first
  full round of audits to be completed by Summer
  2007

20
How to set ALTE Quality Standards?
21
The role of the Manual Relating Language
Examinations to the CEFR

• An alignment process based on 3 sets of
  procedures
• specification of content and purpose
• standardisation of interpretation
• empirical validation studies
• ALTE supporting this work
• Authoring
• Piloting and Case Studies
• Sample Materials
• Involvement in Ad Hoc Project Group

22
Reference Summary of ALTE CoP and QMS

• Research Notes 22
• November 2005
• www.cambridgeesol.org/rs_notes