Developing CEF-related proficiency scales for the Escoles Oficials d

1
Developing CEF-related proficiency scalesfor
the Escoles Oficials dIdiomes

• Neus Figueras
• nfiguera_at_pie.xtec.es
• www.xtec.es/eoi

2
The EOI system in Catalonia

• State funded language schools (16)
• There are two levels defined
• Certificat de Cicle Elemental (B1 ?)
• Certificat dAptitud (B2 ?)
• 13 different languages
• Standardised language proficiency certificates in
  place since 1995
• The exams are
• developed centrally
• related to published curricula
• written according to set specifications
• submitted to pre-testing
• standardised administration and marking
• analysed statistically

3
Issues to be solved

• Revise EOI certificate tests after 10 years.
• Develop an item banked system.
• Link levels to Common European Framework levels
  (Manual).
• Is examination difficulty equivalent across time?
• Is the lower certificate consistently easier than
  the higher certificate?
• Are the different certificates in the different
  languages comparable?

4
To sum up
Teaching/learning context
5
The EOI project Outline 2003-06

• July 2003 2004
• Define objectives
• Project design
• Empirical scale development
• Scale validation
• Item banking (English) starts
• July 2004 2005
• Publication of scales
• Item banking (English) continues
• Linkage to CEF Procedures in the Manual
  (specification)
• July 2005 2006
• Confirmation of linkage
• standardisation
• empirical validation
• comparability across languages

6
And..... CEF adoption or adaptation?

• A scale, like a test, has validity in relation
  to contexts in which it has been shown to work
  (200122)

7
Scale Development

• Step 1 selecting level descriptors (per skill)
• CEF
• Curriculum objectives for EOI
• Finnish scales (YKI)
• Step 2 translation of the descriptors into
  Catalan
• Step 3 mapping descriptors onto levels (CEF and
  EOI)
• Experts assign levels to descriptors
• Analysis of sorting exercises
• Collating resulting surviving descriptors
• Step 4 developing new scales
• Drafting
• Validation
• Scale revision
• Validation
• Final version

8
Sorting tasks

• Descriptors
  Teachers Languages
• Reading 40 103 10
• Listening 46 99 10
• Speaking 80 92 12
• Writing 53 89 12
• Grammar 34 81 12
• Vocabulary 61 73 12
• Arabic, Basque, Catalan, Dutch, English
  (gt40), French (gt20), German (gt15), Italian,
  Japanese, Portuguese, Russian, Spanish

9
Analysis of sorting exercises (Nnumber of
judges/teachers)

• Listening
• Reliability
• B1?

10
Comparing language groups Reading1-English
2-French 3-German 4-Mixed

11
Language groups an outlier
Can understand clearly written, straightforward
instructions for a piece of equipment.
German
English
French
12
Reading range

• Listening
• Reliability
• B1?

13
Reading range - examples

• n.23. Can find specific information in simple
  everyday texts.
• A1-14 A2-82 B1-7
• n.14. Can use the context, linguistic and non
  linguistic, as a means to increase comprehension.
• A1-9 A2-17 B1-28 B2-13 C1-11 C2-8

14
Reading descriptors analysis
Range90
Median
Score
15
Drafting the scales

• Using descriptors preferably with range 0-2
• Following guidelines Appendix A CEF
• Positiveness
• Definiteness
• Clarity
• Brevity
• Independence

16
Checking for

• Regular progression from band to band
• Overall coherence and consistency
• Conditions
• Limitations

17
Validating the scales

• Because the whole is different from its parts.
• To guarantee
• the consecutive order of band descriptors
• the degree of separation between them.
• To establish the link between EOI scales and CEF
  scales
• How ?
• Pair comparison exercise comparing two scales
  (CEF and EOI), 66 pairs of descriptors per skill.
• Read carefully each pair of descriptors and
  choose the one that describes a higher level of
  language proficiency.

18
How EOI teachers perceive CEF scales

19
Which is B2 and which is C1? (Spoken interaction)

• Can express him/herself fluently and
  spontaneously, almost effortlessly. Has a good
  command of a broad lexical repertoire allowing
  gaps to be readily overcome with circumlocutions.
  There is little obvious searching for expressions
  or avoidance strategies only a conceptually
  difficult subject can hinder a natural, smooth
  flow of language.
• Can use the language fluently, accurately and
  effectively on a wide range of general, academic,
  vocational or leisure topics, marking clearly the
  relationships between ideas. Can communicate
  spontaneously with good grammatical control
  without much sign of having to restrict what
  he/she wants to say, adopting a level of
  formality appropriate to the circumstances. Can
  interact with a degree of fluency and spontaneity
  that makes regular interaction, and sustained
  relationships with native speakers quite possible
  without imposing strain on either party. Can
  highlight the personal significance of events and
  experiences, account for and sustain views
  clearly by providing relevant explanations and
  arguments.

20

• C1 Can express him/herself fluently and
  spontaneously, almost effortlessly. Has a good
  command of a broad lexical repertoire allowing
  gaps to be readily overcome with circumlocutions.
  There is little obvious searching for expressions
  or avoidance strategies only a conceptually
  difficult subject can hinder a natural, smooth
  flow of language.
• B2 Can use the language fluently, accurately and
  effectively on a wide range of general, academic,
  vocational or leisure topics, marking clearly the
  relationships between ideas. Can communicate
  spontaneously with good grammatical control
  without much sign of having to restrict what
  he/she wants to say, adopting a level of
  formality appropriate to the circumstances. Can
  interact with a degree of fluency and spontaneity
  that makes regular interaction, and sustained
  relationships with native speakers quite possible
  without imposing strain on either party. Can
  highlight the personal significance of events and
  experiences, account for and sustain views
  clearly by providing relevant explanations and
  arguments.

21
How EOI teachers perceive EOI scales
22
Where is the problem? (EOI Reading)

• C1 Can understand long complex texts and
  contemporary literary prose, being able to
  appreciate subtle differences regarding meaning,
  register and tone. Nuances conveyed by rare
  expressions may cause problems in comprehension.
• C2 Can understand with ease virtually all forms
  of the written language, although highly
  idiomatic expressions may still cause some
  difficulty.
• C1 Can understand with reasonable ease long and
  complex texts of different types, including
  contemporary literary prose, and appreciate
  subtle differences regarding meaning, register
  and tone. Highly idiomatic expressions still
  cause some problems in comprehension.
• C2 Can understand with ease virtually all forms
  of the written language, even complex literary
  texts, identifying the affective and evaluative
  content and interpreting it critically. Only
  occasionally may nuances conveyed by some rare or
  highly idiomatic expressions cause some
  difficulty.

23
Linking CEF and EOI scales
24
Comparing EOI and CEF levels (Voc)

• CEF - C1 Has a good command of a broad lexical
  repertoire allowing gaps to be readily overcome
  with circumlocutions little obvious searching
  for expressions or avoidance strategies. Good
  command of idiomatic expressions and
  colloquialisms.
• EOI C1 Can understand and use a broad lexical
  repertoire, including synonyms and antonyms of
  less common words, idiomatic expressions, to
  express himself/herself correctly and
  appropriately on a wide range of general,
  academic, professional and leisure topics. Can
  understand word play based on multiple meanings
  and use prefixes and suffixes to form less common
  words. Less frequent words may cause some effort
  in comprehension.

25
Linking CEF and EOI scales
26
Comparing EOI and CEF levels (Voc)

• EOI B1 Can understand and use a sufficient
  vocabulary to express himself/herself with a good
  command on topics related to his/her everyday
  life and to common concrete situations and needs.
  Lexical gaps may cause hesitation or make
  circumlocution necessary. Errors may occur when
  expressing more complex thoughts or when handling
  new or less common topics or situations.
• CEF - B1 Has a sufficient vocabulary to express
  him/herself with some circumlocutions on most
  topics pertinent to his/her everyday life such as
  family, hobbies and interests, work, travel and
  current events.
• EOI - A2 Can understand and use a basic
  lexical repertoire, including antonyms and
  synonyms of most frequent words. Can infer the
  meaning of some words outside his limited lexical
  repertoire. Vocabulary errors or shortage of
  words may cause interruptions and
  misunderstandings.

27
Lessons learntimplications

• The CEF as a catalyst
• Reading is not understanding
• Continued training is necessary
• Appearances need to be checked empirically
• Exact correspondence may not be possible, but is
  it desirable?
• Linkage does not mean equivalence

28
Work continues

• The EOI scales as a point of reference for
• curriculum objectives
• reporting learner achievement
• reporting exam levels
• To meet the need for
• Improving test quality
• Item banking
• Setting standards
• Validating established standards

29
References

• EOI scales and a full description of their
  development can be found at
• www.xtec.es/eoi
• Council of Europe (2001) Common European
  Framework of Reference for Languages Learning,
  teaching, assessment. Cambridge. CUP. (available
  on line at http//www.coe.int/T/E/Cultural_Co-ope
  ration/education/Languages/Language_Policy )
• Council of Europe (2003) Manual for relating
  language examinations to the CEF. A preliminary
  pilot version. Strasbourg. Language Policy
  Division. DGIV/EDU/LANG
• Kaftandjieva, F. and S. Takala (2002) Council of
  Europe Scales of Language Proficiency A
  Validation Study. In Alderson, J.C.(ed), Common
  European Framework of Referencefor Languages
  Learning, Teaching, Assessment. Case Studies,
  106-129. Strasbourg Council of Europe
  Publishing.
• North, B. and Schneider, G. (1998) Scaling
  descriptors for language proficiency scales.
  Language Testing 15, 2 217-262