Raili Hild

1
Raili Hildén University of Helsinki

• Relating the Finnish School Scale to the CEFR

2
Starting point

• The construction of a comprehensive, transparent
  and coherent framework for language learning and
  teaching does not imply the imposition of one
  single uniform system. On the contrary, the
  framework should be open and flexible, so that it
  can be applied, with such adaptations as prove
  necessary, to particular situations. (CEFR,
  2001, p. 7)

3
How was the Finnish school scale (FSS) designed?

• Decision to adapt/adopt the CEFR levels was made
  by the language experts invited to be members of
  a national curriculum development working group
  approved by the supervising board
• Sources consulted
• CEFR scales for multiple communicative
  activities
• Canadian Benchmarks
• Teacher judgement
• Working group members commented on the drafts

4
How was the school scale designed?

• Several versions were produced based on internal
  feedback
• First round of empirical check for inter- rater
  consistency (by the team members)
• Re-formulation of descriptors with low
  agreement/consistency
• Second round of empirical validation
• Agreeing on the current formulations

5
Proficiency levels (and their labels) in Finnish
language curricula http//www.oph.fi
A1 Basic communication in the most familiar situations
A1.1 First stage of elementary proficiency
A1.2 Developing elementary proficiency
A1.3 Functional elementary proficiency
A2 Communication in basic social situations and simple description
A2.1 First stage of basic proficiency
A2.2 Developing basic proficiency
B1 Communication in everyday life
B1.1 Functional basic proficiency
B1.2 Fluent basic proficiency
B2 Coping with regular relationships with native speakers
B2.1 First stage of independent proficiency
B2.2 Functional independent proficiency
C1-C2 Proficient language use in demanding contexts
C1.1 First stage of skilled proficiency
6
Categories included in the Finnish language
curricula

• Listening comprehension
• Themes, text and tasks (1)
• Conditions and constraints (2)

• Speaking
• Themes, texts and tasks (monologue and
  interaction)
• Fluency
• Pronunciation
• Linguistic range
• Linguistic control

• Reading comprehension
• Themes, texts and tasks (2)
• Conditions and constraints (1)

• Writing
• Themes, texts and tasks (2)
• Linguistic range
• Linguistic control

7
Research Questions

• RQ1. What is the level of agreement between
  judges on the CEFR level of the FSS descriptors?
• RQ2. How are the Finnish syllabus descriptors
  related to CEFR levels?

8
Design

• FSS level descriptors were split up into 184
  statements
• Listening 38, Speaking 66, Reading 31 Writing 49
  statements
• The statements were coded and grouped in terms of
  communicative activities
• Criterion scales used in the rating of FSS
  descriptors were selected from among relevant CEF
  scales

9
Design

• A sample of 40 Finnish language teaching experts
  were contacted by an e-mail questionnaire
• 20 experts returned the questionnaire
• A randomised selection of statements referring to
  each of the four communicative activities was
  e-mailed to the raters (in Finnish translation by
  Huttunen Jaakkola 2003)

10
Conducting the research
Listening comprehensiona (38 descriptors) Raters Raters
Listening comprehensiona (38 descriptors) Sent to Returned by
1. Themes and textsb (13) 40 20
2. Tasks and activities (14) 16 8
3. Conditions and constraints (11) 15 10
Notes a/ CEFR scale used for the rating task-
Overall listening comprehension b/ Anchor
descriptors, rated by all raters
11
Speaking (66 descriptors) CEFR scale Raters Raters
Speaking (66 descriptors) CEFR scale Sent to Returned by
1. Texts, themes and tasks (21) Overall spoken interaction 40 20
2. Fluency (10) Spoken fluency 14 6
3. Pronun-ciation (10) Phonological control 15 8
4.Linguistic range (14) Vocabulary range 15 9
5. Linguistic control (11) Grammatical accuracy 15 10
12
Conducting the research
Reading comprehensiona (31 descriptors) Raters Raters
Reading comprehensiona (31 descriptors) Sent to Returned by
1. Themes and textsb (10) 40 20
2. Tasks (11) a 14 6
3. Conditions and constraints (10) 14 7
4. Tasks and activities c 12 7
Notes a/ CEFR scale - Overall reading
comprehension b/ Anchor descriptors, rated by all
raters c/ CEFR scale Reading for information and
argument
13
Writing (49 descriptors) CEF scale Raters Raters
Sent to Returned by
1. Texts and themes (13) Creative writing 40 20
2. Tasks and activities (12) Overall written interaction 15 10
3. Linguistic range (11) General linguistic range 15 7
4. Linguistic control (13) Grammatical accuracy 15 7
5. Texts and themes (12) Overall written production 15 6
14
FSS level codes CEFR level codes
A11 - 1 A12 - 2 A13 - 3 A21 - 4 A22 - 5 B11 - 6 B12 - 7 B21 - 8 B22 - 9 C11 -10 A1 - 1 A2 - 2 A2 - 3 B1 - 4 B1 - 5 B2 - 6 B2 - 7 C1 - 8 C2 - 9
15
RQ1.The range of raters agreement
Range
Note Ranges of 7 and 8 were checked after the
presentation and detected to be due to clerical
errors. Thus the true range is From 0 to 6.
16
RQ 1.Agreement with the CEF levelsExample of a
descriptor with a low level of agreement between
raters.Can identify the writers bias and the
purpose of the text and locate and integrate
several specific pieces of information in a
longer text. Can quickly identify the content and
relevance of new items deciding whether closer
study is worthwhile
Level
17
RQ 1.Agreement with the CEF levels Example of a
descriptor with a high level of agreement between
raters. Can write the alphabet of the language
and all numbers and numerals. Can write down
basic personal identification information and
write a small number of familiar words and simple
phrases.
18
RQ2. Agreement between syllabus level descriptors
(syllcode) and the original CEFR levels (levcode)
Note A quite good level of agreement was oberved
(65). There is, however, some tendency for an
overestimation 19 descriptors (10) were
assigned to a lower level while 46 (25) were
assigned to higher level.
19
RQ1. Range distribution per skill
The average range is 2.66 for the whole pool. The
range distribution per skill can be seen in the
boxplot above, showing no clear differences
between the four skills.
20
RQ2. Absolute agreement per skill between
syllabus level and CEF level
21
RQ1. Factors affecting the ratings ( raters are
quite homogeneous)
22
RQ2. Agreement between individual rating and
original (initial) levels (Syllabus - syll CEF
- level)
23
Plans for further exploration

• Calibrating the FSS descriptors
• Exploring the link to the Canadian Benchmarks in
  more detail
• Re-formulating or removing problematic
  descriptors
• Empirical validation and exemplication of the FSS
  scales through benchmarks for comprehension tasks
  and for oral and written performance samples

24
Summary

• The correspondence between the new Finnish school
  scale (FSS) and the CEFR scales were studied.
• Research question 1 What is the level of
  agreement between judges on the CEFR level of the
  FSS descriptors?
• Research question 2 How are the Finnish syllabus
  descriptors related to CEFR levels?
• 20 experienced raters judged FSS descriptors
  using relevant CEFR scales
• A good agreement was reached 65 of the FSS
  descriptors were assigned to the original CEFR
  levels. For the rest of the cdescriptorts, some
  tendency of overestimation was observed.
• Inter-rater agreement was also quite good.

25
Acknowledgement

• I wish to thank
• Sauli Takala, University of Jyväskylä
• Feljanka Kaftandieva, University of Sofia
• for their help in carrying out this study.