Important%20Exams%20and%20the%20Concept%20of%20Good%20Practice

1
Important Exams and the Concept of Good Practice

• Dávid Gergely

2
Interpret focus of lecture

• Hardly feasible task to cover all important exams
• Many websites, information booklets, etc. provide
  the information anyway, for interested fellow
  professionals
• New focus about
• quality of examinations
• what may be considered good professional
  practice.
• Tools for fishing a series of criterial
  questions

3
What can you do?

• Study test specifications and other printed info
  packs, mock exams and any published docs.
• Study regulations
• Carry out an observation task, collect
  information.

4
Studying the specifications 1

• Is the target language sampled widely enough?
• The nature of foreign lang. Proficiency
• Skills, communicatve language activities,
  abilities
• Are all four skills grammar/vocab tested?
• Litmus test Is listening comprehension tested?

5
Studying the specifications 2

• Is the candidate required to produce a large
  sample of the target language?
• Are the papers long enough?
• Not enough to include 5-6 papers, but there must
  be enough items etc. in each to give candidates
  as many chances (fresh new starts, fairness) to
  show what they know.
• In terms of our litmus listening test, how many
  items based on how many sound tapes (hangzóanyag)
  is good enough?
• 20 items based on 3 sound tapes is minimum.

6
Explanation

• Not covering enough skills (subskills),
  activities, abilities and/or doing that through
  too few items is called the underrepresentation
  of the test construct a typical problem with
  many tests.
• The difficulty of inferring underlying
  proficiency from test proficiency.
• The quest for information
• Student 1 1,0,1,0,0,1,0,0,1,1
• Student 2 1,2,1,0,2,0,1,1,0,2

7
Studying the specifications 3

• Static specs of papers, tasks types, etc. Do not
  speak about what the test provider do to ensure
  quality
• Is the testing process specified too?
• Procedural validity

8
Model testing process test material

• Drafting tasks
• Moderation
• Revision of drafts
• Pretesting draft tasks
• Revision 2
• Administration
• Post-test quality control
• Score calculation
• Standard (level) setting

9
Model testing process examiner

• Training
• Benchmarking
• Administration
• Post-test quality control
• Score calculation
• Standard (level) setting

10
Study regulations, observe practice

• Is the model testing procedure followed?
• If it is not followed in every respect, how do
  they make up for it?

11
Applying the procedure to ensure validity

• There must be a quality control mechanism that
  assumes, especially
• Candidate response data (all such data) are
  collected systematically
• Response data are analysed statistically after
  the pretest, but most importantly the post-test,
  in order to decide how good the items were, how
  well the raters marked, whether intended level
  was maintained, etc.
• Are response data collected systematically?
• Are the data analysed statistically?

12
Illustrations of data problems

• How is the oral scored?
• Two raters added up? Averaged?
• Raters agree? Separately given scores recorded?
• Is it possible to put together, candidate by
  candidate, scores from the written papers with
  scores from the oral?

13
Data? Analysis? Why the hell?

• The Holy Grail of language testing
• Want to find out candidates lang. proficiency
• Not directly measurable
• Must use tools (items, raters, criteria, i.e.
  test method) to make that proficiency observable
• Whatever the method, it will unavoidably
  influence test performance and results
• Unknown proficiencymethod effect scored test
  performance

14
3232,1,63-74,1,1,1,1,1,1,1,1,0,1,1,1 3233,1,63-74,
1,1,1,1,1,1,0,1,1,0,1,1 3234,1,63-74,0,0,0,0,0,0,0
,0,0,0,0,0 3235,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 32
36,1,63-74,0,0,0,1,0,1,1,1,0,0,0,0 3237,1,63-74,1,
1,1,1,1,1,1,1,1,1,1,1 3238,1,63-74,1,1,0,1,1,1,0,1
,0,1,1,1 3239,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3240
,1,63-74,0,1,1,1,0,1,1,0,1,0,1,0 3241,1,63-74,0,1,
1,1,0,0,1,1,0,0,0,0 3242,1,63-74,1,0,1,1,1,1,1,1,1
,1,1,1 3243,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3244,1
,63-74,0,1,1,1,1,1,0,1,0,1,1,1 3245,1,63-74,1,1,1,
1,1,1,1,1,1,1,1,1 3246,1,63-74,1,1,1,1,1,1,1,1,1,1
,1,0 3247,1,63-74,0,0,1,1,0,0,0,0,0,1,0,0 3248,1,6
3-74,0,0,0,0,0,0,0,0,0,0,0,0 3249,1,63-74,1,1,1,1,
1,1,1,1,1,1,1,0 3250,1,63-74,1,0,1,0,1,1,1,1,1,1,1
,0 3251,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3252,1,63-
74,1,1,1,1,1,1,1,1,1,1,1,1 3253,1,63-74,1,1,1,1,1,
1,1,1,1,1,1,1 3254,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0
3255,1,63-74,0,1,0,1,0,1,0,1,1,0,0,0 3256,1,63-74
,1,1,1,1,1,1,0,1,1,1,0,0 3257,1,63-74,0,1,0,0,1,1,
0,0,1,0,1,1 3258,1,63-74,0,0,0,1,1,0,0,1,0,1,0,0 3
259,1,63-74,0,0,0,0,1,0,1,1,0,1,1,0 3260,1,63-74,0
,1,1,1,0,1,1,1,1,0,1,0 3261,1,63-74,0,1,1,0,1,0,0,
1,1,1,1,0 3262,1,63-74,0,0,0,1,1,0,0,0,1,0,0,0 326
3,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3264,1,63-74,1,1
,1,1,1,1,0,1,1,1,1,0 3265,1,63-74,1,1,1,1,1,1,1,1,
1,1,1,1 3266,1,63-74,1,1,0,1,1,1,0,1,1,1,1,1 3267,
1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3268,1,63-74,1,1,1
,1,1,1,1,1,1,1,1,1 3269,1,63-74,0,1,1,1,0,1,1,1,1,
1,0,1 3270,1,63-74,1,0,0,1,1,1,1,1,0,0,0,0 3271,1,
63-74,1,1,1,1,1,1,1,1,1,1,1,0 3272,1,63-74,0,1,1,1
,1,1,0,1,0,0,0,0 3273,1,63-74,0,0,0,1,0,1,0,1,1,1,
1,0 3274,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3275,1,63
-74,1,1,1,1,1,1,0,1,1,1,1,1 3276,1,63-74,1,1,1,1,1
,1,1,1,1,1,1,0 3277,1,63-74,0,1,0,1,1,1,0,1,1,1,0,
0 3278,1,63-74,1,1,0,0,1,1,0,1,0,0,1,0 3279,1,63-7
4,1,1,0,1,0,1,1,0,1,0,1,0 3280,1,63-74,1,1,1,1,1,1
,1,1,1,1,1,0 3281,1,63-74,1,1,1,1,1,1,0,1,0,1,0,0
3282,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3283,1,63-74,
1,1,0,1,1,1,1,1,0,1,1,1 3284,1,63-74,0,0,0,1,0,0,0
,1,0,1,0,0 3285,1,63-74,1,1,1,1,1,1,0,1,1,0,0,1 32
86,1,63-74,0,1,1,1,0,0,1,0,1,1,1,0 3287,1,63-74,0,
0,1,1,1,1,1,0,1,0,1,0 3288,1,63-74,1,0,1,1,1,1,0,1
,1,1,1,0 3289,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3290
,1,63-74,0,0,0,1,1,1,1,1,0,0,0,0 3291,1,63-74,0,1,
0,1,1,0,1,1,1,1,0,0 3292,1,63-74,1,0,1,1,1,0,0,1,1
,0,1,0 3293,1,63-74,1,1,1,1,1,0,1,1,0,1,1,0 3294,1
,63-74,1,1,0,1,1,0,0,1,0,0,1,0 3295,1,63-74,0,1,1,
1,1,1,0,1,1,1,1,0 3296,1,63-74,0,1,1,1,0,1,1,1,1,1
,1,1 3297,1,63-74,0,0,0,1,0,1,1,1,0,0,1,0 3298,1,6
3-74,0,1,0,1,1,1,0,1,0,1,0,0 3299,1,63-74,1,1,1,1,
1,1,1,1,1,1,1,0 3300,1,63-74,1,1,1,1,1,1,1,1,1,1,0
,1 3301,1,63-74,1,0,1,1,1,1,1,1,0,1,0,0 3302,1,63-
74,1,1,1,0,1,1,1,1,1,1,1,1 3303,1,63-74,1,1,1,1,1,
1,1,1,1,1,1,0 3304,1,63-74,1,0,0,1,1,1,0,1,1,0,1,1
3305,1,63-74,1,0,1,1,1,0,1,1,1,0,0,0 3306,1,63-74
,1,1,1,1,1,1,0,1,1,0,0,0 3307,1,63-74,0,0,1,1,0,1,
0,1,0,1,0,0 3308,1,63-74,0,1,0,1,0,0,0,1,1,0,0,0 3
309,1,63-74,1,1,1,1,1,1,1,1,1,1,0,1 3310,1,63-74,0
,1,0,1,1,1,0,1,0,1,1,0 3311,1,63-74,0,0,0,1,0,1,0,
1,0,1,0,1 3312,1,63-74,1,1,1,1,1,1,1,1,0,1,1,0 331
3,1,63-74,1,1,1,1,1,1,0,1,0,0,1,0 3314,1,63-74,0,0
,0,1,0,1,0,1,0,1,0,0 3315,1,63-74,1,1,1,1,1,1,1,1,
1,1,1,1 3316,1,63-74,0,0,0,1,0,0,0,1,0,1,0,0 3317,
1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3318,1,63-74,1,1,1
,1,1,1,0,0,1,1,0,0 3319,1,63-74,0,0,0,1,0,0,0,1,0,
0,1,1 3320,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3321,1,
63-74,1,1,1,1,1,1,1,1,1,1,1,1 3322,1,63-74,0,1,0,1
,1,1,1,1,0,1,1,1 3323,1,63-74,1,1,1,1,0,1,0,1,1,1,
1,1 3324,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3325,1,63
-74,0,0,0,0,0,0,0,0,0,0,0,0 3326,1,63-74,0,0,0,0,0
,0,0,0,0,0,0,0 3327,1,63-74,0,0,0,0,0,0,0,0,0,0,0,
0 3328,1,63-74,0,1,0,0,0,1,1,1,0,1,1,0 3329,1,63-7
4,1,1,1,1,1,1,0,0,1,0,1,0 3330,1,63-74,0,1,0,1,1,1
,1,1,0,0,0,0 3331,1,63-74,0,1,0,0,1,1,0,1,0,1,0,1
3332,1,63-74,1,1,1,1,1,1,0,1,1,0,0,0 3333,1,63-74,
0,1,1,1,0,1,1,1,1,1,1,1 3334,1,63-74,1,1,1,1,1,1,1
,1,1,1,1,1 3335,1,63-74,1,1,0,1,1,1,1,1,0,1,1,1 33
36,1,63-74,1,1,0,1,1,1,1,1,0,1,1,1 3337,1,63-74,1,
0,1,0,1,1,1,1,1,0,1,0 3338,1,63-74,1,1,1,1,1,1,1,1
,1,1,0,0 3339,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3340
,1,63-74,0,0,0,1,1,0,0,0,1,1,0,0 3341,1,63-74,1,0,
1,0,1,1,1,1,1,1,1,1 3342,1,63-74,1,1,1,1,1,1,0,1,1
,0,1,0 3343,1,63-74,0,1,1,1,0,1,0,1,0,1,0,1 3344,1
,63-74,1,1,0,1,0,1,0,1,1,1,0,1 3345,1,63-74,0,0,0,
0,0,0,0,0,0,0,0,0 3346,1,63-74,1,1,0,1,0,1,1,1,0,1
,1,0 3347,1,63-74,0,1,1,1,0,0,0,1,1,1,1,0 3348,1,6
3-74,0,0,0,0,0,1,0,0,0,1,0,0 3349,1,63-74,1,1,1,1,
1,1,0,1,1,1,1,0 3350,1,63-74,1,1,1,1,1,1,1,1,1,1,1
,0 3351,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3352,1,63-
74,1,1,1,1,1,1,0,1,0,1,1,1 3353,1,63-74,0,1,1,0,1,
1,0,1,1,1,1,0 3354,1,63-74,1,1,1,1,1,1,0,1,0,0,0,0
3355,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3356,1,63-74
,0,0,1,1,1,0,0,1,1,0,1,0 3357,1,63-74,1,1,1,0,1,1,
0,1,1,1,1,1 3358,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3
359,1,63-74,1,1,1,1,1,1,0,1,0,1,1,0 3360,1,63-74,1
,1,1,1,1,1,1,1,1,1,1,1 3361,1,63-74,1,1,1,0,1,1,0,
1,1,1,1,0 3362,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 336
3,1,63-74,0,1,0,0,1,1,1,1,1,1,1,1 3364,1,63-74,1,1
,1,1,0,1,1,1,1,0,1,1 3365,1,63-74,0,0,0,0,0,0,0,0,
0,0,0,0 3366,1,63-74,1,1,1,1,1,1,0,1,0,0,0,0 3367,
1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3368,1,63-74,1,0,0
,1,1,1,0,1,0,0,0,1 3369,1,63-74,1,1,1,1,1,0,0,0,0,
1,0,0 3370,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3371,1,
63-74,0,1,1,1,0,1,1,1,1,1,1,0 3372,1,63-74,1,1,1,1
,1,1,1,1,1,1,1,1 3373,1,63-74,0,0,1,1,1,1,0,1,1,0,
1,0 3374,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3375,1,63
-74,1,1,1,1,1,1,0,1,1,1,1,1 3376,1,63-74,1,1,1,1,1
,1,1,1,1,1,1,1 3377,1,63-74,1,1,1,1,1,1,1,0,1,1,1,
0 3378,1,63-74,0,0,1,1,0,1,1,1,0,0,0,0 3379,1,63-7
4,1,1,1,1,1,1,0,1,1,1,1,0 3380,1,63-74,1,1,1,1,1,1
,1,1,0,1,1,0 3381,1,63-74,1,0,1,0,1,1,0,1,1,0,1,0
3382,1,63-74,0,1,0,1,1,1,1,1,1,1,1,0 3383,1,63-74,
1,1,0,1,1,1,0,1,1,1,1,1 3384,1,63-74,1,1,1,1,1,1,1
,1,1,1,0,1 3385,1,63-74,1,1,1,1,1,1,1,1,0,1,0,1 33
86,1,63-74,1,0,0,1,1,0,1,0,0,0,0,0 3387,1,63-74,1,
0,1,1,0,1,0,1,0,0,0,1 3388,1,63-74,1,1,1,1,1,1,0,1
,1,1,1,0 3389,1,63-74,0,1,1,1,0,1,1,1,1,1,1,0 3390
,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3391,1,63-74,0,0,
0,0,0,0,0,0,0,0,0,0 3392,1,63-74,0,1,1,0,1,0,0,1,0
,1,0,1 3393,1,63-74,0,1,0,1,1,1,1,1,0,1,0,0 3394,1
,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3395,1,63-74,1,1,1,
1,1,1,1,1,1,1,1,0 3396,1,63-74,0,0,0,1,1,1,0,1,0,0
,1,1 3397,1,63-74,0,0,1,1,1,1,0,1,0,0,0,0 3398,1,6
3-74,0,1,1,1,0,1,1,1,1,1,1,0 3399,1,63-74,1,1,1,1,
1,1,1,1,1,1,1,1 3400,1,63-74,1,1,1,1,1,1,0,1,1,1,1
,1 3401,1,63-74,0,1,1,1,0,1,0,1,1,1,0,1 3402,1,63-
74,0,0,1,1,1,0,0,1,0,0,0,0 3403,1,63-74,0,0,1,1,0,
0,0,1,1,0,1,0 3404,1,63-74,0,1,1,1,0,0,0,0,0,0,1,0
3405,1,63-74,1,1,1,1,1,1,1,0,1,1,1,1 3406,1,63-74
,0,0,0,1,0,0,0,1,0,0,0,0 3407,1,63-74,1,1,1,1,1,1,
0,1,1,1,0,1 3408,1,63-74,0,0,0,0,0,1,0,0,0,0,1,0 3
409,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3410,1,63-74,1
,1,1,1,1,1,1,1,1,1,1,1 3411,1,63-74,0,0,1,0,1,0,1,
0,1,0,1,0 3412,1,63-74,0,1,0,1,1,1,0,1,1,1,1,0 341
3,1,63-74,1,0,0,1,1,1,0,1,0,1,1,0 3414,1,63-74,1,1
,1,0,1,1,1,1,0,1,1,0 3415,1,63-74,1,1,1,1,1,1,0,1,
1,1,1,1 3416,1,63-74,0,1,1,1,1,1,0,1,1,1,0,0 3417,
1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3418,1,63-74,1,1,1
,1,1,1,1,1,0,1,1,1 3419,1,63-74,1,0,0,1,0,1,0,1,0,
1,0,0 3420,1,63-74,0,1,1,1,0,1,1,1,0,0,1,1 3421,1,
63-74,0,1,1,1,1,1,0,1,0,0,1,1 3422,1,63-74,1,1,1,1
,0,1,0,1,1,1,0,0 3423,1,63-74,0,0,0,0,0,0,0,0,0,0,
0,0 3424,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3425,1,63
-74,0,0,0,0,0,0,0,0,0,0,0,0 3426,1,63-74,1,1,1,1,1
,1,0,1,0,1,1,1 3427,1,63-74,0,0,1,1,0,1,0,0,1,1,1,
0 3428,1,63-74,1,1,1,1,1,1,0,0,0,1,0,0 3429,1,63-7
4,0,0,0,0,0,0,0,0,1,0,0,0 3430,1,63-74,1,1,1,1,1,1
,0,1,0,1,1,1 3431,1,63-74,0,0,1,1,0,0,0,0,1,1,0,1
3432,1,63-74,0,0,0,0,1,0,0,0,0,0,1,0 3433,1,63-74,
0,1,0,1,0,1,1,1,0,0,1,1 3434,1,63-74,1,1,1,1,1,1,1
,1,1,1,1,1 3435,1,63-74,0,1,0,1,1,1,0,1,0,1,1,0 34
36,1,63-74,0,0,0,1,0,0,0,0,0,0,1,0
15
Study regulations, observe practice

• Has the exam gone through accreditation procedure
  by the Ministry?
• Many exams have, some have not (list www.nyak.hu)
• The Hungarian accreditation system not perfect,
  but at least
• a body of experts to review exams regularly and
  on demand
• documentation must be submitted
• Some achievements

16
Study regulations, observe practice

• How soon are results announced?
• If announced very soon, it is unlikely to be a
  good test, as it is impossible to do the quality
  control work that needs to be done.

17
Study regulations, observe practice

• Are most complaints handled favourably for the
  candidates?
• If quality control procedures are weak, such
  practice sends a wrong message
• Kaparj kurta neked is jut, making the candidate
  responsible for finding problematic items,
  raters, etc.
• Implicit in law
• Unfair. Quality control should not depend on
  candidates awareness of their rights.

18
Study regulations, observe practice

• Are pass rates very high?
• Bear national educational context in mind
• falling numbers in the age-group where candidates
  might come from (14-22)
• falling foreign language education standards
• should predict falling pass rates too if
  standards (CEFR levels) are maintained .
• The horrible case of a prestigious German exam 10
  years ago.

19
Summary

• What is good testing practice?
• Use this checklist of questions
• Read up on language testing and add more
  questions
• What you need
• a keen eye
• a sane mind.

20
Required reading

• McNamara, T. (2000). Language Testing. Oxford
  Introductions to Language Study. Series Editor
  H.G Widdowson.Oxford OUP.
• The required chapters are
• The Testing Cycle (Chapter 3) pp. 23-33
  Validity testing the test (Chapter 5) pp. 47-54
  
• Measurement (Chapter 6) pp.55-65.