Examination Impact: Monitoring how teachers cope with change

1
Examination Impact Monitoring how teachers
cope with change

• Dianne Wall
• Tania Horák
• Lancaster University
• UK

2
Background

• Next Generation TOEFL (NGT) to be launched
  September 2005 (in certain sites only)
• Two impact studies commissioned by Educational
  Testing Service (producer of TOEFL)

3
Our impact study

• Where?
• Central and Eastern Europe
• Who?
• TOEFL preparation course providers

4
Phase 1 Phase 2 Phase 3
Baseline Transition Impact
Jan 03 to Oct 04 Oct 04 to Mar 06 After introduction of NGT
-Description of current TOEFL preparation classes -Class observations, interviews with teachers, DoS and students. -10 sites in 6 countries -Tracking study   -Interviews via Internet -Description of NGT preparation classes -Class observations, interviews with teachers, DoS and students. - same sites
5
Theoretical underpinnings

• 1) Henrichsen (1989) Hybrid Model of the
  Diffusion / Implementation Process
• 2) Messick (1996) - evidential link

6
Henrichsens model
Antecedents
Process
Consequences
7
Process

• The Receivers
• The Innovation
• Characteristics of Communication
• The Resource System
• The User System
• Inter-elemental factors

8
How our study fits the model
Antecedents Process Consequences
Baseline Transition Impact
9
Messicks evidential link
Antecedents Process Consequences
evidential link
10
Phase 2

• Who? 6 teacher participants from Phase 1
• Where? 5 countries Bulgaria, Croatia,
  Lithuania, Poland, Slovakia.
• When? data collection period January - May
  05
• How? Computer Mediated Communication
• What? investigation of pps awareness and
  understanding of and reactions to NGT, to help
  us understand how they are preparing for the
  arrival of NGT

11
What?

• Interviews based on monthly tracking questions
• Completion of tasks designed to familiarise
  participants with NGT
• Interviews discussing the task responses

12
Data collection activities

Interview 1
Task sent to pps
Interview 2
Pps return task
Researchers discuss responses
13
The Tasks

• January - Features of TOEFL preparation classes
• February - Awareness of current and future TOEFL
• March - Focus on the new writing tasks
• April - Focus on the new speaking tasks
• May - Prediction of actual content and
  methodology of pps NGT preparation classes

14
Findings

• Understanding of NGT
• Materials
• Teacher Support

15
Understanding of NGT

• Teachers seem clear about the format of NGT but
  less clear about the actual constructs.

16
Understanding of NGT

• I don't think the matter of accent was really
  addressed, though maybe those who grade the tests
  have an implicit understanding of how accent
  plays a role and will score accordingly. In
  preparing students for tests, it would help to
  have the issue of accents and how they affect
  scoring addressed directly in the rubric.

17
Understanding of NGT

• Conclusion
• Awareness-raising requires more than providing
  examples of the test.

18
Materials

• Teachers rely heavily on published materials.

19
Materials

• I cant even dare think of something like that
  but I suppose Ill have to improvise. The other
  possibility is to reschedule the NGT classes to
  the moment when there will be materials.

20
Materials

• Conclusions
• Impact will probably be affected by the textbook
  writers
• Impact of the test may well be hard to
  distinguish from impact of the textbook

21
Teacher Support

• The test provider has supplied various forms of
  teacher support but they are not necessarily
  accessible to the teachers who need them.

22
Teacher Support

• I just heard it mention by the Director
  She also said it is very expensive . I cannot
  afford, the center cannot afford.

23
Teacher Support

• Conclusion
• Careful consideration must be paid to the
  logistics of teacher support if it is to be
  effective and reach those who need it, when they
  need it.

24
Professionalism

• Test Provider - by commissioning impact studies
• TOEFL teachers by participating in our study

25
For further information

• About the project
• d.wall_at_lancaster.ac.uk
• t.horak_at_lancaster.ac.uk
• About NGT
• http//toeflpractice.ets.org

26
(No Transcript)
27
Tracking questions

• Have you learnt anything about New TOEFL which
  you didnt know last month? If so, what?
•  Is the new exam being discussed in your
  institution (e.g. by other teachers/ management)?
  If so, what are the main issues being
  discussed? 
• Have students asked about the new TOEFL? If so,
  what have they asked about?
• Are you worried/ concerned about anything
  concerning the new TOEFL? If so, what?
• Is there anything else of interest concerning new
  TOEFL which has happened this month?