Challenging traditional forms of assessment: University teachers

1
Challenging traditional forms of assessment
University teachers views on examinations

• Lin Norton, Katherine Harrington,
• Bill Norton Lee Shannon
• Liverpool Hope University, London Metropolitan
  University

2
Background

• Assessment widely acknowledged as having a
  profound influence on what students learn and how
  they learn
• Move away from traditional learning theory to an
  increasing recognition that learning is a
  process of knowledge construction rather than
  knowledge reproduction (Maclellan, 2001)
• Concomitant move towards alternative methods of
  teaching and learning (e.g PBL, work based
  learning, experiential learning)
• But not necessarily so great a shift in
  assessment practices - in a study carried out at
  LHU in 5 disciplines, the most frequently used
  assessment tasks were exams (Steward et al, 2003)

3
Traditional exams a definition

• An unseen closed book timed assessment, the form
  of which might be
• An essay
• A short answer
• A multiple choice test

4
What do exams do? The pros

• Represents the end product of the education
  process
• Validity of a degree the external recognition of
  a qualification that is wanted by universities ,
  employers and individuals (ODonovan, 2005)
• An antidote to plagiarism
• Revision period acts as an opportunity for
  synthesis and a coherent understanding of the
  subject ( Entwistle Marton, 1994)
• Concentrates students minds and motivates them
• Aids standardisation
• Gives tutors an idea of how much students have
  understood
• Tests different skills and abilities to those
  demanded in coursework
• Maybe more appropriate at certain stages of
  development and for varying levels of self
  direction in learning (Macdonald 2002)

5
What do exams do? The cons

• Unpopular with students who have often had
  negative experiences
• Causes stress and anxiety
• Students feel they are an unfair assessment of
  their ability (Norton Brunas-Wagstaff, 2000
  Sambell et al 1997)
• Tends to test learning that is linear and
  decontextualised
• Does not assess meaningful and authentic learning
  (Maclellan, 2001)
• Encourages strategic selection of parts of the
  curriculum that students think will come up on
  the exam
• Encourages rote learning and memorisation
• What is learned for an exam is often quickly
  forgotten afterwards
• Represents a snapshot in time rather than a
  richer picture of student ability
• Marking of exam scripts is unreliable (Newstead
  Dennis, 1994)

6
Rationale for our research

• Transformational agendas in assessment, such as
  switching from traditional exams, require
  individual lecturers, subject disciplines and
  institutions to be adaptive.
• Although there is much in the literature on
  students perceptions of exams as a form of
  assessment, there appears to be relatively little
  on university teachers beliefs about the
  pedagogical value of exams.

7
The research study

• A semi-structured interview study carried out at
  two universities in the UK to ascertain
  university teachers views on examinations
• Since exams are a contested form of assessment
  we used the concept of core assessment criteria
  (Elander et al, 2004) to more obliquely approach
  our main research questions
• what forms of exam were used
• what pedagogical value exams were held to have
• if there were any problems affecting assessment
  procedures

8
The interview schedule

• Question schedule adapted from study on essay
  assessment criteria (Norton et al., 2004)
  including such questions as
• What sorts of examinations are you typically
  involved in setting?
• Of the various methods of examination, which do
  you feel produces the best responses from your
  students?
• What do you think a student gains from this form
  of assessment?
• Do you think there are any problems with current
  assessment procedures?

9
Focus of the presentation

• This presentation focuses on a subset of 24 out
  of 29 interviews
• Examination experience ranged from 6 to 37 years
  for University A from 3 to 25 years for
  University B

University Business Computing English Psychology Total
A 2 3 2 4 11
B 4 3 4 2 13
Total 6 6 6 6 24
10
Analysis

1. Transcripts were subject to repeated readings
   from which several themes emerged
2. Discussion of the emergent themes between the
   researchers produced a framework in which to
   present the findings
3. Variation in how exams were used
4. Pedagogical justification
5. Constraints

11
Forms of exam in use
Unseen essay Seen essay Problem scenario/ case study Short answers MCQ In class tests
Business 1 1 4 3   1
Computing 1 1 4 3 1 1
English 6 4   3 2  
Psychology 6   1 2 1  
Total 15 6 9 11 4 2
12
Pedagogical justification

• Most lecturers were alive to the need to align
  assessment and used or wished to use a range of
  appropriate methods
• There was less unanimity about the pedagogical
  value of exams
• Ambivalence about worth of exams Im not in
  favour of the pressure that (exams) create, but
  it shows you who can focus their mind on the
  problem. (B/Business lecturer)
• Some staff were making the best of it- I dont
  really like exams but I dont have a choice if I
  want to control cheating (A/Computing lecturer)

13
Pedagogical value positive

• allowed distinction between grades readily
  identified better students
• redressed the gender bias
• allowed students to show what they had really
  learnt
• encouraged students to cover the syllabus
• required working under pressure, thinking on your
  feet as you have to in real life
• short answers were appropriate in some
  circumstances e.g. close textual analysis in
  English (B), reviewing programs in Computing (A)
• in class tests could provide good formative
  feedback
• MCQ if properly designed could test across the
  syllabus

14
Pedagogical value negative

• lack of opportunity for formative feedback from
  end of year/module exams
• over reliance on memory regurgitation
• may be examining exam technique rather than what
  had been learned
• unreal, inauthentic, time limited situation
• should not be the only method of assessment
• adverse effect on students, anxiety provoking,
  gender bias

15
Constraints what held them back from alternative
assessment methods

• Tension in the sector about the role of Higher
  Education, attaining qualifications rather than
  transformational learning
• Change in the nature of students in HE, widening
  participation, secondary experience poor
  preparation
• what is educationally appropriate versus what is
  financially viable
• institutional needs well set up for exams
  more effort spent on exams than on coursework,
  quicker and more reliable to mark on the other
  hand coursework seen as easier option and can
  give better pass rates
• lack of time procedures to develop more varied
  appropriate assessment methods. Institutional
  restrictions on feedback.
• concern about plagiarism a powerful incentive for
  exams
• lecturers own experience as a student, attitudes
  beliefs they worked for me

16
Conclusions

• Most of the teachers in both institutions would
  abandon exams if
• Better ways of preventing plagiarism in
  coursework
• More time/support available to develop an
  appropriate range of assessment methods (the
  question of slowness of institutional change cf
  Bransford adaptive organisations)
• If exams are used then
• more flexible formats and user friendly
  introduction for students
• timing procedures should permit opportunity to
  give formative feedback.
• students should get feedback on exam scripts

17
Implications

• Most university teachers in this study did have
  an underlying pedagogical philosophy about
  assessment which was not always put into practice
  because of the non adaptive discipline and/or
  institution.
• This confirms findings from a study of new
  lecturers views about assessment (Norton et al
  2010)
• This pedagogical research has produced an
  evidence base that we hope will enable us to
  encourage change at the discipline/institutional
  level

18
The last word
When I taught in America it was different, I
would take my prejudices for assessment that I
would favour and design the courses to suit that.
Here they are quite rigid, I think things have
opened up a great deal in the last five to ten
years I would say but I think there's still room
to grow the types of assessment that we do
include. I think there are certain prejudices,
with the new universities at the forefront of
redesigns of assessment, there is some prejudice
in the more established universities I've taught
at, you know that things have to be proven, maybe
some of the new forms of assessment that we've
attempted do need a little bedding down and some
monitoring and evaluation themselves. (B/English
lecturer)
19
References

• Elander, J., Harrington, K., Norton, L.,
  Robinson, H., Reddy, P. Stevens, D. (2004).
  Core assessment criteria for student writing and
  their implications for supporting student
  learning. In C. Rust (ed.), Improving Student
  Learning 11. Theory, Research and Scholarship
  (pp. 200-212). Oxford, Oxford Centre for Staff
  and Learning Development.
• Entwistle, N Marton, F. (1994) Knowledge
  objects understandings constituted through
  intensive academic study, British Journal of
  Educational psychology, 64, 161-178
• Macdonald, M. (2002) Systematic assessment of
  learning outcomes developing multiple choice
  exams. Jones Bartlett Learning
• Maclellan, E. (2001) Assessment for learning the
  differing perceptions of tutors and students.
  Assessment and Evaluation in Higher Education,
  26,4, 307-318.
• Newstead, S.E. Dennis, I. (1994). Examiners
  examined The reliability of exam marking in
  psychology. The Psychologist, 7,216219.
• Norton , L. and Brunas-Wagstaff, J. (2000)
  Students perceptions of the fairness of
  assessment. Paper presented at the first annual
  conference of the Institute for Learning and
  Teaching in higher education, iltac 2000, York,
  27-29 June 2000.
• Norton, L.S., Ward-Robinson, H., Reddy, P.,
  Elander, J. and Harrington, K. (2004) Exploring
  psychology lecturers view on assessment
  criteria. Psychology Learning and Teaching
  Conference (PLAT 2004), University of
  Strathclyde, 5-7 April 2004
• O'Donovan, N. (2005) There Are no Wrong Answers
  An Investigation into the Assessment of
  Candidates' Responses to Essay-Based
  Examinations, Oxford Review of Education, 31, 3 ,
  395-422
• Sambell, K., Brown, S McDowell, L. (1997) But
  Is It Fair?" An Exploratory Study of Student
  Perceptions of the Consequential Validity of
  Assessment Studies in Educational Evaluation,
  23,4 349-71
• Steward, S., Norton, L.S., Evans, I. Norton,
  J.C.W. (2003) Lecturers! What are you assessing?
  Paper given at the Learning Skills Research
  Network Annual Conference Research for all GMB
  National College, Manchester, 6 June, 2003