A fresh look at assessment Learning

1
A fresh look at assessmentLearning Teaching
Week 3-7 April 2006

• Lin Norton
• Professor of Pedagogical Research
• Associate Dean, Learning and Teaching

2
Some views in the assessment literature

• Assessment is learning. The idea of learning
  without some form of assessment is inconceivable
  Brown Knight (1994)
• Students take their cues from what is assessed
  rather than from what the lecturers assert as
  important Brown, Bull Pendlebury(1997)
• Learning is so driven by assessment that the
  form and nature of assessment often swamps the
  effects of any other aspects of the curriculum
  (Boud ,1990)

3
Have we changed our assessment practices since
Bouds (1990) review ?

• Boud found that in many HEIs in the late 80s
  there was a heavy emphasis on assessment which
  encouraged dependent learning and a surface
  approach. Is this still true today?
• Students are assessed on those matters on which
  it is relatively easy to assess them
• Assessment encourages students to focus on those
  topics which are assessed at the expense of those
  which are not
• The nature of assessment tasks influences the
  approaches to learning which students adopt.
• Students who perform well in university exams can
  retain fundamental misconceptions about key
  concepts in the subjects they have passed
• Students give precedence to assessment which is
  graded
• Successful students seek cues from tutors to
  enable them to identify what is important for
  formal assessment purposes

4
Some principles of assessment lets challenge
them!

• Rust (2002) describes the principles
• Reliability (down with the connoisseur
  approach, up with QAA, standards, fairness and
  objectivity. Make explicit learning outcomes and
  assessment criteria.)
• Validity (can you assess my competence in riding
  a bike by asking me to write an essay on the
  skills needed, or by giving me an exam on the
  parts of a bike?)
• Relevance transferability (emphasis in
  traditional assessment on declarative vs.
  functioning knowledge (Biggs, 2003) debate about
  transferability).
• Criterion vs. norm referenced assessment (in HE
  we tend to use the latter even though we claim we
  use the former)

5
Making assessment criteria transparent What are
the advantages for students?

• Pedagogically sound research evidence that
  students dont understand assessment criteria
  (i.e. Higgins et al 2002)
• Makes demands of task clear
• Enables tutors to give meaningful feedback
  students opportunity to improve
• Meets QAA principles of equity, fairness and
  accountability

6
And what are the disadvantages?

• They encourage a strategic marks-orientated
  approach rather than a meaningful engagement with
  the subject matter (Norton, 2004)
• Tutors often cannot agree on meanings and values
  attached to criteria
• Tutors have mental models of marking which are
  resistant to new guidelines in applying
  assessment criteria (Wolf, 1995)
• Transparency itself is a contested notion (Orr,
  2005)

7
Using assessment criteria in marking What are
the advantages for staff?

• Will help to reduce
• Lack of inter-rater reliability on marks
  (Newstead Dennis, 1994)
• Order and practice effects, fatigue personal
  bias
• Sounder overall judgment than intuitive mental
  model (Elander, 2002)
• Helpful for novice assessors

8
Research on assessment criteriaAssessment Plus
www.assessmentplus.net

• The interview study (Norton et al, 2005)
• 30 min semi-structured interview with psychology
  lecturers from 3 institutions
• Institution A N 15 first
• Institution B N 7 wave
• Institution C N 9 second wave
• Lecturers included full time and part time and
  length of teaching experience in HE ranged from 2
  months to over 21 years

9
The concept of core assessment criteria
Core criteria are those that appear very
commonly in assessment criteria across
disciplines and institutions, and that appear to
have a central role in the shared perception of
what is important in a good student essay
Elander et al (2004)
10
What are core assessment criteria?

• Addressing the question
• Structuring the answer
• Demonstrating understanding
• Developing argument
• Using evidence
• Evaluating sources
• (Use of language)

11
Reactions to the core criteria

• Recognised by all
• I think theyre the main criteria evaluation
  and critical evaluation are areas that give you
  the opportunity to move beyond a pass to a higher
  grade, (6/B)
• 2. 2. Originality not featured
• When I give the examiner an outline of what
  Im expecting from the essay I always put be
  prepared to be surprised. Originality I
  suppose. (2/A)
•  

12
Do we have shared understandings of core
criteria?

• What does critical evaluation mean to you?
• What would you expect students to do in your
  subject to demonstrate they had met this
  criterion?

13
Psychology interviewees responses to this
question on critical evaluation
at a primitive level showing some emotion
for what youre doing, may even mean taking
sides when you have very little evidence to
support it, display of personal and emotional
involvement in whats being studied, attempt to
give other sides of argument (7) Looking at
the research and not assuming it is the last say,
that it is right checking if research carried
out appropriately (2) Tough to define
Perhaps a misnomer evaluation includes
potential for criticism, which entails thinking
about theory in relation to both evidence and
other theories (e.g., does theory stand up in
light of empirical evidence, does another theory
do a better job of explaining the evidence?)
(11)
14
Do staff attach same weightings to the criteria
when marking? (1)

• using appropriate language, I dont think is
  so important, Im quite happy if they can
  criticize and evaluate. That said, its unusual
  to find someone who has the ability to critically
  evaluate but cant write properly.
• Theyre all important, but demonstrating
  understanding and evaluating are things that I
  personally pay more attention to if Im trying to
  decide if an essay is merely acceptable or good.
• I think it depends on who the students are.
  There are some whose writing skills need a lot of
  attention and work so wed concentrate on
  language and structure but for those who have got
  the writing sussed we move on to understanding
  and evaluating. Its not easy for some students
  to express themselves coherently.

15
Do staff attach same weightings to the criteria
when marking? (2)

• Addressing the question obviously. Use of
  evidence is really important in Psychology and
  that demonstrates their understanding. Im less
  worried about structure in exam work because
  theyre under pressure. Developing an argument
  and critically evaluating are done very poorly by
  students.
• 5. No, some are more important than others.
  It depends partly on the title and structure is
  always important.
• I think theyre all as important.
• I think understanding is important and
  developing the argument are more important for me
  personally because those things are key to the
  student being able to go off and be an
  independent learner. But thats difficult to
  assess because a well-structured essay may not
  always have good understanding but its more
  compelling some how.

16
Staff views on using assessment criteria in
marking guidelines

• Limited usefulness (protects in appeals)
• Concerns about artificially high marks
• Staff have their own ideas about marking which
  tend
• to take precedence over any given guidelines
•  

17
Staff views of students understanding of
assessment criteria
I think in some respects they must understand
the criteria because their essays improve. they
must perhaps have picked up what we mean.
(3/A) It takes a while, theres a noticeable
progression through the course and clearly some
criteria are less well understood because youre
constantly having to make the same comments. And
still they make the same omissions or errors,
even when they get feedback. Perhaps they dont
fully understand the importance of the criteria.
(6/B)  
18
Conclusions from staff interviews (1)

• Lecturers appear to use mental models of
  assessment in which assessment criteria feature
• Mental models appear resistant to change
• Assessment criteria seen as useful in terms of
• Training tool
• Resolving differences in second marking
• Moderation
• Protection against appeals of unfair marking

19
Conclusions from staff interviews (2)

• Wide variation in views on how best to use
  assessment criteria
• Preference for them to be used as a tool to aid
  consistency rather than a rigid framework to be
  adhered to

20
Conclusions from staff interviews (3)

• Lecturers thought students did not generally
  understand assessment criteria
• 77 of lecturers (wave 1) thought students did
  not bother to read the assessment criteria
• Consensus that students improved as they
  progressed through their degrees
• Wide variability in how lecturers use assessment
  criteria in marking

21
Student interviews focus groups

• Institution A
• 5 focus groups (2nd, 3rd, 4th years and
  postgraduates)
• 3 individual interviews (1st, 2nd year and
  postgraduate)
• Institution C
• 3 focus groups (1st, 2nd and 3rd years)

22
Students understandings of the importance of
assessment criteria
I used to not really look at the assessment
criteria and wonder what the tutor was looking
for but as I have progressed through the years I
have realised that if you dont follow what they
ask then you tend not to do very well, thats
probably why I failed my first year. (James, 2nd
yr) In the first year I didnt really utilise
the assessment criteria and therefore I didnt
know what the tutors were looking for, it seems
silly now not to look at the assessment criteria
but I guess in the first year I just took it for
granted that I would be told everything without
having to actually do any independent thinking
(David, 3rd yr)
23
Students understandings of critical
evaluation
A clear outline of what youve been looking at
from a variety of angles. (Janet) Well, looking
at, taking the two opposite theories, discussing
them in a sort of I suppose a basic sort of way,
you know seeing what the evidence is for one
theory and the counter-evidence is for the
opposing theory. (Graham) Look at the strengths
and weaknesses of what you read. Can you think
of any alternative explanations for the claims
that the theory/research makes? If so, bring
them into your essay, and if possible support
them with evidence. (Rich) I actually think it
is more than strengths and weaknesses. Its
actually criticizing their strengths and
weaknesses. (Lana)
24
Students views about how staff use assessment
criteria

• ..umm I dont think that it is standardised
  throughout the college at all. Sport and
  Psychology and how they introduce the assessment
  criteria to you is completely different. I dont
  think we have ever been really told to be honest,
  (Brett, 3rd year, Institution A)
• I find that the tutors and the way that we are
  taught and explained to us about assessment its
  sketchy and all over the place for the
  undergraduates, this is what I am finding at the
  moment (Lana, M.Sc undergrad conversion
  diploma simultaneously, Institution A)
• Its hard to know exactly what they want, some
  tutors write what they want in the lectures
  others dont, so its a bit of a guessing game
  really. (Karen, 2nd year)

25
Are students conscientious consumers?

• In general, the interview and focus groups show
  students are very concerned to improve their
  essay writing, but often dont take active steps
  particularly in the first year
• In a study evaluating the effects of optional
  workshops (Norton et al, 2004), the main findings
  were
• Students tend not to attend optional workshops
  but those that do find them supportive and
  helpful but wanted MORE guidance, MORE feedback
  and MORE hands-on experience
• Workshop attendance tends to improve essay
  performance

26
Conclusions from student interviews

• Some students were very shaky in their
  understanding of assessment criteria
• Conscientious attention to lecturer feedback
• Difficulties in interpreting lecturers comments
• Variability between lecturers when marking
• Positively disposed to workshops but attendance
  quite low

27
Assessment criteria some personal conclusions

• A two-edged sword because
• There appear to be little agreed understandings
  between staff about core assessment criteria or
  their values.
• Defining criteria in terms of relating to marks
  results in vague statements like excellent,
  good, adequate, poor
• BUT
• Many students are conscientious consumers
  (Higgins et al, 2002) and make active efforts to
  understand criteria and improve their essay
  writing
• An over-emphasis on assessment criteria may have
  the paradoxical effect of encouraging a
  mechanistic approach Tell me what do I have to
  do to get an A for this essay?

28
Following on

• Resource available by email from Lin
• Writing essays in higher education A guide for
  students by students Norton, Pitt, Harrington,
  Elander Reddy (assessment Plus publication)
• Practical session on assessment and feedback
• Led by Michelle ODoherty, Writing Specialist,
  Write Now Centre for Excellence in Teaching and
  Learning
• This afternoon 2-4pm in WAR 011A

29
References

• Biggs, J. (2003) Teaching for Quality Learning at
  University. 2nd ed. Buckingham Society for
  Research into Higher Education and The Open
  University Press.
• Boud, D (1990) 'Assessment and the promotion of
  academic values', Studies in Higher Education,
  15, 1, 101-111.
• Brown, S. Knight, P. (1994) Assessing learners
  in higher education. London Kogan Page
• Brown, G., Bull, J. Pendlebury, M. (1997)
  Assessing student learning in higher education
  London Routledge
• Elander, J. (2002) Devleoping aspect specific
  assessment criteria for examination answers and
  coursework essays in psychology. Psychology
  Teaching review, 10, 1, 31-51.
• Higgins,R., Hartley, P. Skelton, A. (2002) The
  conscientious consumer reconsidering the role of
  assessment feedback in student learning. Studies
  in Higher Education, 27, 1, 53-64
• Newstead, S.E. Dennis, I. (1994) Examiners
  examined the reliability of exam marking in
  psychology. The Psychologist, 7, 216-219
• Norton, L.S. (2004) Using assessment criteria as
  learning criteria. A case study using Psychology
  Applied Learning Scenarios (PALS). Assessment
  Evaluation in Higher Education. Vol 29(6)
  687-702.
• Norton, L., Harrington, K., Elander, J.,
  Sinfield, S., Lusher, J., Reddy, P., Aiyegbayo,
  O Pitt, E. (2005) Supporting students to
  improve their essay writing through assessment
  criteria focused workshops . In C. Rust (ed.),
  Improving Student Learning 12. Diversity and
  Inclusivity Oxford, Oxford Centre for Staff and
  Learning Development.
• Orr, S.(2005) Transparent opacity assessment in
  the inclusive academy In C. Rust (ed.), Improving
  Student Learning 12. Diversity and Inclusivity
  Oxford, Oxford Centre for Staff and Learning
  Development.
• Rust, C (2002) Purposes and principles of
  assessment. OCSLD Learning and teaching Briefing
  papers series www.brookes.ac.uk/services/ocsd
• Wolf, A (1995) Competence based assessment.
  BuckinghamOpen University Press