Assessment in Higher Education: where next

1
Assessment in Higher Education where next?

• GEES conference
• Sally Brown
• PVC (Academic)
• Leeds Metropolitan University

2
Approaches to innovative HE assessment

• How can we ensure students make the
• most of assessment as part of their
• overall learning experience?
• How can staff incorporate innovative assessment
  approaches that are
• manageable and fit-for-purpose?
• How can HEIs ensure that assessment strategies
  support students and staff
• while assuring quality?

3
Some uncomfortable questions

• Are we certain that the way we assess students is
  working?
• Is Phil Race right when he says assessment is
  broken?
• Where is the quality in quality assurance?
• Do students understand what we are trying to do
  with assessment?
• Can assement actually help students learn?

4
Current trends in HE design, delivery and
assessment

• A move towards blended learning as opposed to
  e-learning or traditional face-to-face approaches
  (assessment implications?)
• Recognition of the centrality of assessment for
  (rather than just of) learning (stop marking,
  start assessing!)
• The importance of teams rather than individuals
  in assignment design
• Recognition of divergent international approaches
  to assessment

5
My predictions for the next decade

• The move away from universities being the
  guardians of content, where everything is about
  delivery, towards universities having two major
  functions
• Recognising and accrediting achievement, where
  ever such learning has taken place (not
  necessarily in our university but from anywhere)
• Supporting student learning and engagement.

6
Some contextual issues

• Impact of the recession (staffing, etc)
• Central importance of student retention (balanced
  with maintaining quality)
• Impacts of the National Student Survey and
  various other means of expressing student (dis)
  satisfaction
• NUS interest the National Feedback amnesty
• External scrutiny of quality of assessment,
  particularly the quality and speed of turnaround
  of feedback.

7
Assessment methods and requirements probably
have a greater influence on how and what students
learn than any other single factor. This
influence may well be of greater importance than
the impact of teaching materials (Boud 1988)
8
Students can avoid bad teaching they cant
avoid bad assessment. (Boud 1994)
9
What are students getting like?

• More value(s)-conscious
• More litigious?
• More diverse
• Blurred distinction between part-time and
  full-time students
• Demonstrating the impact of different approaches
  to study in schools (new 14-19 curriculum)
• Having increased expectations of diverse kinds of
  support.

10
Implications of widening participation

• Ever more diverse student population
• Retention of diverse students is paramount
• Research (Yorke etc) tells us assessment is
  central to retention
• Feedback and feed forward are at the heart of
  retention
• Detailed and timely feedback is hugely demanding
  of staff.

11
Some sample innovatory approaches to assessment

• Assessment of learning in practice settings e.g.
  use of PDAs on site in clinical settings (ALPS)
• Use of blogs to assess reflective practice
• Groups projects to replace final year
  dissertations
• Exploratory work on computer-based assessment of
  short answer questions
• Assessment of multiple small tasks to demonstrate
  achievement of practical competence (OSCEs, PASS)
• Audio Feedback (Sounds good) www.soundsgood.org.
  uk

12
My fit-for-purpose model of assessment the key
questions

• Why are we assessing?
• What is it we are actually assessing?
• How are we assessing?
• Who is best placed to assess?
• When should we assess?

13
To integrate assessment we need to realign it
with the curriculum by

• Exploring ways in which assessment can be made
  integral to learning
• Constructively aligning (Biggs 2003) assignments
  with planned learning outcomes and the curriculum
  taught
• Providing realistic tasks students are likely to
  put more energy into assignments they see as
  authentic and worth bothering with
• Us more and better formative assessment.

14
The guru Sadler

• The indispensable conditions for improvement are
  that the student comes to hold a concept of
  quality roughly similar to that held by the
  teacher, is able to monitor continuously the
  quality of what is being produced during the act
  of production itself, and has a repertoire of
  alternative moves or strategies from which to
  draw at any given point. In other words, students
  have to be able to judge the quality of what they
  are producing and be able to regulate what they
  are doing during the doing of it. (Sadler
  1989).(my italics)

15
Formative and summative feedback two ends of a
continuum

• Formative assessment is primarily concerned with
  feedback aimed at prompting improvement, is often
  continuous and usually involves words.
• Summative assessment is concerned with making
  evaluative judgments, is often end point and
  involves numbers.

16
Whats the difference between formative
summative assessment?

• Summative contrasts with formative assessment in
  that the former is concerned with summing up or
  summarizing the achievement status of a student,
  and is geared towards reporting at the end of a
  course of study especially for purposes of
  certification. It is essentially passive and does
  not normally have immediate impact on learning,
  although it often influences decisions which may
  have profound educational and personal
  consequences for the student.. (Sadler 1989).

17
Using formative and summative assessment
appropriately

• Students often dont value formative assessment
• All assessment needs to be fair, consistent,
  reliable, valid, manageable and transparent
• Many assessment systems fail to clarify for
  students the purposes of different kinds of
  assessment activity
• Low-stakes formative assessment helps students,
  especially those from disadvantaged backgrounds,
  understand the rules of the game.

18
Some problems with formative assessment

• Students may not take it as seriously as
  summative assessment, if it doesnt count
• It can be hugely time consuming
• Students are likely to need different kinds of
  formative assessment at different stages in their
  learning journeys
• It can be difficult to gauge how best to do it
  with groups of students who may be at different
  stages of development.

19
How can we make students take feedback more
seriously? We can

• Spend time and energy helping students to
  understand the importance of feedback and the
  value of spending some time after receiving work
  back to learn from the experience. Most students
  dont do this at the moment, concentrating
  principally on the mark.
• Withhold the mark until after the student has
  received and responded to feedback
• Provide assessed opportunities for reflection on
  previously marked work.

20
We can make feedback timely

• Aim to get formative feedback on work back to
  students very quickly, while they still care and
  while there is till time for them to do something
  with it.
• The longer students have to wait to get work
  back, especially if they have moved into another
  semester by the time they receive their returned
  scripts, the less likely it is that they will do
  something constructive with lecturers
  hard-written comments.
• Dont bother with detailed feedback for
  non-continuing students

21
We can concentrate formative feedback where it
can do most good

• Dont give detailed written feedback to students
  on work that is handed back at the end of the
  semester if that area of study is no longer being
  followed by the student just give a mark or
  grade
• Give more incremental feedback throughout the
  semester (and if university systems dont allow
  this, change the systems!)
• Ensure that students at the top end of the
  ability range dont feel short changed by minimal
  feedback.

22
Use formative assessment to promote independence

• Investigate how learning can be advanced in small
  steps using a scaffolding approach
• Provide lots of support in the early stages when
  students dont understand the rules of the game
  and may lack confidence
• This can then be progressively removed as
  students become more confident in their own
  abilities.

23
Consider providing opportunities for multiple
assessment

• Consider allowing resubmissions of work as part
  of a planned programme
• Students often feel they could do more will work
  once they have seen the formative feedback and
  would like the chance to have another go
• Particularly at the early stages of a programme,
  consider offering them the chance to use
  formative feedback productively
• Feedback often involves a change of orientation,
  not just the remediation of errors.

24
Use formative assessment to help students with
reading

• Help them also to understand that there are
  different kinds of approaches needed for reading
  depending on whether they are reading for
  pleasure, for information, for understanding or
  reading around a topic
• Guide them to become active readers with a pen
  and Post-its in hand, rather than passive
  readers, fitting the task in alongside television
  and other noisy distractions
• Give them clear guidance in the early stages
  about how much they need to read and what kinds
  of materials they need to focus on.

25
From Bowl (2003) Non-traditional students in HE

• The hardship was not understanding. When they
  give you an assignment and say it was on this
  handout. But my difficulty is not understanding
  what to do at first I think that theres a lack
  of my reading ability, which I cant blame anyone
  for. I can only blame myself because I dont like
  reading. And if you dont read, youre not going
  to learn certain things. So I suppose thats to
  do with me..its reading as well as putting what
  you read into your essay. You can read it and
  understand it. I can read and understand it, but
  then you have to incorporate it into your own
  words. But in the words they want you to say it
  in, not just She said this, and this is the way
  it should be. The words, the proper language.
• (Bowl 2003 p90).

26
Use formative assessment to help students with
writing

• Devote energy to helping students understand what
  is required of them in terms of writing
• Work with them to understand the various academic
  discourses that are employed within the
  subject/institution
• Help them to understand when writing needs to be
  personal and based on individual experience, such
  as in a reflective log, and when it needs to be
  formal and using academic conventions like
  passive voice and third person, as in written
  reports and essays.

27
Involve students in their own and each others
assessment

• Consider ways of getting students to give each
  other meaningful formative feedback
• Shared reflection is the best means available to
  help them really get inside the criteria
• Asking students to review each others draft
  material prior to submission particularly helps
  those who lack confidence about what kinds of
  things are expected of them.

28
Providing opportunities for multiple assessment?

• Consider allowing resubmissions of work as part
  of a planned programme
• Students often feel they could do more will work
  once they have seen the formative feedback and
  would like the chance to have another go
• Particularly at the early stages of a programme,
  consider offering them the chance to use
  formative feedback productively
• Feedback often involves a change of orientation,
  not just the remediation of errors.

29
To integrate assessment we need to realign it
with the curriculum by

• Exploring ways in which assessment can be made
  integral to learning.
• Constructively aligning (Biggs 2003) assignments
  with planned learning outcomes and the curriculum
  taught
• Providing realistic tasks students are likely to
  put more energy into and play fairer with
  assignments they see as authentic and worth
  bothering with.

30
Making assessment work well

• Intra-tutor and Inter-tutor reliability need to
  be assured
• Practices and processes need to be transparently
  fair to all students
• Cheat and plagiarisers need to be
  deterred/punished (see bonus features)
• Assessment needs to be manageable for both staff
  and students
• Assignments should assess what has been
  taught/learned not what it is easy to assess.

31
Students benefit if we can make feedback timely

• Aim to get feedback on work back to students very
  quickly, while they still care and while there is
  till time for them to do something with it.
• The longer students have to wait to get work
  back, especially if they have moved into another
  semester by the time they receive their returned
  scripts, the less likely it is that they will do
  something constructive with lecturers
  hard-written comments.

32
Can we provide opportunities for multiple
assessment?

• Consider allowing resubmissions of work as part
  of a planned programme
• Students often feel they could do better once
  they have seen the formative feedback and would
  like the chance to have another go
• Particularly at the early stages of a programme,
  we can consider offering them the chance to use
  formative feedback productively
• Feedback often involves a change of orientation,
  not just the remediation of errors.

33
Using formative assessment to promote
independence and learning

• Investigate how learning can be advanced in small
  steps using a scaffolding approach
• Provide lots of support in the early stages when
  students dont understand the rules of the game
  and may lack confidence
• This can then be progressively removed as
  students become more confident in their own
  abilities.

34
Play fair with students by avoiding using final
language (Boud)

• Avoid destructive criticism of the person rather
  than the work being assessed.
• Try not to use language that is judgmental to the
  point of leaving students nowhere to go.
• Words like appalling, disastrous and
  incompetent give students no room to manoeuvre.
• However, words like incomparable and
  unimprovable dont help outstanding students to
  develop ipsatively either.

35
However

• The sound recording on this assignment is
  exemplary. I always feel as a tutor that I
  should give you ideas on how to improve your work
  further but I doubt you will ever in your life do
  a drum recording that is better balanced and
  pitched than this

36
Play fair by giving feedback to students with
diverse abilities

• Students at the top end of the ability range
  sometimes feel short changed by minimal feedback
• Students with many weaknesses easily become
  dispirited if there is too much negative
  feedback
• Consider giving an assessment sandwich. Start
  with something positive, go into the detailed
  critique and find something nice to say at the
  end (to motivate them to keep reading!)
• Explore ways to incentivise reading of feedback
• Consider which medium to use for students with
  disabilities (e.g. dont use bad handwriting for
  those with visual impairments or dyslexia!).

37
Conclusions

• Concentrating on giving students detailed and
  developmental formative feedback is the single
  most useful thing we can do for our students,
  particularly those who have had a struggle to
  achieve entry to higher education.
• Summative assessment may have to be rethought to
  make it fit for purpose
• To do these things may require considerable
  imagination and re-engineering, not just of our
  assessment processes but also of curriculum
  design as a whole if we are to move from
  considering delivering content the most important
  thing we do.

38
Useful references 1

• Biggs J (2003) Teaching for Quality Learning at
  University (Maidenhead SRHE Open University
  Press)
• Bowl, M (2003) Non-traditional entrants to higher
  education they talk about people like me Stoke
  on Trent, UK, Trentham Books
• Brown, S. Rust, C Gibbs, G (1994) Strategies
  for Diversifying Assessment Oxford Centre for
  Staff Development.
• Boud, D. (1995) Enhancing learning through
  self-assessment London Routledge.
• Brown, G. with Bull, J. and Pendlebury, M. (1997)
  Assessing Student Learning in Higher Education
  London Routledge.
• Brown, S. and Glasner, A. (ed.) (1999) Assessment
  Matters in Higher Education, Choosing and Using
  Diverse Approaches, Maidenhead Open University
  Press.
• Brown, S. and Knight, P. (1994) Assessing
  Learners in Higher Education, London Kogan Page.

39
Useful references 2

• Brown, S., Race, P. and Bull, J. (eds.) (1999)
  Computer Assisted Assessment in Higher Education
  London Routledge.
• Carroll J and Ryan J (2005) Teaching
  International students improving learning for
  all Routledge SEDA series
• Falchikov, N (2004) Improving Assessment through
  Student Involvement Practical Solutions for
  Aiding Learning in Higher and Further Education,
  London Routledge.
• Gibbs, G (1999) Using assessment strategically to
  change the way students learn, In Brown S.
  Glasner, A. (eds.), Assessment Matters in Higher
  Education Choosing and Using Diverse Approaches
  Maidenhead SRHE/Open University Press.
• Kneale, P. E. (1997) The rise of the "strategic
  student" how can we adapt to cope? in Armstrong,
  S., Thompson, G. and Brown, S. (eds) Facing up to
  Radical Changes in Universities and Colleges,
  119-139 London Kogan Page.

40
Useful references 3

• Knight, P. and Yorke, M. (2003) Assessment,
  learning and employability Maidenhead, UK
  SRHE/Open University Press.
• McDowell E Brown S 1998 Assessing students
  cheating and plagiarism, Red Guide 10/11
  University of Northumbria, Newcastle
• Mentkowski, M. and associates (2000) p.82
  Learning that lasts integrating learning
  development and performance in college and beyond
  San Francisco Jossey-Bass.
• Peelo, M and Wareham, T (eds) (2002) Failing
  Students in higher education Buckingham, UK,
  SRHE/Open University Press.
• Sadler, D R (1989) Formative assessment and the
  design of instructional systems Instructional
  Science 18, 119-144.
• Sadler, D R (1998) Formative assessment
  revisiting the territory Assessment in Education
  Principles, Policy and Practice 5, 77-84
• Pickford, R. and Brown, S. (2006) Assessing
  skills and practice London Routledge.

41
Useful references 4

• Race, P. (2001) A Briefing on Self, Peer Group
  Assessment in LTSN Generic Centre Assessment
  Series No 9 LTSN York. Race P. (2006) The
  lecturers toolkit (3rd edition) London
  Routledge.
• Race P (2006) The Lecturers toolkit 3rd edition
  London Routledge
• Race P and Pickford r (2007) Making Teaching
  work Teaching smarter in post-compulsory
  education, London, Sage
• Rust, C., Price, M. and ODonovan, B. (2003).
  Improving students learning by developing their
  understanding of assessment criteria and
  processes. Assessment and Evaluation in Higher
  Education. 28 (2), 147-164.
• Ryan J (2000)A Guide to Teaching International
  Students Oxford Centre for Staff and Learning
  Development
• Stefani L and Carroll J (2001)A Briefing on
  Plagiarism http//www.ltsn.ac.uk/application.asp?a
  ppresources.aspprocessfull_recordsectiongener
  icid10
• Yorke, M. (1999) Leaving Early Undergraduate
  Non-completion in Higher Education, London
  Routledge.

42
Bonus features

• Plagiarism

43
Cheating and plagiarism why have concerns about
them increased?

• Mass access to HE.
• More reported bad practice.
• Changes in assessment practice.
• Wider use of communication and information
  technologies, especially the web.
• Higher stakes the importance of getting good
  grades.

44
How do new technologies change things?

• Plagiarism used to be hard workhours in the
  library, researching what to copy..the
  plagiarist used to learn a lot while trying to
  get out of doing the work
• Jim Evans, University of Warwick, quoted by Jude
  Carroll, Oxford Brookes University.

45
What is plagiarism?

• Passing off someone elses work as your own.
• Wholesale lifting of entire assignments/ texts.
• Patching and paraphrasing.
• Purchasing or commissioning work.

46
What about unintentional plagiarism?

• Readers may not be conscious how much they have
  themselves absorbed
• Schools may encourage the learning and re-use of
  model answers
• In some cultures, your teacher or text books are
  honoured sources and there is nothing in
  appropriate about repeating their words.

47
Tell-tale signs

• Outdated or obscure references
• Dramatic changes in writing ability
• Inappropriate cultural references
• Layout peculiarities
• American spellings/vocabulary (other than from US
  students and those used by conventional spell
  checkers).
• Modified from Bill Johnson (2003) The concept of
  plagiarism Manchester Metropolitan University
  Learning and Teaching in Action Volume 2 No 1

48
Four strategies to control plagiarism and cheating

• Use strict controls.
• Make the rules clear and have known penalties
  (and apply them).
• Design assessment instruments that make cheating
  difficult.
• Develop a climate that will reduce the likelihood
  of cheating.

49
Design assessment instruments that make
cheating difficult

• Use openbook rather than closed book exams.
• Use assignments reliant on personal experience.
• Ask students to produce learning/reflective
  journals and critical incident accounts.
• Use vivas and orals.
• Design differentiated assignments.
• Provide assignments with choice and individual
  activity.

50
More anti-plagiarism assignments

• Use computer based assessment.
• Involve 3rd party verification.
• Ask them to provide photocopied annotated source
  material.
• Use group assessment.
• Involve an element of peer assessment.
• Give students tasks in learning teams.
• Monitor the production of assessed work use
  staged assignments.

51
Bonus features

• Using technologies to support assessment

52
Use technologies to support assessment for
learning

• Employ computer-assisted formative assessment
  with responses to student work automatically
  generated by email
• Students seem to really like having the chance to
  find out how they are doing, and attempt tests
  several times in an environment where no one else
  is watching how they do
• We can monitor what is going on across a cohort,
  so we can concentrate our energies either on
  students who are repeatedly doing badly or those
  who are not engaging at all in the activity.

53
Giving feedback electronically you can use

• Emailed comments from you to students on their
  individual work.
• As students sit the exam, put up generic feedback
  on the VLE
• Email overall generic comments on assignments to
  the whole cohort of students or through a
  computer conference.
• Use computer-delivered feedback. (There is an
  interesting OU research project currently being
  undertaken to give formative feedback to students
  on electronically submitted work).

54
Computer-based assessment is valuable to learning
when

• It provides students with multiple opportunities
  to test and re-test their understanding
• It offers formative feedback in response to wrong
  (and right) answers
• It blurs the distribution between teaching and
  assessment.

55
CAA requires capital investment because

• Any fool can produce CAA (and a lot of fools do)
• Good CAA requires good pedagogic understanding,
  up-to-date content and sound technological
  systems
• The investment of time to produce, pilot and test
  good CAA resources must be extensive.

56
Social networks can support assessment when

• There are clear purposes for the assessment of
  learning opportunities
• They foster a genuine shared learning
  environment, aiming to build a learning
  community
• Students feel equally able to contribute to
  network communities.

57
They are less successful when

• Academics attempt to hijack students own social
  networks for their own purposes
• There is no clear focus for learning or
  assessment activities in shared spaces
• Tasks and activities are minimal/ tokenistic.

58
Key questions on future TEL developments for
assessment

• How can we assess student learning effectively
  and efficiently in ways that go
  beyond multi-choice questions, so we are testing
  deeper learning behaviours than simply recall and
  memorisation?
• How can we evaluate more effectively what
  learning students have already and what more they
  need to do in order to achieve the outcomes we
  can recognise as being worthy of a degree or
  other award?
• How can we help students learn to differentiate
  between poor quality and good quality content?

59
More questions

• How can we help students learn cooperatively in
  ways that emulate but don't replicate their
  social learning spaces (they hate it when they
  find faculty staff intruding onto face book" and
  yet they don't use the learning spaces we set up
  for them much)?
• Are there technology supported templates we can
  develop which would be better than the cumbersome
  and people-intensive APEL (Accreditation of Prior
  Experiential learning) processes we currently
  use.

60
And some more

• How can we best make use of the technologies
  almost all students have nowadays for assessment
  (mobile phones, MP3 players, iPods, PDAs to
  support learning?
• Some of our distance learning students in
  developing countries, don't have much in the way
  of PCs or laptops but lots of them have mobile
  phones.