Can we design a supportive assessment system

1
Can we design a supportive assessment system ?

• Paul BlackSchool of EducationKings College
  London

2
Outline

• Purposes
• Fitness for purpose
• Whats it like now ?
• Improvement - some building blocks

3
Purposes

• For learning, and as learning
• For individuals decisions
• Transfer year to year
• Transfer school to school
• Transfer beyond school
• For monitoring
• Of institutions
• Of national standards

4
Fitness for Purpose

• Dependable
• Reliable and valid
• Reflect aims of learning and dont mislead
• Do no harm positive backwash
• For pupils
• For teachers
• For schools
• For all users
• Synergy not contradiction or conflict
• Cost - time and money

5
Whats it like now ?Assessment for, as, in
learning

• Strong potential but weak practice
• TINA
• Radical, fundamental, slow P.D. crucial

6
Seeing the need for change

• They and I knew that if the QA wasnt going
  smoothly, Id change the question, answer it
  myself or only seek answers from the brighter
  students. There must have been times (still
  are?) where an outside observer would see my
  lessons as a small discussion group surrounded by
  many sleepy onlookers.
• I had always been wary of wrong answers. How do
  you respond to a child who willingly,
  enthusiastically puts their hand up, and every
  time has picked up the wrong end of the stick (or
  even a different stick altogether). I want to
  encourage children to answer in class, but I need
  to challenge incorrect responses, so Id usually
  end up with a lame Thats an interesting way of
  looking at it, or Hmm, not quite what I had in
  mind. No one learnt anything from this exchange
  except the willing student, who gradually learnt
  to be less willing.
• James Two Bishops School

7
Involving pupils

• There have been two very positive results from
  this approach. The most significant one is that
  because they have to explain their answers each
  time orally this has carried through to their
  written work and now they set out their answers
  fully without being prompted. The second one is
  with a girl with a statement for being unable to
  talk or communicate with an adult. Having got
  used to the extra thinking time she now offers
  answers orally and will now tentatively explain
  her answers.
• (Gwen, Waterford School)

8
The Teachers Role

• There was a definite transition at some point,
  focusing on what I was putting into the process
  and what the pupils were contributing.
• It became obvious that one way to make a
  significant sustainable change was to get the
  pupils to do more of the thinking. I then began
  to search for ways to make the learning process
  more transparent to the pupils. Indeed I now
  spend my time looking for ways to get pupils to
  take responsibility for their learning at the
  same time making the learning more collaborative.

9
Making a difference

• Finally, the project has impressed upon me the
  need to encourage independence in students too
  often they expect to be passively spoon-fed a
  syllabus whilst we succeed not in extending their
  creativity, but in quashing it. By the careful
  use of questioning, by encouraging students to
  critically reflect on their own and on others
  work and by making them partners in the teaching
  and learning process I believe we can make a real
  difference for the better.

10
Changing the students role

• a number of pupils are content to get by
  Every teacher who wants to practice formative
  assessment must reconstruct the habits acquired
  by his pupils.(Switzerland/Perrenoud 1991)
• Now I know she is interested in what I think,
  not in whether I can get the right answer
  (school student)
• The hard thing is not getting new ideas into
  their heads, its getting the old ones out.
  (teacher)
• They feel that the pressure to succeed in tests
  is being replaced by the need to understand and
  the test is just an assessment along the way of
  what needs more work and what seems to be fine.
  (teacher)

11
Whats it like now ?Assessment for, as, in
learning

• Strong potential but weak practice
• TINA
• Radical, fundamental, slow P.D. crucial
• Successes and failures
• KMOFAP
• Scottish Education Department
• Intitatives in England
• The scale-up problem

12
Guiding not judging

• A comprehensive review of research studies of
  feedback showed that feedback improved
  performance in 60 of them. In the cases where it
  was not helpful, the feedback turned out to be
  merely a judgment or grading with no indication
  of how to improve (Kluger DeNisi, 1996).
• In a competitive system, low attainers attribute
  their performance to lack of ability, high
  attainers to their effort in a task oriented
  system, all attribute to effort, and learning is
  improved, particularly amongst low attainers
  (Craven et al. 1991).
• Feedback given as rewards or grades enhances ego
  rather than task involvement and can damage the
  self-esteem of low attainers. Feedback which
  focuses on what needs to be done can encourage
  all to believe that they can improve. (Dweck
  1986).

13
Helpless or confident?

• Helpless children -
• Are motivated by a desire to be seen to do well
• Seem to accept that they will fail because they
  just are not clever enough
• Believe that if something seems too hard there
  is nothing they can do about it
• Mastery children-
• Are motivated by desire to learn
• Will tackle difficult tasks in flexible and
  reflective ways
• Are confident of success, believing that they
  can do it if they try
• Kathy Silva

14
Whats it like now?For Individuals Decisions A

• The reliability scandal
• Do 30 of pupils get the wrong grade?
• We know there is error but we dont know how big
  it is
• Does it matter ?
• Unfair or mis-guided decisions for or by
  learners
• Making good progress
• test again and again and again standards will
  go up
• Optimising the system not possible without
  information

15
Summative a Fair Test ?

• To determine anyones life chances on the
  strength of a few hours of stressful work in the
  artificial environment of the examination room
  seems strange. There are few aspects of desirable
  performance for which the formal timed written
  test is a defensible surrogate. Examiners do
  their best in setting a variety of written tasks
  to reflect as wide a range of types of knowledge
  and understanding as possible within the test
  constraints, but the limitations are inescapable.
  No novelist, historian, or mathematician would
  want their effectiveness judged under such
  conditions.

16
The influence of current summative

• As currently constituted external assessment in
  school science education would appear to have a
  malign effect on the teaching of science,
  encouraging teachers to teach by transmission
  which, in turn, result in negative student
  attitudes towards school science.
• Too often assessment in school science supports
  a practice which sees science as a body of
  knowledge to be learnt rather than as a way of
  knowing which has transformed the world in which
  we live.
• Royal Society Report 2004

17
Whats it like now?For Individuals Decisions B

• Validity - does it tell us what we need to know
• Effects on teachers
• teach to the test
• summative skills weakened
• Effects on pupils
• wrong view of learning
• competition
• stress not enjoyment

18
Whats it like now?For monitoring

• Focus on accountability of institutions
• Using data of limited range and validity
• Monitoring of national standards is weak
• Self defeating Goodharts Law
• Teachers skills and status de-valued
• and impoverished
• Interest of the School vs. the interest of the
  Pupils

19
High-Stakes Testing Is Putting the Nation At Risk

• Because so much depends on how students perform
  on tests, it should not be surprising that, as
  one Florida superintendent noted, "When a
  low-performing child walks into a classroom,
  instead of being seen as a challenge, or an
  opportunity for improvement, for the first time
  since I've been in education, teachers are seeing
  that child as a liability." Shouldn't we be
  concerned about a law that turns too many of the
  country's most morally admired citizens into
  morally compromised individuals?
• The scores we end up praising and condemning in
  the press and our legislatures are actually
  untrustworthy, perhaps even worthless.
• The law makes all who engage in compliance
  activities traitors to their own profession. It
  forces education professionals to ignore the
  testing standards that they have worked so hard
  to develop.
• David C. Berliner Sharon L. Nichols

20
Teaching a moral enterprise

• To ask of other human beings that they accept and
  memorise what the science teacher says, without
  any concern for the meaning and justification of
  what is said, is to treat those human beings with
  disrespect and is to show insufficient care for
  their welfare.
• It treats them with a disrespect, because
  students exist on a moral par with their
  teachers, and therefore have a right to expect
  from their teachers reasons for what the
  teachers wish them to believe.
• It shows insufficient care for the welfare of
  students, because possessing beliefs that one is
  unable to justify is poor currency when one needs
  beliefs that can reliably guide action.
• S. Norris (Alberta), 1997 in Science Education

21
ImprovementSome building blocks A

• Strengthen formative
• Strengthen teachers summative
• To measure attainment of individuals
• use combination of school and external
• calibration, or moderation, or accreditation
• take progression seriously graded assessment

22
Towards a better way ?

• The national system should employ tests for which
  a wide range of modes of presentation, operation
  and response should be used so that each may be
  valid in relation to the attainment targets
  assessed. These particular tests should be
  called "standard assessment tasks" and they
  should be so designed that flexibility of form
  and use is allowed wherever this can be
  consistent with national comparability of
  results.
• (TGAT Paragraph 50)
• A mixture of standardised assessment instruments
  including tests, practical tasks and observations
  should be used in the national assessment system
  in order to minimise curriculum distortion.
• (TGAT Paragraph 59)

23
Use teachers evidence about their pupils

• Teachers' ratings of pupil performance should be
  used as a fundamental element of the national
  assessment system. Just as with the national
  tests or tasks, teachers' own ratings should be
  derived from a variety of methods of evoking and
  assessing pupils' responses.
• (TGAT Paragraph 60)
• The national assessment system should be based on
  a combination of moderated teachers' ratings and
  standardised assessment tasks.
• (TGAT Paragraph 63)

24
Support and use teachers summative judgments

• Group moderation should be an integral part of
  the national assessment system. It should be
  used to produce the agreed combination of
  moderated teachers' ratings and the results of
  the national tests.
• (TGAT Paragraph 77)
• The final reports on individual pupils to their
  parents should be the responsibility of the
  teacher, supported by standardised assessment
  tasks and group moderation.
• (TGAT Paragraph 80)

25
ImprovementSome building blocks A

• Strengthen formative
• Strengthen teachers summative
• To measure attainment of individuals
• use combination of school and external
• calibration, or moderation, or accreditation
• take progression seriously graded assessment
• Revise accountability
• Replace market model by community model

26
Accountabilityback to the future ?

• The only form in which results of national
  assessment for, and identifying, a given school
  should be published should be as part of a
  broader report by that school of its work as a
  whole.
• (TGAT Paragraph 132)
• Any report by a school which includes national
  assessment results should include a general
  report for the area, prepared by the local
  authority, to indicate the nature of
  socio-economic and other influences which are
  known to affect schools. This report should give
  a general indication of the known effects of such
  influences on performance.
• (TGAT Paragraph 134)

27
ImprovementSome building blocks B

• Learn to manage change
• Learn the lessons of professional development
• Use genuine consultation, pilots, trials
• Evaluate before going further
• Tackle the obstacles
• trust teachers, trust researchers,
• consensus not rule from the top
• public understanding

28
Summative Evaluation

• The fact that it was then welcomed by the Labour
  party, the National Union of Teachers and the
  Times Educational Supplement was enough to
  confirm for me that its approach was suspect. It
  proposed an elaborate and complex system of
  assessment - teacher dominated and uncosted. It
  adopted the 'diagnostic' view of tests, placed
  the emphasis on teachers doing their own
  assessment and was written in an impenetrable
  educationalist jargon.
• (Margaret Thatcher's memoirs)

29
Illuminative evaluation

• The British pedagogue's hostility to written
  examinations of any kind can be taken to
  ludicrous extremes. The British left believe that
  pencil and paper examinations impose stress on
  pupils and demotivates them. . . . This
  remarkable national obsession lies behind the
  more vehement opposition to the recent
  introduction of 7 year old testing. They were
  made a little too complicated and we have said we
  will simplify them. . . The complications
  themselves were largely designed in the first
  place in an attempt to pacify opponents who
  feared above all else 'paper and pencil' tests. .
  . This opposition to testing and examinations is
  largely based on a folk memory in the left about
  the old debate on the 11-plus and grammar
  schools.
• (Kenneth Clarke 1991)

30
1992 Chair of the HMC

• for the child, the encounter with the teacher
  is the first major step into outside society, the
  beginning of a long journey towards adulthood, in
  which the role of the teacher is going to be
  decisive. .
• . . . the profession of teaching is seen as a
  complement to the vocation of parenthood. . .
• Teachers are , therefore, not in the first
  instance agents either of the National Curriculum
  Council (or whatever follows it) or of the state.
  They are bridges between individual children and
  the culture to which they belong.
• . . . . This culture consists partly of a
  heritage, which links them to the past, and
  partly of a range of skills and opportunities,
  which links them to the future.
• The role of the teachers is, in this respect,
  irreplaceable.
• (Milroy 1992, pps 57-59)