Connoisseurship, coursework and the credibility of teacher assessments

1
Connoisseurship, coursework and the credibility
of teacher assessments

• Dr. Jonathan H Robbins
• The Talent Centre Ltd

2
Credibility is an issue
This might be the exception rather than the rule
but public perceptions of the extent to which
assessments made by teachers can be trusted are
an issue that must be addressed. Such
perceptions are not susceptible to rational
argument or the production of facts like
statistics or assertions that all is well.
3
Public credibility and social meanings

• The use of terms like evidence and witness
  statement create an adversarial context in which
  a candidate, and by implication the teacher is to
  be found guilty of failing or passing a
  qualification or award
• evidence and proof have become more important
  than process and judgement
• At the heart of this are perceptions about the
  extent to which judgements made by education
  professionals can be trusted

4
Different traditions purposes
In very general terms, the North American
tradition tends to emphasis metrics and
prediction, whereas the European tradition
emphasises judgement and results in illocutionary
pronouncements about what has been achieved.
5
Connoisseurship

• Three characteristics of a connoisseur
• the person is qualified to make judgements
• the exercise of critical faculties is based on
  knowledge and experience
• there is an ability to make comparisons in
  relation to perceived qualities
• Authentic connoisseurship is achieved through a
  process of induction into the community of
  assessors it appears to be an iterative process
  that generates hermeneutical understandings that
  are then used to mediate the norm referenced
  assumptions used as the basis for judgements

6
The connoisseur as assessor

• Applies metacognitive knowledge through a process
  of sitting down and valuing together (with the
  learner, the outcomes, by observation of process,
  performance etc)
• Is concerned with the declarative knowledge the
  assessor has about the interplay between personal
  characteristics, task characteristics and the
  available strategies in a learning situation
• See Flavell, J. (1979) Metacognition and
  cognitive monitoring, American Psychologist,
  Vol. 45, pp. 906-911.
• Norm-referencing mediated by past experience and
  is used to describe a threshold for performance.
• See Christie T and Forrest G M (1980) - Schools
  Council Research Studies Defining Public
  Examination Standards, Macmillan,56.

7
Affective qualities 1

• Morrison and McIntyre (1969) ranked the aspects
  of pupils most discussed by a group of primary
  school teachers (n56). While the first, general
  ability, is commonly assessed by formal means,
  the others, carelessness, laziness,
  talkativeness, cooperativeness, persistence,
  courtesy, ability to use language and
  originality, are largely assessed by teacher
  observation of overt pupil behaviour.

8
Affective qualities 2

• ... characteristics, of which more than 50 felt
  some note should be made by each teacher, were
  Interest (83), Perseverance (85), Reliability
  (77), Effort (77), Acceptance of Discipline
  (74), Carefulness (76), Enterprise (72),
  Co-operation (64), Responsibility (60),
  Attendance (65), Punctuality (57), Confidence
  (55) and Self-reliance (50).
• Pupils in Profile Project (Dockrell Broadfoot,
  1977), teachers (n 375 in seven schools)
• From Black, H, D., and Dockrell, W, B., (1979)
  Assessment in the Affective Domain - Do we, Can
  we, Should we? British Educational Research
  Journal, 6.2, 1980, p 197-208

9
Affective qualities 3

• The top 10 competencies sought by employers
  (based on website search by Edwards (2001) ) are
• 1. Flexibility, adaptability and the capacity to
  cope with and manage change (88)
• 2. Self-motivation and drive (88)
• 3. Analytical ability and decision making (75)
• 4. Communication and Interpersonal skills (75)
• 5. Team working ability and skills (63)
• 6. Organisation, planning and prioritisation
  abilities (50)
• 7. Customer focus, and service orientation (25)
• 8. Ability to innovate (25)
• 9. Mental and physical resilience (25)
• 10. Leadership ability (25)
• Edwards A. M. Clear, F. (2001) Supporting the
  Collaborative Learning of Practical Skills with
  Computer mediated Communications Technology,
  Educational Technology Society, 4(1)

10
Same process - different purposes
Tis with our Judgments as our Watches, none go
just alike, yet each believes his own
Alexander Pope (16881744), Essay on
Criticism
11
Purposes and paradigms

• ... the purposes of coursework are two fold.
  Developing skills and collaborative working. What
  is actually assessed at the moment is not
  necessarily one of the prime purposes of
  coursework. (There is) the need for coursework in
  terms of a collaborative development. When we
  think of coursework, we think of individual
  pieces of work, perhaps ... we are missing an
  opportunity (and) need to develop opportunities
  in the future, for that type of collaborative
  work to be part of the assessment model.
  (Verbatim statement by a teacher discussing the
  assessment of coursework)

12
Credibility

• What is needed is an explicit and agreed
  understanding not just of the purposes of
  coursework but also of what may be properly
  assessed, together with the limitations of such
  assessments. Without an explicit understanding of
  purpose that is shared by all involved,
  assessment lacks validity and consequently is
  unreliable.
• So - what does credibility mean when assessment
  is more an art than science?

13
Clearly stated limitations
The repeatability and reproducibility of
assessor judgements are a key factor in ensuring
dependable and therefore credible results. It is
not possible to control for all the variables at
work in assessments of coursework even if
purposes can be agreed. If we are going to find
new paradigms for assessment it is necessary to
shift away from metrics, specifications, criteria
and indicators or targets to being able to
demonstrate that results are within acceptable,
published bounds that are relevant, believable
and appropriate.
14
Crossing boundaries
Connoisseurship, coursework, credibility ... The
time has come for a shift in emphasis - away from
tests, targets, checklists and metrics to
judgements made by real, live human beings who
can deal with the things we cannot tell. This
means focusing on the trustworthiness and
credibility of assessors and assessment decisions
and a different set of assessment principles
born of educational, rather than measurement,
priorities. To do this Educational assessment
must be understood as a social practice, an art
as much as a science, a humanistic project with
all the challenges this implies and with all the
potential scope for both good and ill in the
business of education. Quotations from
Broadfoot, P. and Black P. (2004) Redefining
assessment? The first ten years of Assessment in
Education. Assessment in Education, Vol. 11, No.
1, March 2004
15
(No Transcript)