Integrated assessment: challenges and opportunities

1
Integrated assessment challenges and
opportunities

• Dylan Wiliam
• ETS

2
Uses of educational assessments

• Formative Supporting learning
• Summative Certifying individuals
• Evaluative Evaluating educational provision

3
Formative and summative

• Fine-scaled data that supports formative uses can
  be aggregated to serve a summative function
• Aggregate summative data cannot be dis-aggregated
  to identify learning needs
• Assessment for formative purposes should be the
  foundation of all assessment in schools and
  colleges

4
Formative assessment

• Frequent feedback is not necessarily formative
• Feedback that causes improvement is not
  necessarily formative
• Assessment is formative only if the information
  fed back to the learner is used by the learner in
  making improvements
• To be formative, assessment must include a recipe
  for future action

5
The problem

• Most assessment systems provide poor answers to
  the wrong questions, have negative backwash
  effects and direct attention on the assessment of
  learning, rather than assessment for learning
• Sins of commission
• Reliability (too many wrong levels/grades)
• Validity (misleading levels/grades)
• Social consequences (backwash effects)
• Sins of omission
• Focus (summative rather than formative)

6
Reliability

• With an 85 reliable test, in a cohort of 30
• Half the students will get a score within 4 of
  their true score
• Three-quarters of the students will get a score
  within 7 of their true score
• One candidate will get a score differing from
  their true score by more than 12

7
Reliability v consistency

• Classical measures of reliability
• are meaningful only for groups
• treat negative and positive errors equally
• average out over the whole score scale
• are misleading when decisions are made on test
  scores
• Decision theoretic approaches
• look at the accuracy of diagnostic systems
• can distinguish false-positive and
  false-negatives
• can include costs and benefits of decisions

8
Validity

• A property of a test
• the responses to a test
• the inferences drawn from a test
• One validates not a test, but an interpretation
  of data arising from a specified
  procedure(Cronbach, 1971 p447, original
  emphasis)

9
Reliability and validity

• Reliability the extent to which inferences about
  the items actually assessed are warranted
• Content validity the extent to which inferences
  about the domain from which the items were
  selected are warranted
• Predictive and concurrent validity the extent to
  which inferences beyond the domain from which the
  items were selected are warranted

10
Validity argument
concurrent predictive validity, utility
content validity
social consequences
value implications
after Messick (1980)
11
The Macnamara Fallacy

• The first step is to measure whatever can be
  easily measured. This is OK as far as it goes.
• The second step is to disregard that which cant
  easily be measured or to give it an arbitrary
  quantitative value. This is artificial and
  misleading.
• The third step is to presume that what cant be
  measured easily really isnt important. This is
  blindness.
• The fourth step is to say that what cant be
  easily measured really doesnt exist. This is
  suicide.(Charles Handy, The empty raincoat, 1994
  p219).

12
Lake Wobegon

• All the women are strong, all the men are
  good-looking, and all the children are above
  average

X
13
Improvements are limited

• In England
• scores on national tests in mathematics and
  English for 11-year olds are increasing
• scores on other, comparable, tests have remained
  constant
• so, improvement is limited to those things that
  are actually tested

14
Improvements are transient

• Proportion of 11 year olds achieving grade level
  in mathematics has increased steadily over the
  last five years
• According to a survey report, 25 of those who
  achieved grade level at the end of grade 5 fail
  to achieve the same level at the end of grade 6.
• So, achievement has improved at age 11, but not
  at age 12.

15
Goodharts law

• All performance indicators lose their meaning
  when adopted as policy targets
• Inflation and money supply
• Railtracks performance targets
• National Health Service waiting lists
• National curriculum targets
• The clearer you are about what you want, the more
  likely you are to get it, but the less likely it
  is to mean anything

16
Validity

• Validity is an overall evaluative judgement of
  the adequacy and appropriateness of inferences
  and actions based on test scores (Messick 1988
  p42)
• An assessment is valid to the extent that you are
  happy for teachers to teach towards the test

17
The (only?) solution

• Involve teachers in summative assessment
• Increases reliability and validity
• Externalize standards
• Locates teacher as coach, not judge
• Requires teachers to form a community of
  practice

18
Referents of assessment

• Norm-referenced
• Cohort-referenced
• Criterion-referenced
• Ipsative
• Construct-referenced

19
Criterion-referenced assessment

• John where Paul had had had had had had had had
  had had had a clearer meaning
• John, where Paul had had had, had had had
  had. Had had had had a clearer meaning.

20
Speech acts

• Perlocutionary speech acts are statements about
  what was, is or will be (eg this person will make
  a good doctor)
• Illocutionary speech acts are performative they
  create social facts (eg I now pronounce you
  husband and wife)

21
Social facts

• Interviewer Did you call them the way you saw
  them, or did you call them the way they were?
• Umpire The way I called them was the way they
  were.

22
Construct-referenced assessment

• Criteria do not define but exemplify grades
• Standards are shared by the community of
  practice
• Standards are implicit and evolve

23
Quality

• Maxims cannot be understood, still less applied
  by anyone not already possessing a good practical
  knowledge of the art. They derive their interest
  from our appreciation of the art and cannot
  themselves either replace or establish that
  appreciation.(Polanyi, 1958 p50).
• Quality doesnt have to be defined. You
  understand it without definition. Quality is a
  direct experience independent of and prior to
  intellectual abstractions.(Pirsig, 1991 p64).

24
Standard-setting

• Standard setting amounts to a claim that a
  particular assessment outcome has a particular
  meaning.
• Setting a standard requires the identification of
  a performance standard with a particular score or
  set of scores
• passing scores are arbitrary
• performance standards are arbitrary

25
Development process

• Traditional development process
• Standardsgt Curriculumgt Assessment
• Result assessment drives standards in a
  disjointed way
• Integrated development process
• Standards, Curriculum and Assessment developed
  together

26
What do we need?

• Effective summative assessment
• requires teachers to share a construct of quality
• Effective formative assessment
• Requires students to share the same construct of
  quality
• Requires teachers to posses an anatomy of quality

27
(No Transcript)
28
Mathematics in context

• Mathematics in the real world
• Fluid v crystalized mathematics
• Invisible mathematics
• Cabbage

29
Issues in real-world math

• Student centering the math vs student centering
  the assessment
• Inevitable mathematics
• Problem posing vs. problem de-posing
• Separate vs. connected knowing
• Models of differentiation
• Extent of progress vs. degree of difficulty
• Support model of assessment