Title: Assessing instructional and assessment practice: What makes a lesson formative? CRESST conference, September 2004 UCLA Sunset Village, CA
1Assessing instructional and assessment
practiceWhat makes a lesson formative?CRESST
conference, September 2004UCLA Sunset Village, CA
- Dylan Wiliam
- ETS
- dwiliam_at_ets.org
www.dylanwiliam.net
2What makes a lesson formative?
- Moving from I know it when I see it
- Expanding ideas of formative assessment
- Feedback
- Elicitation and feedback
- The students role
- Systems thinking
3Preliminary assumptions
- The primary purpose of educational research is
the improvement of education. - The purpose of education is the improvement of
student achievement - The improvement of student achievement will be
achieved primarily through changes in what
happens in classrooms - The role of the teacher is not to teach but to
create situations in which students learn
4Functions of assessment
- For evaluating institutions
- For describing individuals
- For supporting learning
- Monitoring learning
- Whether learning is taking place
- Diagnosing (informing) learning
- What is not being learnt
- Forming learning
- What to do about it
5Effects of feedback
- Kluger DeNisi (1996)
- Review of 3000 research reports
- Excluding those
- without adequate controls
- with poor design
- with fewer than 10 participants
- where performance was not measured
- without details of effect sizes
- left 131 reports, 607 effect sizes, involving
12652 individuals - Average effect size 0.4, but
- Effect sizes very variable
- 40 of effect sizes were negative
6Effects of formative assessment
- Several major reviews of the research
- Natriello (1987)
- Crooks (1988)
- Black Wiliam (1998)
- Nyquist (2003)
- All find consistent, substantial effects
7Kinds of feedback (Nyquist, 2003)
- Weaker feedback only
- Knowledge or results (KoR)
- Feedback only
- KoR clear goals or knowledge of correct results
(KCR) - Weak formative assessment
- KCR explanation (KCRe)
- Moderate formative assessment
- (KCRe) specific actions for gap reduction
- Strong formative assessment
- (KCRe) activity
8Effects of formative assessment (HE)
No. Effect
Weaker feedback only 31 0.16
Feedback only 48 0.23
Weaker formative assessment 49 0.30
Moderate formative assessment 41 0.33
Strong formative assessment 16 0.51
9Formative assessment
- Classroom assessment is not (necessarily)
formative assessment - Formative assessment is not (necessarily)
classroom assessment
10Formative assessment
Assessment for learning is any assessment for
which the first priority in its design and
practice is to serve the purpose of promoting
pupils learning. It thus differs from assessment
designed primarily to serve the purposes of
accountability, or of ranking, or of certifying
competence. An assessment activity can help
learning if it provides information to be used as
feedback, by teachers, and by their pupils, in
assessing themselves and each other, to modify
the teaching and learning activities in which
they are engaged. Such assessment becomes
formative assessment when the evidence is
actually used to adapt the teaching work to meet
learning needs. Black et al, 2002
11Bad items/good items 1
- Which is larger 0.33 or 1/3?
- 0.33 is larger than 1/3
- 1/3 is larger than 0.33
- They are the same
- You need more information to be sure
12Bad items/good items 2
- What is the general rule for the following
- pattern 3, 7, 11, 15, ?
- 3 n
- n 4
- 3n 4
- 4n - 1
- 4n 3
13Bad items/good items 3
- If f g 8, then f g h ?
- 9
- 12
- 15
- 16
- 8 h
14Bad items/good items 4
- Which of the following statements are true ?
- A square has two 90 angles
- A square is a trapezoid
- A kites are squares
- A trapezoid is a rectangle
- A rectangle is a trapezoid
15Bad items/good items 5
- Ice-cubes are added to a glass of water. What
happens to the level of the water as the
ice-cubes dissolve? - A) The level of the water drops
- B) The level of the water stays the same
- C) The level of the water increases
- D) You need more information to be sure
16Bad items/good items 6
- What is the pH of gasoline?
- A) Less than zero
- B) Zero
- C) Greater than zero but less than 7
- D) 7
- E) Greater than 7
- F) None of the above
17Bad items/good items 7
- In what year did World War II begin?
- A) 1919
- B) 1937
- C) 1938
- D) 1939
- E) 1941
- F) 1942
18Good formative items are different
- Can have more (or less!) than one correct answer
- Items need to be generative
- of learning
- of insights into learning
- of insights into how to promote learning
- Increasing ability may reduce your chance of
getting the item correct - Distractors must be explicitly connected to
incorrect or incomplete conceptions (facets) - Item responses must provide clues to effective
action - Some items support both summative and formative
functions well most do not
19How do we make good items?
- Good items are not universal
- In some domains, facets of thinking are
reasonably universal (e.g. physics) - In most domains, facets of thinking are dependent
on instruction - So there is no universal set of facets
- And we have to rely on teachers to do it for
themselves - Good items have to be used
- Requires a model of effective, scalable teacher
professional development
20Why research hasnt changed teaching
- The nature of expertise in teaching
- Aristotles main intellectual virtues
- Episteme knowledge of universal truths
- Techne ability to make things
- Phronesis practical wisdom
- What works is not the right question
- Everything works somewhere
- Nothing works everywhere
- Whats interesting is under what conditions
does this work? - Teaching is mainly a matter of phronesis, not
episteme
21Countdown
3
25
1
4
9
Target number 127
22Aspects of formative assessment
Where the learner is Where they are going How to get there
Teacher Evoking information Curriculum philosophy Feedback
Peer Peer-assessment Sharing criteria Peer-tutoring
Learner Self-assessment Sharing criteria Self-directed learning
23A formative lesson
- Grade 9 Algebra lesson
- Weather balloon set off, rising at 8fs-1
- 2s later, a flare is launched, initial velocity
80fs-1 - Students complete a table and a graph to answer
three questions - Maximum height of flare
- How long to max height
- Average velocity of flare
24Regulation of learning
- Teaching as engineering learning environments
- Key feature well-regulated
- Teaching vs. learning
- Regulation of activity vs. regulation of learning
- Long feedback cycles vs. variable feedback cycles
- Quality control vs. quality assurance in learning
25Regulation of learning
- Proactive (upstream) regulation
- Planning regulation into the learning environment
- Planning for evoking information
- Interactive (downstream) regulation
- Negotiating the swiftly-flowing river
- Moments of contingency
- Tightness of regulation (goals vs horizons)
- Retrospective regulation
- Structured reflection (e.g. lesson study)
26(No Transcript)