e'scape task

1
Coursework assessment in high stakes
examinations authenticity, creativity,
reliability Richard Kimbell and Alistair
Pollitt Technology Education Research Unit
Goldsmiths University of London
2
Learners independent coursework geography
fieldwork science investigations design
projects Has always been part of high stakes
assessment
It was banned by Ministers from 2006 because of
its insecurity - not sure whose work it is -
the assessment is insufficiently reliable
3
project e-scape authentic performance
e-portfolios radical reliable
e-assessment phase 1 (2004-5) play-time gt
proof of concept phase 2 (2005-7) building
a prototype system phase 3
(2007-9) transferability and scalability
4
project e-scape prototype (2005-07) performance
e-portfolios live in the context of design
studios
5
e-scape phase 2 prototype
the drawing tool is critical to the activity
6
ideas are created and swopped around and
extended by team-mates
7
modelling remains central to the activity
design-talk as creative reflection
8
(No Transcript)
9
learners work is dynamically loaded into the
website where the e-portfolio progressively
emerges
10
Assessment methodology based on multiple
intersecting holistic judgements.
L.Thurstone (1927) law of comparative
judgement
A.Pollitt (2004) comparative pairs
methodology for reliability studies between
exam boards
e-scape (2007) the 1st complete data set
using comparative pairs for front line
assessment
11
249 portfolios 7 judges 3 rounds of judging
Round 1 (random) 21 sets of portfolios in
groups of 11/12 each judge has 142 chained
pairs (10 - 59 59 - 38 38 - 12) each set seen
by 2 judges each portfolio judged 8 times
Problems arose at the extremes - as 31
portfolios won or lost all The core converged
quickly
12
Round 2 (targetted) portfolios sorted by
parameter value again divided into 21 sets
each judge assigned two sets from the high, mid
and low sets portfolios much closer in quality
- median separation 0.08 logits - round 1
was 2.79 logits arguably round 2 was too close
and therefore too hard but statistical
efficiency raised from average of 36 to 94
Combined data from round 1 and 2 created a scale
at least as good as conventional marking
13
multiple judges (7) multiple comparisons (17)

reliability coefficient 0.931
14
Some interesting assessment issues
1. robustness
250 portfolios 250 x 250 matrix 20
comparisons each very thin data matrix 96
empty But it worked
How sparse can the matrix be? Current plans are
for 12 targeted judgements
15
2. Optimal difference between pairs
Portfolios that are about 0.7 logits apart on the
emerging scale. A gap of 0.7 logits means
that the odds that A will beat B are 21, or
0.67
16
the first 10 judgements do most of the work
the second 10 virtually fix the scale
3. Reducing Standard Error with more judgements
thereafter the gain in accuracy per extra
judgement is small
17
4. Identifying problematics
the system immediately identifies portfolios
where judges disagree
and individual profiles show the consensuality
of the judges
18
project e-scape phase 2
(2005-7) building a prototype system
The prototype system was seen to work -
authentic learner performance - reliable
assessment outcome All work was presented to
Ministers With recommendations for next-steps
Phase 3 of project e-scape was then
commissioned - with two priorities
19
Transferability Scalability raising new
research questions
Transferability Do e-scape activities and
assessments work in other subjects? at
other age groups?
Scalability Do e-scape activities and
assessments work when run by schools and exam
boards? on a huge scale (0.5 million entries) ?
20
A new tool to explore transferability
The activity authoring tool Structured Coursework
Assessment Activities (SCAA) (currently alpha
beta by mid Feb release version by end Mar)
created by teachers authored on a PC/laptop
to run the class activity on the Amios outline
a task break into sub-tasks (timed) each with
an output specification resulting in a
web-portfolio
21
The importance of science The study of science
fires pupils' curiosity about phenomena in the
world around them and offers opportunities to
find explanations. It engages learners at many
levels, linking direct practical experience with
scientific ideas. Experimentation and modelling
are used to develop and evaluate explanations,
encouraging critical and creative thought, and to
show how knowledge and understanding in science
are rooted in evidence. Pupils discover how
scientific ideas contribute to technological
change - affecting industry, business and
medicine and improving quality of life. They
trace the development of science worldwide and
recognise its cultural significance. They learn
to question and discuss issues that may affect
their own lives, the directions of societies and
the future of the world.

• Science
• the e-scape task Distractors and
• traffic accidents

22

• Geography
• the e-scape task re-branding Porthcawl
  (note the challenge of fieldwork)

a sense of wonder about places
to think spatially
how people and environment interact
to become global citizens by exploring their
own place in the world
23
A new tool to explore scalability
the pairs engine
as a judgement is fed in the engine
dynamically selects the next pair to be judged
aimed at maximising the efficiency of judging
capable of taking pairs and triples and
up to sixes
linking into Exam Board (and QCA) back-end
systems
24
scalability and the culture of schools
With the authoring tool and the pairs engine how
much training is needed (and how much does it
cost)
We have hot and cold schools Hot schools
have the kit for a whole term to develop their
own activities and run their own (school
based) assessment Cold schools get dropped on
just for the test (but have to be able to run it
with minimum support)

Maximum potential . Minimum requirement
25
project e-scape phase 3 (2007-9)
Trials complete end of March 08 Finalised
software April 08 System active May 08 Main
survey June/July 08 Pairs judging Sept/Oct
08 Data analysis Nov/Dec 08 Report by QCA/exam
boards Jan 09 Final research report Mar 09
.. and then we launch phase 4
26
This e-scape research is aimed at preserving
and transforming the role of course-work
assessment in high stakes examinations at
creating new protocols for the adoption of mobile
learning in schools at empowering
teachers to reflect the vision of their
subjects into learning activities at guiding
education policy and practice
e-scape is technologically-rich but the project
is really about good learning
27
project e-scape e-solutions for creative
assessment in portfolio environments
Contact us at
Richard Kimbell / Tony Wheeler / Chloe Nast
r.kimbell_at_gold.ac.uk