James Christie

1
James Christie

• Automated Marking of Essays for Content

2
How long have essays been used for assessment?
Earliest recorded use is some 2,000 years BC by
the Chinese for their Administrators!
3
Essay Definition 1 of 2

• requires a response composed by the examinee,
  usually in the form of one or more sentence, of a
  nature that no single response or pattern of
  responses can be listed as correct, and the
  accuracy and quality of which can be judged
  subjectively only by one skilled or informed in
  the subject,

4
Essay Definition 2 of 2

• but even an expert cannot usually classify a
  response as categorically right or wrong. Rather,
  there are different degrees of quality or merit
  which can be recognized.
• attributed to
• Stalnaker, 1951

5
Possible criteria for automated essay marking

• Ease of creating a scoring schema
• Ability to score on various mark regimes
• Ease of identification on non-scoring elements
• Ease of modification should scoring error(s)
  occur
• Consistent and reproducible scoring
• Acceptability of results to human markers,
  essayists,
• Defensibility
• Accuracy and precision
• Coachability avoidance
• Cost

6
(No Transcript)
7
Model Essay

• The cat sat on the mat.

8
Marking Schema

• Mark Item
• 3 cat sat mat
• max3

9
Content Data Structure

• 1 z a a 0 2 3 4 0 3
  3 0 n 3
• 2 z f cat 1 0 0 0 0 0
  0 1 n 0
• 3 z f sat 1 0 0 0 0 0
  0 1 n 0
• 4 z f mat 1 0 0 0 0 0
  0 1 n 0

10
Essay Set

• ALPHA
• The cat sat on the mat.
• BRAVO
• The cat sat on the floor.
• CHARLIE
• The dog lay on the floor.
• MODEL
• The cat sat on the mat.

11
Process interface

• What LEVEL of Diagnostics to use 0 ... 3
  0
• What ESSAY SET to use ....................
  catmat
• Enter SCHEMA to use ......................
  catmat
• ALPHA.EXT .
• BRAVO.EXT .
• CHARLIE.EXT .
• MODEL.EXT .
• Started on Thursday, February 06 2003 at
  155905
• Finished on Thursday, February 06 2003 at
  155906

12
Schema Report

• Essay set ............. catmat
• Schema Report using ... catmat
• Entities 1 - 4
• Entity ID 1 2 3 4
• Entity Type a f f f
• Part ID's z
• Essay Name
• ALPHA.EXT y y y y
• BRAVO.EXT _ y y _
• CHARLIE.EXT _ _ _ _
• MODEL.EXT y y y y

13
Content Report

• Essay set .............. catmat
• Content Report using ... catmat
• Essay Name Words Sentences Usage Coverage
  Part z 3 Mark 3 100
• ALPHA.EXT 6 1 50.00 100.00
  3 3 100.00
• BRAVO.EXT 6 1 33.33 50.00
  0 0 0.00
• CHARLIE.EXT 6 1 0.00 0.00
  0 0 0.00
• MODEL.EXT 6 1 50.00 100.00
  3 3 100.00
• Started on Thursday, February 06 2003 at
  155905
• Finished on Thursday, February 06 2003 at
  155906
• Marked 4 file(s) scanned 4 file(s)

14
Marking Performance
Essay Set First v Second Markers Human v SEAR
A 0.704 / 0.700 0.594 / 0.596
B 0.810 / 0.740 0.404 / 0.376
C 0.164 / 0.277 0.302 / 0.394
D N/A 0.238 / 0.336
Pearson / Spearman Significance 0.01
0.05
15
Future work style

• obtain marked essays for style marking
• plain ASCII essays
• using a common set of metrics
• word-processed essays
• using a common set of metrics
• augmented with word-processing based metrics

16
Future work content

• maximise use of active and passive voices
• cope with spelling (and grammar) errors
• increased coverage of Blooms Taxonomy
• include non-textual feature(s)
• develop
• better feedback to the essayist
• better feedback to the examiner
• plagiarism detection mechanism(s)

17
If manual marking equalsDa Vincis Helical Screw,
18
then does SEAR equal the first powered flight?
19
Is the future equal to the ISS?
20
Is this the future for style and content?
English The cat sat on the mat.
Italian Il gatto era seduto sullo zerbino.
Greek I gata ekatse ston kanape.
Russian Koshka sidit na matrase.
French Le chat sest assis sur le tapis.
German Die Katze sass auf dem Teppich.
Dutch De kat zat op de mat.
Spanish El gato se sents en la alfombra.
etc
21
James R Christie

• The Robert Gordon University
• Faculty of Design and Technology
• School of Computing
• Room B23b St Andrew Street, Aberdeen
• Phone [phone]
• Fax [phone]
• E-mail j.christie_at_rgu.ac.uk
• URL
• http//www.comp.rgu.ac.uk/staff/jrc