Try once, refine once a pattern for formative eassessment

1
Try once, refine once(a pattern for formative
e-assessment)

• Aliy Fowler
• University of Kent
• 28-04-09

2
Pattern summary

• A two-step approach to assessment/question
  answering/problem solving

• Encourages students to carefully consider initial
  answers
• Then, having received feedback, to give equal
  consideration to refining/correcting

3
The initial problem (source case)

• A University ab-initio Spanish module
• Large student numbers
• Skills-based course
• Provision of sufficient formative assessment
  meant unmanageable marking loads
• Impossible to provide immediate feedback
• leading to fossilisation of errors

4
The solution (source case)

• A CALL system designed to enable students to

• Independently practise sentence translation
• Receive immediate (and robust) feedback on all
  errors
• Attend immediately to the feedback (before
  fossilisation can occur)

5
The Pattern
6
Wider applicability

• Any skills-based learning situation where
• Multiple errors/misconceptions are possible
• Feedback can be given which identifies the
  location/type of errors
• without revealing the correct answer(s)
• Feedback generation (and grading?) can be
  automated
• otherwise the resubmission element will
  contribute to staff overloading
• it needs to be E-assessable!

7
Wider applicability

• Possible subject areas
• Mathematics
• Logic
• Computer Science
• algorithms/programming
• databases
• mark-up languages
• CSS
• MFL

8
Some theoretical justification

• Ferreira Atkinson (2009) divide feedback
  strategies for language learning into
• GAS (Given-Answer Strategies)
• target forms corresponding to students errors
  are given
• PAS (Prompting-Answer Strategies)
• students pushed to notice errors in their
  responses and repair the errors for themselves
• Found that in a tutorial context PAS seemed to
  promote more constructive student learning

9
Some theoretical justification

• Nicol Macfarlane (2006) maintain that good
  feedback practice includes
• Activities which encourage reflection on both the
  processes and products of learning
• Providing opportunities to close the gap between
  current and desired performance
• Providing opportunities to repeat the same
  task-performance-feedback cycle
• for example by allowing resubmission

10
Some theoretical justification

• Sadler (1989)
• Can only tell whether learning results from
  feedback if students have the opportunity to act
  on the feedback
• Boud (2000)
• Unless students can use feed- back to produce
  improved work, neither they nor their teachers
  can gauge its efficacy

11
How is the final mark arrived at?

• The two submissions are unequally weighted
• Best to give more weight to the first attempt
• since this ensures that students give careful
  consideration to the construction of their first
  answer
• but can improve their mark by refining the answer
• The marks ratio can vary (depending on
  assessment/feedback type)
• the more information given in the feedback, the
  lower the weight the second mark should carry

12
How is the final mark arrived at?

• If the ratio is skewed too far in favour of the
  first attempt
• students are less inclined to try hard to correct
  non-perfect answers
• If the ratio is skewed too far in favour of the
  second attempt
• students exhibit less care over the construction
  of their initial answer

13
Why try once, refine once?

• The resubmission limit is important
• Prevents a mindless iterative approach to
  solving the problem

• In which students begin with a stab-in-the-dark
• Then proceed by allowing the system/tutor to
  guide them step-by-step to the correct answer
• often via numerous minimally altered attempts
• with little critical engagement

14
More theoretical justification

• Looking at the teaching of programming
• Turkle Papert (1990)
• Used the term bricolage to refer to the try it
  and see approach
• Deemed it a valid alternative to the planning
  approach
• However... Ben-Ari (2001)
• Bricolage is not an effective epistemology for
  dealing with the massive amount of detailed
  knowledge must be constructed and organized in
  levels of abstraction sic

15
More theoretical justification

• Researching automatic feedback and resubmissions
  in Computer Science
• Malmi Korhonen (2004)
• Found results indicating that allowing high or
  unlimited numbers of resubmissions discouraged
  active pondering
• learners do not concentrate on finding the errors
  in their programs on their own.
• they use the automatic assessment system as a
  kind of debugger Try something and look at if
  it works

16
More theoretical justification

• And in a follow-up paper Malmi Korhonen (2005)
  noted
• When multiple submissions were permitted, about
  10 of students spent an unreasonable amount of
  time on exercises
• when measured against their success in the
  examination

17
More theoretical justification

• Hattie Timperley (2007)
• Receptivity to feedback can be affected by the
  degree of confidence students have in the
  correctness of their responses
• Kulhavy Stock (1989)
• Feedback has its greatest effect when a learner
  expects his/her response to be correct and it
  turns out to be wrong
• since the learner will study the item more
  intently in order to correct the misconception

18
More theoretical justification

• With the Try once, refine once pattern a higher
  proportion of the marks are given for the first
  attempt

• So students are likely to give initial answers in
  which they have a considerable degree of
  confidence
• Thus if an answer is found to be incorrect, it is
  then that feedback will be most effective

18
19
References

• Ben-Ari, M. (2001). Constructivism in computer
  science education. Journal of Computers in
  Mathematics and Science Teaching, 20 (1), 4573.
• Boud, D. (2000). Sustainable assessment
  rethinking assessment for the learning society.
  Studies in Continuing Education, 22 (2), 151-167.
  
• Ferreira, A. Atkinson, J. (2009). Designing a
  feedback component of an intelligent tutoring
  system for foreign language. Knowledge Based
  Systems, doi10.1016/j.knosys.2008.10.012
• Hattie, J. Timperley, H. (2007). The power of
  feedback. Review of Educational Research, 77,
  81-112
• Kulhavy, R.W. Stock, W.A. (1989). Feedback in
  written instruction The place of response
  certitude. Educational Psychology Review, 1(4),
  279308.

20
References

• Malmi, L. Korhonen, A. (2004). Automatic
  Feedback and Resubmissions as Learning Aid,
  Proceedings of the IEEE International Conference
  on Advanced Learning Technologies, ICALT04,
  186-190
• Malmi, L., Karavirta, V., Korhonen, A.
  Nikander, J. (2005). Experiences on automatically
  assessed algorithm simulation exercises with
  different resubmission policies, ACM Journal of
  Educational Resources in Computing, 5 (3),
  http//doi.acm.org/10.1145/1163405.1163412
• Nicol, D.J. Macfarlane-Dick, D. (2006),
  Formative assessment and self-regulated learning
  A model and seven principles of good feedback
  practice. Studies in Higher Education, 31 (2),
  199-218.
• Sadler, D.R. (1989). Formative assessment and the
  design of instructional systems. Instructional
  Science. 18 (2), 119-144.
• Turkle, S. Papert, S. (1990). Epistemological
  pluralism Styles and cultures within the
  computer culture. Signs Journal of Women in
  Culture and Society, 16 (1), 128-148.