End of module evaluation: Meaningful evaluation of student learning

1
End of module evaluationMeaningful evaluation
of student learning?

• Dr Derek M Peters
• School of Sport Exercise Science

2
Session outline

• Disseminate the findings of a pilot end of
  module evaluation research case study in the
  School of Sport Exercise Science
• Explore participants current implementation of
  the Student Feedback Cycle at the modular level
• Facilitate discussion surrounding the purpose of
  modular evaluation,
• Attempt to identify ways in which effectiveness
  can be ascertained when using modular evaluation
  as a means of improving learning and teaching.

3
pilot end of module evaluation research case
study

• Illogical research progression or seizing the
  moment?
• Started with concern regarding reliability
• i.e. if students completed the SSES evaluation
  form twice, would we get the same results?
• Spontaneous data collection opportunity
• Voluntary and anonymous (student ID)
• Able to link evaluation scores to module grade!

4
Results of data analysis

• 72 gave ID (n69) 28 didnt (n27)

5
Volunteers Vs non-volunteers

• Mann Whitney U (plt0.05, n69)
• ID given sig. greater than no ID for
• How appropriate and clear were the Assessments
  and Feedback on your work
• How would rate the module in terms of Developing
  your Knowledge Understanding and Skills within
  the module
• Taking all of the above factors into account,
  your rating of this module
• But medians mode 4, 4, 4, Vs 4, 4, 4.

6
Test-retest reliability

• Wilcoxon Signed Ranks (Plt0.05, n69)
• No significant GROUP differences..
• BUT percentage level of agreement?
• i.e. did INDIVIDUALS put the same answer twice?

7
Agreement analysis
37 (n22)
15 (n9)
Kappa ranged from 0.46-0.74
8
So.....

• Group level, reliable
• Individual level, less so
• and surely, some part of the evaluation must
  relate to grade as the measurable outcome of
  learning?

9
Evaluation Vs grade

• No significant relationships (plt0.05, Spearman
  Rho).
• No differences for any question between grade
  groups (plt0.05, Kruskal-Wallis).
• Limited apparent differences in nature of
  qualitative feedback between grade groups.
• .So what are we actually evaluating?
• Became a question of validity rather than
  reliability as it had started out!

10
Stage 2 - background research (oops!)

• John Brennan and Ruth Williams (2004). Collecting
  and Using Student Feedback. A Guide to Good
  Practice. http//www.heacademy.ac.uk/955.htm
• The mechanisms for collecting, analysing and
  using student feedback vary, which raises
  questions about how effective these practices are
  in improving learning and teaching.

11
Student Feedback Cycle
12
Task

• Individually
• What tools do you use to evaluate your modules
  and what do you evaluate? (3 mins)
• Consider how your practice and the practice of
  your department aligns with this cycle at the
  modular level and identify gaps. (3 mins)
• In small mixed department groups
• Discuss your practice and on the flip chart,
  identify your 3 key hints, tips, practices,
  methods, means of maximising the cycles
  effectiveness at the modular level. (10 mins)
• Feedback to the group. (2mins each group max)

13
Quick run-through for SSES starting where we all
start!!

• Mid and end of module questionnaire (quantitative
  and qualitative).Validity of questions asked?
• Analysis.Means and SD? Staffs own summary of
  qualitative comments?
• Summary checklist to DQAC (then where/what?)
• Paragraph added to module outline to show changes
  made based on last years feedback (importance
  raised).
• Process starts again
• but limited reconsideration of either purpose
  or evaluation of the effectiveness of the process
  to improve student learning?

14
Discussion questions

• So, what is the purpose of module evaluation?
• Roughly, what proportion of your end of module
  evaluation
• Is about your teaching e.g..?
• Is about resources and facilities e.g..?
• Involves students other than them completing
  the tool used?

15
What proportion of your module evaluation

• Is about meaningful evaluation of the student
  learning that has taken place e.g.?
• How can this be assessed?
• How can we determine if the evaluation process
  improves this?
• How can we involve students in the entire
  evaluation process?
• Pedagogic research for students by students?

16
Further research?

• Anyone fancy participating in an
  inter-departmental student led pedagogic research
  project to
• Embed the student voice in the entire modular
  evaluation process
• Empower students to design, complete and
  disseminate the findings from a rigorous LT
  project researching the effectiveness of module
  evaluation for enhancing student learning
• and in doing so
• Enhance future student engagement with, and the
  effectiveness of, module evaluation practice?