Principles of Effective Evaluation

1
Principles of Effective Evaluation

• Prof Keithia Wilson
• Academic Leader Student Success Retention

2
Importance of Evaluation

• Results in evidence-based practice
• Helps us to identify the most effective
  strategic interventions (make a difference or add
  value)
• Leads to sustainable enjoyable practice
• Provides accountability to School Group
  leadership
• Contributes to scholarship of the FYE

3
Kirkpatrick (1994) Model of Evaluation - Overview

• 4 levels of outcome evaluation
• Level 1 Reaction
• Level 2 Learning
• Level 3 Behaviour
• Level 4 Results

4
Level 1 - Reaction

• Measures participants reactions to the training
  program, including
• reactions to the overall program (outcomes) e.g.,
• To what extent did you find the Orientation
  program useful?
• reactions to specific components of the program
  (processes) e.g.,
• What aspects of Orientation did you MOST
  APPRECIATE and found USEFUL?
• What aspects of Orientation did you LEAST
  APPRECIATE and feel is MOST IN NEED OF
  IMPROVEMENT?

5
Level 1 - Reaction

• Important to include closed-ended items including
  rating scales e.g.,
• How effective was the Orientation program
  overall in helping you to make a transition to
  university?
• not at all effective
  very effective
• 1 2
  3 4 5 6 7
• How useful was the initial information
  session on the predictors of success in first
  year?
• not at all useful
  very useful
• 1 2
  3 4 5 6 7
• As well as open-ended items
• What else would you have liked included in the
  Orientation program today?

6
Level 1 - Reaction

• Some FYE examples
• Starting_at_Griffith Survey questions on Orientation
  (retrospective student survey)
• School specific Orientation Program evaluation at
  the end of O-Day the Common Time (ongoing
  Orientation Program) student surveys, focus
  groups
• Peer Mentoring Program final evaluation by
  mentors mentees

7
Level 2 - Learning

• Measures what participants have learned from
  involvement in the program
• What is measured needs to relate to what was
  covered in the program viz. learning objectives
• Typically covers knowledge, skills, or attitudes
• Needs to include both rating scales open-ended
  questions
• Can include self-report tests of actual
  knowledge

8
Level 2 - Learning

• Some examples
• Student self-report
• I feel confident that I know what to
  do to be successful academically
• this semester
• not at all confident
  very confident
• 1 2 3
  4 5 6
  7
• What have you learned about the
  transitions you will need to make as
• a commencing student in order to be
  successful?
• 2. Student knowledge
• I know the name of my First Year
  Advisor yes no (circle one)
• I know the difference between a
  lecture and a tutorial (circle one)
• Strongly Disagree Disagree
  Neutral Agree Strongly Agree

9
Level 2 - Learning

• Some FYE examples
• Starting_at_Griffith Survey data evaluates learning
  outcomes in terms of knowledge attitudes (e.g.,
  knowing where to go to get help perceptions of
  good teaching)
• Outcome evaluation for Peer Mentees (What are
  your 3 most important learnings from your
  participation in the Peer Mentoring Program?)
• First-Assessment-First-Feedback amber alert
  intervention (What have you learned as a result
  of your participation?)

10
Level 3 - Behaviour

• Measures the transfer of knowledge, skills
  attitudes from the training context to in vivo or
  real-life contexts (the classroom study).
• Evaluate both before after the program if
  practical
• Use survey, focus groups, interviews with
  students, mentors, staff (convenors, sessionals)
• Starting_at_Griffith Survey (Did you attend
  Orientation? How much time are you spending on
  study?)

11
Level 3 - Behaviour

• Some FYE examples
• Starting_at_Griffith survey data
• Attendance at E-Day, O-Day Common Time/ASP
  Program
• Attendance at lectures tutes
• Assessment submission rates in courses

12
Level 4 - Results

• Measures return-on-investment, or the extent to
  which the program has produced results.
• Some FYE examples include
• Hard student outcomes such as -
• pass rates in courses for assessment items
• GPA in semester 1, 2 FY overall
• student retention into year 2
• Soft student outcomes such as
• student satisfaction

13
Evaluation

• Key question What difference do our FYOE
  interventions (e.g., Orientation, Peer Mentoring,
  Attendance, First Assessment Intervention etc.)
  make overall?
• Program/Intervention?1 Reactions?2 Learning? 3
  Behaviour? 4 Overall Productivity/Result
• Program/Intervention?Feel?Know?Do?what is the
  result or difference overall?