Evaluating your EQUIP Initiative

1
Evaluating your EQUIP Initiative

• Helen King

2
Objectives

• To enable teams to develop a shared understanding
  of the purpose, use and stakeholders for
  evaluation
• To provide teams with a range of examples of
  approaches to evaluation
• To develop a platform on which teams might build
  their evaluation strategy.

3
Developing a Shared Understanding

• 2 minutes
• Individually, write on post-its (one idea per
  post-it)
• What is evaluation (for)?
• Who is evaluation for?

4
What is evaluation?

• Evaluation is the purposeful gathering, analysis
  and
• discussion of evidence from relevant sources
  about the
• quality, worth and impact of provision,
  development or
• policy. (Murray Saunders)

5
What is evaluation for?

• Evaluation for accountability eg measuring
  results or efficiency identifying links between
  initiative objectives and behaviour change or
  development
• Evaluation for development eg providing
  evaluative help to strengthen institutions or
  projects
• Evaluation for knowledge eg obtaining a deeper
  understanding in some specific area or policy
  field

6
When to evaluate

• Formative and summative
• When the cook tastes the soup, it is formative
  evaluation when the dinner guest tastes the
  soup, it is summative evaluation..
• Formative evaluation for dissemination.

7
Who is evaluation for? Stakeholders

• Students
• Academic staff
• Senior Management
• Your EQUIP team
• Others...

8
Developing a Shared Understanding

• What is evaluation (for)?
• Who is evaluation for?
• 10 minutes as a team, sort the ideas, prioritise
  and identify what is most important / meaningful
  to your team

9
Evaluation can help to...

• Make sense of what is happening
• Bridge projects to their wider contexts
• Identify outcomes and impact
• Identify by-products and serendipitous outcomes

10
Approaches to Evaluation

• What is it that you wish to evaluate?
• Your outcomes
• Product
• Process (internal and external to the EQUIP team)
• What data do you need to collect? And from whom?
• Stakeholders
• Evaluation methods
• Quantitative
• Qualitative

11
Prioritising / Focusing your Approach to
Evaluation

• Good Evaluation Method (GEM Grant 2010)
• Modest knows what its trying to do and is
  focused
• Ethical attends to issues of consent, anonymity
  and power
• Economical right size for purpose, does not
  collect data that wont be used
• Uses of evaluation will it
• Inform academic practice?
• Feed in to policy?
• Feed back in to your EQUIP initiative?
• Be filed away / not used?
• Other?

12
Models of Evaluation

• RUFDATA
• Appreciative Inquiry
• Academy Framework

13
RUFDATA

• RUFDATA is an acronym for the procedural
  decisions, which would shape evaluation activity
  in a dispersed office. They stand for the
  following elements.
• Reasons and purposes
• Uses
• Focus
• Data and evidence
• Audience
• Timing
• Agency

14

• What are our Reasons and Purposes for evaluation?
• These could be planning, managing, learning,
  developing, accountability
• What will be our Uses of our evaluation?
• They might be providing and learning from
  embodiments of good practice, staff development,
  strategic planning, PR, provision of data for
  management control
• What will be the Foci for our evaluations?
• These include the range of activities, aspects,
  emphasis to be evaluated, they should connect to
  the priority areas for evaluation

15

• What will be our Data and Evidence for our
  evaluations?
• numerical, qualitative, observational, case
  accounts
• Who will be the Audience for our evaluations?
• Community of practice, commissioners, yourselves
• What will be the Timing for our evaluations?
• When should evaluation take place, coincidence
  with decision-making cycles, life cycle of
  projects
• Who should be the Agency conducting the
  evaluations?
• Yourselves, external evaluators, combination

16
Appreciative Inquiry

• The power of the unconditional positive question
• James Ludema et al in
• Handbook of Action Research Peter Reason
  Hilary Bradbury, SAGE, 2001

17
Evaluation and Impact The Academy Framework
Level Label Intended outcomes at this level people will Link to the next level
1 Awareness of Recognise the Academy as enhancing learning and teaching in HE through its activities and products, and thereby... ...make an informed choice to learn more about or engage in some way with the Academy
2 Reactions Feel positive about the Academy as a whole and/or particular activities and products, and thereby... ... make a further informed choice to engage in some particular way with the Academy
3 Engagement Engage with enhancing learning and teaching through the work of the Academy e.g. sign up for mailings, attend an event, read a publication and thereby... ...begin to learn and develop some ideas relevant to their practice
4 Learning from Learn or develop ideas relevant to their work, and thereby... ...begin to plan how to use these ideas in their practice
5 Applying the learning Apply what they have learned or developed to their practice, and thereby... ...begin to identify the effects of applying these new ideas in practice
6 Effects on student learning Identify instances of students learning better as a result of enhanced practice, and thereby... ... develop evidence-based confidence in these new methods and more broadly in sound pedagogic innovation
18
and so

• What impact will the evaluation have? (How) will
  it be used?
• Formative and summative
• Involving / informing key stakeholders (including
  yourselves)
• Capturing those unintended consequences
• University of Sheffield, Evaluation Resources
• http//www.sheffield.ac.uk/lets-evaluate/general/m
  ethods-collection/questions.html

19
This session helped our team to develop a shared
understanding of the purpose, use and
stakeholders for evaluation

• Agree
• Disagree
• Neutral