YMCA National Sport, Health, Exercise

1
YMCA National Sport, Health, Exercise Fitness
Conference13th May 2006

• Creating an evidence base for physical activity
  through effective monitoring evaluation

2
Who we are

• Social and economic regeneration consultants
• Over 18 years experience
• 20 consultants 3 offices
• Work across public, private and voluntary sector
• Specialist expertise
• Monitoring Evaluation
• Strategic planning
• Feasibility studies
• Research Analysis
• Work across health, area based regeneration,
  economic development, neighbourhood renewal,
  sport leisure

3
Who we are..

• Adam Clarke
• Specialist Sport Leisure Management Consultant
• 15 years experience across public, private and
  voluntary sector
• Leading on the evaluation of the Community Sport
  Initiative in Northern Ireland, Scotland Wales
• Supporting the Community Club Development
  Programme to develop monitoring practices for
  sports clubs
• Supporting projects funded by the Active England
  programme
• Involved in Best Value pilot

4
Who we are

• Michael Dall
• Specialist in monitoring and evaluation
• North West contact for projects funded through
  the Active England programme
• Value Mapping Trainer
• Expertise in soft outcome measurement

5
Purpose of workshop

• Outcomes
• Understand what is meant by the terms evidence,
  monitoring and evaluation
• Explain why monitoring and evaluation is
  important within the sport and leisure sector for
  building an evidence base
• Receive practical guidance and tips on how to
  develop an evidence base for your own activities
• Dispel any fears and misconceptions about
  monitoring and evaluation
• Fun learning??!

6
Format of workshop

• Presentation
• Definitions
• Basic Principles
• Guidance
• Activity (not physical!)
• Case studies
• Report back
• Further sources of support
• No role play!

7
Issues and challenges in sport and leisurewhat
evidence do we have?

• Is there any?
• Prescribed monitoring Audit Commission / Best
  Value
• Visits per 1,000 population
• Spend per head of population
• Satisfaction with service
• Benchmarking?
• Measuring the unmeasurablesoft outcomes
• Anecdotal evidence / unsubstantiated claims
• Measuring the right thing? Linked to service
  objectives?
• Outcomes?

8
Creating an evidence base

• Creating an evidence base
• What types of evidence
• Primary and secondary
• How we are going to collect and use the
  evidence
• monitoring evaluation

9
Primary Evidence Direct sources
beneficiaries
10
Primary Evidence Indirect sources
staff and others
11
Secondary data
and
12
What do we mean by monitoring evaluation?

• Monitor systematically collect progress
  information during implementation / delivery
• Evaluate Assess the impact of a project,
  programme or strategy
• Indicator / Performance Indicator A measure or
  partial measure of an input, activity, output or
  outcome

13
What do we mean by monitoring evaluation?

• Input resource used on a project, service or
  programme
• Activity action taken to deliver
  infrastructure, goods and services
• Outputs the results of activities that can be
  clearly stated or measured which relate to the
  outcomes
• Outcomes the eventual benefits to society that
  projects programmes are intended to achieve.
  (Intermediate and Long term)
• Targets specific levels of indicators achieved
  by a specific time.

14
Example
15
Who is Monitoring Evaluation for?
You !!!

• If YOU cant use the information, then what is
  the point?

16
Benefits of collecting and using evidence

• To demonstrate you are doing what you claim to
  be doing
• To demonstrate what you do works
• Improve your service performance and planning
• To show you should get future funding
• Because your job depends on it!
• To gather data to report to client or target
  group
• Provide aftercare to beneficiaries
• Keep focused on your purpose
• Use in marketingblowing your own trumpet
• Influence wider strategy and policy

17
Benefits for YMCA

• YMCA Activate England project collecting
  evidence
• Use of guidelines / tools
• More in next presentation

18
How to develop an evidence base in 5 (simple)
steps

• Step 1 Choose your indicators
• Step 2 Draw up a monitoring framework
• Step 3 Collect you evidence data
• Step 4 Use your evidence
• Step 5 Evaluate

19
Step 1 Choose your indicators

• 1-2 input indicators
• 2-4 activity indicators
• 1-4 output indicators
• 1-3 outcome indicators
• BUT at least one to show link between the cause
  and effect

20
Step 1 - Check your indicators

• How will I measure this?
• How often will I measure this?
• How accurate will this information be?
• How much time money will it cost?

• Specific
• Measurable
• Achievable
• Relevant (to what you are doing)
• Time-bound

21
Step 2 Draw up a framework
Funding for project
Financial records
6 monthly
Manager
No. of dance sessions
Programme
Monthly
Manager
No. of Members
Membership form
Manager
6 monthly
No. of women physically active
Survey
Annually
Manager
22
Step 2 Draw up a framework
23
Step 3 Collect the data

• Various methods
• Quantitative and qualitative data
• Traditional methods
• Innovative methods
• All methods have particular strengths and
  weaknesses, advantages and disadvantages
• Butthere is something for everyone!
• Child care setting
• Exercise referral schemes
• Youth Work

24
Step 3 Collect the data

• Some potential methods
• Attendance sheets
• Membership registration forms
• Swipe cards
• Footfall counters / sensors
• Postal surveys
• Web Surveys
• Interviews
• Photography
• Video

25
Step 3 Collect the data Sign in Sheet

• What is it?
• Every time a person attends they fill it in
• 3 or 4 basic questions
• Allows attendance to be tracked

• Advantages
• Attendance is tracked
• Low Cost
• Easy to analyse

• Disadvantages
• Labour Intensive
• Not always accurate
• Users object to filling it out
• Does anyone look at it?

26
Step 3 Collect the data Prize draw survey

• What is it?
• A simple survey form, attractively presented
• Entry into a prize draw on completion
• Publicise it and inform the winner

• Advantages
• Cost effective
• Prize boosts interest
• Higher response rate
• Easy to analyse results
• Non intrusive
• No bias

• Disadvantages
• Unable to probe responses
• Wont measure soft outcomes
• Does the intended person fill it out?
• Some people may not feel comfortable filling it
  out

27
Step 3 Collect the data - Photography

• What is it?
• You track users over time
• Take picture before
• Take picture after
• Compare the two
• Can use it with attendance records to keep a
  track informally

• Advantages
• Good visual method
• Shows tangible changes
• Good for open access projects
• Tracks soft outcomes

• Disadvantages
• People may not be comfortable being photographed
• Relatively high cost
• Can be time consuming to track
• Can be biased

28
Step 3 Collect the data Rickter Scale

• What is it?
• Innovative tool
• Understanding beneficiaries needs/ developing a
  baseline
• motivational interviewing target setting
• reviewing progress towards targets evaluation

• Advantages
• Non paper based assessment tool
• Empowers clients
• Focuses attention
• Instantly engages participants
• Overcomes communication barriers
• Measures change over time
• Allows clients to set realistic targets
• Fun to do!

• Disadvantages
• Time consuming - interviews can last an hour
• Interviewers must be accredited Rickter users
• Findings are limited without follow up
  interviews

29
Step 3 Collect the data Electronic Surveys

• What is it?
• Intelligent survey software
• A simple online tool to create electronic
  surveys
• Collects results on-line
• Performs basic analysis
• Tracks respondents

• Advantages
• Great when you need to collect significant
  amounts of data
• Good to collect information quickly or cheaply
• Least labour intensive method for the
  respondentsso
• Good response rates
• You can chase people to respond
• Requires limited data input and staff time
• Utilises technology in the spirit of the
  e-agenda

• Disadvantages
• People may treat it as junk mail and delete
• Data such as e mail addresses are not always
  readily available
• People may not have access to the web / e-mail
• Consider data protection and spam

30
Step 3 Collect the data Using Video

• Advantages
• An engaging method of research
• Seen as exciting and interesting
• Holds peoples attention
• Gets key messages across to different audiences
• People give honest accounts

• Disadvantages
• Willing participants
• Filming skills!
• Capacity needed to edit footage

Lights Camera Action
31
Step 3 Collect the data Vox Pops

• Advantages
• Its enjoyable
• Captures information well
• Gets people talking
• People are often more honest

• Disadvantages
• Time consuming to analyse in detail
• Equipment is costly
• Capacity is needed to edit footage and produce a
  video

• What is it?
• Journalistic tool
• Widespread public opinion on one or two key
  issues
• Short and snappy

32
Step 4 Use the monitoring data

• Time series
• Comparisons
• Causal links
• Anything but tables of results and targets!

33
Step 5 - Evaluation
34
Step 5 - Evaluation

• Is the programme or project working well and
  could it improve?
• Evidence for next years plans
• Is it really making any long-term difference
  would the money have been better spent elsewhere?
  
• What are the unintended results and lessons for
  the future?

35
Your Turn

• Split into groups
• Case studies
• Discuss different ways of collecting evidence
• Need to consider
• project aims and objectives
• types of activity
• target beneficiaries
• best tools to use to collect evidence within the
  budget
• Report back to the group on methods you have
  chosen

36
Good monitoring checklist

• a    Looks at only a few indicators
• a    Choose measurable indicators
• a    Needs modest time and money
• a Gathers and spreads information regularly
• a    Communicates information clearly
• a  Gets used to make decisions

37
Additional Guidance

• Active England Website aelz.org
• Big Lottery Fund bigcsi.com
• Community Club Development Programme
  ccdprog.org
• Hall Aitken hallaitken.co.uk / 0161 835 2011