Good Practice Evaluation

1
Good Practice Evaluation
MPHP State Conference October 16th 2003

• Helen Jordan Theonie Tacticos
• h.jordan_at_unimelb.edu.au t.tacticos_at_unimelb.edu.a
  u
• Program Evaluation Unit (Formerly CHPE)
• Department of Public Health
• The University of Melbourne

2
Good Practice Program

• Aims to stimulate new developments and creative
  approaches in Municipal Public Health Planning.
• Wealth of information to be captured and
  disseminated models, lessons learnt, resources
  developed
• Good Practice Evaluation aims to capture and
  disseminate this.

3
Good Practice Evaluation

• Aims to inform DHS, existing and future
  projects and Municipal Public Health Planners,
  about models and approaches that have worked,
  have not worked, why or why not, how they could
  be made to work better and applied more broadly
  across other municipalities.

4
Good Practice Evaluation

• In particular, the evaluation identifies
• the nature of the strategies both planned and
  adopted to attain specific goals,
• the degree to which the goals were attained,
• whether the strategies were successful or not
• factors that contributed to success
• barriers to success
• strategies introduced to overcome the barriers
• lessons learnt, recommendations and
• tools and resources as they are developed and
• shares this information.

5
Good Practice Evaluation

• The evaluation was also designed to
• Encourage evaluation planning from the beginning
• Demonstrate the benefit of early evaluation
  planning to project planning
• Promote sustainability as a measure of success
• Identify the potential for transferability of
  the initiatives to other municipalities. Identify
  factors that contribute to its wider application.
• Promote evaluation knowledge and skills during
  and beyond term of the project

6
Good Practice Evaluation

• The evaluation also aimed to adopt a partnership
  model to evaluation by
• Involving DHS and individual projects (users of
  evaluation) in the negotiation of the evaluation
  questions and design,
• Providing active support to DHS and individual
  projects, and
• Provide projects with evaluation information as
  it is produced.

7
Good Practice Evaluation

• The evaluation required each project to
• Gain a shared understanding of evaluation
  concepts and terms
• Be given the opportunity to provide input into
  the overall evaluation questions and design
• Develop a project specific evaluation plan
  using a template and obtain guidance and
  feedback on the plan
• Provide information to the overall evaluation
  through the use of user friendly data collection
  tools and
• Obtain information from the evaluation via timely
  written reports and opportunities for direct
  information sharing.

8
Good Practice Evaluation

• Steps
• Developed overall evaluation plan and draft
  evaluation planning tool/template with advice and
  feedback from Good Practice Steering Committee
• Good Practice Project Officers
• Regional DHS Staff
• DHS local government partnership team
• Evaluators

9
Good Practice Evaluation

• 2) Run a workshop with all Good Practice Projects
  to
• Gain a shared understanding of evaluation
  concepts and terms
• Advise on nature of Good Practice evaluation
• Get participants to consider evaluation questions
  that they would like asked of their and other
  projects
• Provide feedback on evaluation planning
  tool/template to be used by all projects and
  overall evaluation questions.

10
Good Practice Evaluation

• 3) Finalise evaluation planning tool and
  disseminate to all projects
• 4) Receive and review completed tools
• 5) Site visit each project and provide advice and
  feedback on submitted evaluation planning tool

11
Good Practice Evaluation

• 6) Compile First Evaluation Report
• Nature of project
• Objectives, strategies adopted to meet
  objectives, success criteria, data collection
  methods for these and other questions (e.g.
  barriers, success factors, other outcomes)
• Composition and roles of steering committee
• Project personnel
• Expected resources to be developed.
• Report found at Good Practice Site
  http//www.dhs.vic.gov.au/phd/localgov/goodpractic
  e.htm

12
Good Practice Evaluation

• 7) Develop second evaluation data collection tool
  with advice and feedback from steering committee
• 8) Disseminate and receive completed second
  evaluation data collection tools for each
  project

13
Good Practice Evaluation

• 9) Compile data and Second Evaluation Report
• Establishment and operation of steering
  committees
• Rural Projects issues
• Joint Projects issues
• Start Up Issues
• Implementation issues successful strategies,
  unsuccessful strategies.
• Critical Success Factors
• Lessons Learnt/Recommendations for others
• Resources developed and currently available
• Issues of sustainability activities that will
  continue, would like to see embedded, supports
  needed.
• Report found at Good Practice Site
  http//www.dhs.vic.gov.au/phd/localgov/goodpractic
  e.htm

14
Good Practice Evaluation

• 10) Provide feedback during a workshop/information
  session for all projects.
• 11) Develop a third evaluation data collection
  tool with advice and feedback from steering
  committee
• 12) Disseminate and receive completed third
  evaluation data collection tools for each project.

15
Good Practice Evaluation

• 13) Compile Third Evaluation Report (due to be
  completed in few weeks)
• Contextual information (e.g. other initiatives,
  management councillor involvement, staff
  changes)
• Resources used by each project (e.g. Environments
  for Health, Leading the Way, other GP Projects,
  DHS)
• Generic questions for all projects (e.g. meeting
  objectives, barriers, success factors, lessons
  learnt, recommendations)
• Specific project thematic questions (e.g.
  indicator development, integrated planning,
  partnership development, workforce development,
  health status profile development)
• Issues of sustainability
• Report will be posted at Good Practice Site
  http//www.dhs.vic.gov.au/phd/localgov/goodpractic
  e.htm

16
Project Themes Activities

• Integrated Planning
• general narrow focus (physical activity, water,
  youth)
• audit of planning processes or staff knowledge
• staff training workshops - lit reviews
• mapping Council activities health impact
• Indicators Health Status Profile Devt
• decisions about focus/ priorities data
  collection
• Community Engagement
• development of MPHP priorities or feedback on
  goals
• Mental Health Research
• working groups, training, focus groups review
  of services
• Workforce Development
• building rural links with universities,
  recruiting undergrads

17
Key Issues Which Emerged

• very similar issues for Councils regardless of
  focus
• reforming nature of Good Practice projects rather
  than a discrete project
• social model of health - every Council activity
  has potential health impact
• active partnerships (internally externally)
  with those not previously seen their role as
  health
• broadening understanding of health
• changing current practices - ie integrated
  planning

18
Key Issues Which Emerged

• importance of organisational context
• significant differences understanding support
  for social model of health
• training, support reform implications
• much longer start up period
• importance of pre-existing partnerships
• past tendering practices hindered cross
  departmental/organisational communication

19
Key Factors Influencing Ease Success of
Projects

• extent quality of relationships with internal
  external partners (including PCPs)
• level of senior management understanding
  commitment to social model of health
• understanding ( skill level) of external
  internal partners of the social model of health

20
Common Start Up Issues

• overlap between start up implementation
• longer lead time staff/consultants
  partnerships
• rural recruitment, diverse roles, fires
  drought
• joint projects similar partnership issues - need
  different processes outcomes to suit the
  different nature of Councils
• establishment of working groups
• less of issue where pre-existing relationships
• getting commitment from partners
• developing shared understanding of outcomes
  processes

21
Critical Success Factors

• Strong support from senior managers Councillors
• Leading the Way - where linked into specific
  program
• Other initiatives /changes in Council eg TBL
• Active working groups
• Attributes of the Team - skills, goodwill
• Resourcing - staff committee time
• Being informed of current initiatives
• not re-inventing wheel
• examples support material for wider
  communication and to maintain interest of
  committee
• networking support

22
Barriers Strategies Used Integrated Planning

• major challenges of stakeholder recruitment
  developing shared vision
• issue for both internal external focused
  projects
• lack of understanding social model of health
• MPHP responsibility of health branch
• Strategies used
• briefing reports written using language
  understood by non-health professionals
• narrowed focus of project
• mapping highlighting how health/MPHP/
  Environments for Health fits into core Council
  functions
• formal informal communication
• sought opportunities for cross-departmental
  projects

23
Barriers Strategies Used

• Staff changes (ctee project) - re-engagement
  of new people in organisation reformulating
  project
• Steering Committees - time, changing
  representation commitment, competition
• key agencies represented promote project
• amend project to interests of key members
  partners
• ongoing promotion communication
• indiviual meetings with new members
• Resourcing - time, project activity funding

24
Barriers Strategies Used

• Organisational issues culture
• identify best fit for introduction
  implemntation
• link into existing planning processes
• higher the level of commitment the better
• devote time energy to build public health
  awareness
• Indicator Health Status Profile Development
• timeframes for data inconsistent - decide on cut
  off date
• postcodes cover more than one municipality - make
  assumptions
• delay in release of 2001 census data
• accessing data - next projects will benefit

25
Barriers Strategies Used

• Community Engagement
• identifying engaging community leaders was
  challenge
• requires ongoing communication not just once
  off
• involve people in initial processes
• Mental Health Research
• engaging with target group - used services who
  deal with group to promote recruit
• commitment of key groups - changing focus,
  targetting individuals within organisations,
  communication

26
Sustainability

• Integrated planning to become normal practice
• higher profile commitment to MPHP
• stronger cross Council reporting
• greater understanding of social model of health
  relevance to Council activities
• greater senior management commitment
• resources skills for development use of
  meaningful indicators
• ongoing commitment to community participation
• resources for community participation
• common commitment practices across Council

27
Sustainability

• Some Councils more successful than others in
  integrating project achievements and processes
• staff turnover
• Capacity Building Framework (NSW Health 2000)
  provides insight
• organisational development changes in policies
  practice
• workforce development staff training (management
  Councillor)
• resource allocation Council resources
• partnership internal external, collaboration
  communication
• leadership at a range of levels, but
  particularly at senior levels