The RRFSS Evaluation Framework 2005 Background Information and Progress to Date

1
The RRFSS Evaluation Framework 2005 Background
Information and Progress to Date

• Presenters
• Amira Ali
• Jane Hohenadel
• Kate OConnor
• Deborah Radcliffe
• June 22, 2005

2
Group Members

• RRFSS Evaluation Group
  Amira Ali, Elaine Hector, Jane Hohenadel, Irene
  Koren, Kathy Moran, Karen Moynagh, Lynne Russell,
  Louisa Wong
• PHRED RRFSS Evaluation Group Jane
  Hohenadel, Adriana Newbury, Kate OConnor, Louise
  Picard, Deborah Radcliffe

3
Outline of Presentation

• Link to strategic planning session
• Link to previous evaluation (2002)
• Evaluation Framework RRFSS Evaluation 2005
• Tasks and Timelines

4
History

• Project was identified in March 2004
• Proposal developed in May 2004 and an Evaluation
  committee was formed with representation from
  each of the RRFSS regional working groups
• The PHREDs were identified as partners in the
  project proposal submitted to them in October
  2004
• Terms of reference for the RRFSS group and the
  PHRED group were finalized in December 2004

5
Link to Strategic Planning Session
6
Strategic Planning Session

• Strategic Planning Session in December 2004
• To develop strategic statements (vision, mission,
  values, goals, strategies) that will guide the
  collective RRFSS work over the next 3 years
• To determine the specific objectives and
  activities for the next year
• To create an action document summarizing the
  directions set

7
Strategic Planning Session

• Vision RRFSS envisions that all decisions within
  the public health system to promote and protect
  health and wellbeing and prevent adverse health
  events are informed by valid, timely and relevant
  health intelligence
• Mission Ontarios RRFSS is a flexible, timely
  and responsive surveillance system designed to
  meet local Public Health intelligence needs and
  address information, geographic and time-related
  data gaps

8
Strategic Planning Session

• Four strategies were identified
• Advocacy
• Communication
• Governance
• Quality Improvement and Development

9
Strategic Planning Session

• Strategy Quality Improvement and Development
• Priority Activity Conduct the RRFSS 2005
  Evaluation to identify areas of the surveillance
  system in need of improvement
• The full report Strategic Plan for Action can
  be viewed at www.rrfss.on.ca

10
Link to RRFSS Evaluation 2002
11
RRFSS Evaluation 2002

• Covered first year of operation (2001)
• Number of health units ranged from 6 to 12 during
  the evaluation period
• Purpose
• Inform change to system procedures
• Contribute to continuous quality improvement
• Act as a baseline and template for future
  evaluations
• The full evaluation report can be viewed at
  www.rrfss.on.ca

12
RRFSS Evaluation 2002

• Evaluation Framework covered 4 areas
• Process Effectiveness
• Collaboration Effectiveness
• Usefulness
• Cost Effectiveness

13
RRFSS Evaluation 2002

• Performance criteria linked to following outcomes
  expected from RRFSS
• Increased local awareness of surveillance
  information
• Increased involvement in generating appropriate,
  accurate information
• Increased practice of evidence-based program
  planning
• Demonstrated or potential cost benefit due to
  RRFSS

14
RRFSS Evaluation 2002 Fourteen Key Issues
Identified(grouped as below for 2005 evaluation)

• Local Resources (2)
• Centralization Issues
• Provincial Implications (3)
• Implications for ISR (4)
• Process Considerations (5)

15
Key Issues from 2002 EvaluationLocal Resources

• Adequate staff/resources for data analysis and
  making data useable
• RRFSS coordinators overwhelmed by work at the
  local level

16
Key Issues from 2002 Evaluation Centralization
Provincial Implications

• Need for stability and support for the central
  work of the RRFSS partnership
• Sustaining funding for existing and new
  participants (smaller health units in
  particular) includes funding for survey
  administration and analysis
• Provincial representation of data

17
Key Issues from 2002 Evaluation Centralization
Implications for ISR

• Increasing the sample size for a health unit
• Access to ISR, decision-making, workload and
  resources
• Improved timeliness of data from ISR
• ISR Ability to handle increase in HU
  participation

18
Key Issues from 2002 EvaluationProcess
Consideration Issues

• Increasing use of data in program planning and
  evaluation in the health units
• Limited new question/module capacity under
  current questionnaire
• Process for revising old modules and developing
  new modules
• Upgrading RRFSS website
• Sharing data among organizations

19
RRFSS Evaluation 2005
20
2005 Evaluation Framework Objectives

1. To document progress made in the key areas
   identified in the 2002 Evaluation
2. To document some of the best practices used by
   Health Units
3. To envisage what it would take to get RRFSS on
   board for all Ontario Health Units and how it
   could work as a provincial system

21
2005 Evaluation Framework Refinement of the Key
Issues

• To address the evaluation objectives, we began
  with what were identified as Key Issues in the
  2002 Evaluation
• Evaluation group members were asked to review
  each of those 14 Key issues, and
• report on progress
• whether it was still an issue now
• if so, how important it was for the 2005
  evaluation

22
2005 Evaluation Framework Refinement of the Key
Issues

• It was felt that seven issues had been largely
  addressed, and that the 2005 evaluation needed
  only to document their evolution since 2002
• The issues that we felt were addressed
• increasing the sample size for a health unit
• improved timeliness of data from ISR
• access to ISR, decision-making, workload and
  resources, and
• limited new question/module capacity

23
2005 Evaluation Framework Refinement of the Key
Issues

• The issues that we felt were addressed (contd)
• limited new questionnaire/module capacity under
  current questionnaire
• process for revising old modules and developing
  new modules
• upgrading RRFSS website

24
2005 Evaluation Framework Refinement of the Key
Issues

• The remainder were felt to still be issues for
  the RRFSS partnership and should remain in the
  2005 evaluation
• The outstanding issues were refined, defined and
  expressed as evaluation questions

25
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Local Resources
• Question
• Do HUs have adequate staff and resources for
  regional and local administrative tasks of RRFSS?
• Data Collection Tool
• Questionnaire
• Source of Data
• RRFSS reps, MOH of participating HU

26
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Local Resources
• Question
• Do HUs have adequate staff and resources locally
  for data analysis and making data useable?
• Data Collection Tool
• Questionnaire
• Source of Data
• RRFSS reps, MOH of participating HU

27
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Local Resources
• Question
• Are local RRFSS reps able to meet the local work
  demand for RRFSS module development?
• Data Collection Tool
• Questionnaire
• Source of Data
• RRFSS reps, MOH of participating HU, program and
  research staff, RRFSS coordinator, RRFSS
  Evaluation Group, ISR

28
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Local Resources
• Question
• Are local RRFSS reps able to meet the local work
  demand for RRFSS module development?
• Data Collection Tool
• Questionnaire
• Source of Data
• RRFSS reps, MOH of participating HU, program and
  research staff, RRFSS coordinator, RRFSS Analysis
  Group, ISR

29
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Local Resources
• Question
• Are local RRFSS reps able to meet the local work
  demand for RRFSS module review?
• Data Collection Tool
• Questionnaire
• Source of Data
• RRFSS reps, MOH of participating HU, program and
  research staff, RRFSS coordinator, RRFSS Analysis
  Group, ISR

30
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Centralization Issues Provincial Implications
• Question
• What resources are needed currently to support
  and to ensure the stability of the provincial and
  administrative work of the RRFSS Partnership?
• Data Collection Tool
• Questionnaire
• Source of Data
• RRFSS reps, MOH of participating HU, ISR, RRFSS
  Coordinator

31
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Centralization Issues Provincial Implications
• Question
• What funds are required to enable continued
  participation of HUs currently participating in
  RRFSS and allow non-participating HUs to join?
• Data Collection Tool
• Key Informant Interviews
• Source of Data
• RRFSS reps, MOH of all HUs, ISR, MOHLTC

32
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Centralization Issues Provincial Implications
• Question
• Is provincial representation of data a desired
  product of RRFSS?
• Data Collection Tool
• Key Informant Interviews
• Source of Data
• RRFSS reps, MOH of all HUs, MOHLTC, PHRED, Chief
  MOH, external partners

33
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Centralization Issues Provincial Implications
• Question
• What would it take to extend RRFSS to all
  Ontario HUs?
• Data Collection Tool
• Key Informant Interviews
• Source of Data
• RRFSS reps, epidemiologists in non-participating
  HUs, MOHLTC, ISR, Chief MOH, MOH of all HUs

34
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Process Considerations
• Question
• To what extent are RRFSS data being used in PPE
  in the health units?
• Data Collection Tool
• Questionnaire, Case Examples
• Source of Data
• RRFSS reps, program and research staff

35
2005 Evaluation Framework Evaluation Questions
for Outstanding Issues

• Issue
• Process Considerations
• Question
• To what extent are provincial RRFSS data being
  shared among agencies and organizations?
• Data Collection Tool
• Questionnaire, Key Informant Interview
• Source of Data
• RRFSS Coordinator, ISR, RRFSS reps

36
Evaluation Framework for RRFSSEvaluation
Questions for Outstanding Issues

• Issue
• Process Considerations
• Question
• To what extent are health unit specific RRFSS
  data being shared among agencies and
  organizations?
• Data Collection Tool
• Questionnaire
• Source of Data
• RRFSS reps

37
2005 Evaluation Framework Evaluation Question
for Objective 2

• Issue
• Process Considerations
• Question
• What Success Stories can be identified for
  RRFSS?
• Data Collection Tool
• Workshop, Questionnaire
• Source of Data
• RRFSS reps, program and research staff

38
2005 Evaluation Framework Evaluation Question

• Issue
• Process Considerations (Parked Issue)
• Question
• What is the quality of RRFSS data?
• Data Collection Tool
• Questionnaire
• Source of Data
• RRFSS reps, ISR, partners

39
Tasks and Timelines
40
RRFSS Evaluation Tasks

• Finalize development of research tools.
• Questionnaire
• Focus groups
• Key informant interviews
• Document reviews
• Data collection
• RRFSS Workshop-- Success stories
• Primarily August and September
• Data Analysis
• Draft and finalize the report

41
RRFSS Evaluation Timelines
42
(No Transcript)