Making an Impact

1
Making an Impact A Preferred Framework and
Indicators to Measure Returns on Investment in
Health Research Full Report available at
http//www.cahs-acss.ca/e/assessments/completedpro
jects.php
Canadian Academy of Health Sciences Académie
canadienne des sciences de la santé
scientific advice for a healthy Canada
2
Table of Contents
2
3
Background
scientific advice for a healthy Canada
3
4
CAHS Who We Are

• Created in 2004
• Non-profit charitable organization
• One of three founding member academies
• Unique collaboration of 6 health disciplines and
  the full spectrum of academic health sciences
• Elected Board (13) plus gt260 Fellows

4
5
What is CAHS?

• A new collaborative body
• Multidisciplinary, accomplished health scientists
• Not an advocacy group
• Scientific advice for a healthy Canada
• Publications including this report on CAHS web
  site http//www.cahs-acss.ca

5
6
Scientific Advice for a Healthy Canada

• Tackle urgent and complex problems
• Use knowledge to address apprehension
• Congregate the best minds
• Assemble the best science
• Listen, deliberate, debate
• Provide the best advice
• Unbiased
• Non-vested
• Balanced
• Feasible

6
7
Evolution of Return on Investment Assessment
Meeting
Sponsors input
Invite panelists, refine question , comprehensive
literature and expertise searches, panel
meetings, draft assessment and related
information, interview experts, identify areas
where information is lacking, commission papers
to fill gaps, obtain consensus on
recommendations, address issues raised by
external reviewers, help disseminate report
Recommend Recruit External Reviewers
Refinement of prospectus, approval of question,
define expertise appoint panelists
7 commissioned papers
Reviewed recommended approval
Initial literature and expertise searches,
drafting prospectus, fundraising, communication
to sponsors, forum planning, financial
management Select appoint chair
Progress report sponsors
Approval, publication preparation,
translation, meeting planning, dissemination
7
8
ROI Sponsors

• Sponsors (continued)
• Manitoba Health Research Council (MHRC)
• Michael Smith Foundation for Health Research
  (MSFHR)
• National Cancer Institute of Canada (NCIC)
• Nova Scotia Health Research Foundation (NSHRF)
• Ontario Neurotrauma Foundation (ONF)
• Saskatchewan Health Research Foundation (SHRF)
• Western Economic Diversification Canada (WD)
• Contributors
• Canada Foundation for Innovation (CFI)
• Canadian Association of Schools of Nursing (CASN)
• Canadian Medical Association (CMA)
• Canadian Nurses Association (CNA)
• Canadian Nurses Foundation (CNF)
• Newfoundland Labrador Centre for Applied Health
  Research (NLCAHR)
• Research Canada

• Major Sponsors
• Canadian Health Services Research Foundation
  (CHSRF)
• Canadian Institutes of Health Research (CIHR)
• Canadas Research-Based Pharmaceutical Companies
  (RxD)
• Public Health Agency of Canada (PHAC)
• Sponsors
• Alberta Heritage Foundation for Medical Research
  (AHFMR)
• Association of Canadian Academic Healthcare
  Organizations (ACAHO)
• Association of Faculties of Medicine of Canada
  (AFMC)
• BIOTECanada
• Canadian Agency for Drugs and Technologies in
  Health (CADTH)
• Fonds de la recherche en santé du Québec (FRSQ)
• Government of Ontario, Ministry of Research and
  Innovation Ministry of Health and Long-Term Care
• Heart Stroke Foundation of Canada (HSFC)

8
9
Why ROI in Health Research? (1)

• Lack of public understanding of the value of
  research applicability to current issues in
  health
• Concern about accessible, affordable, high
  quality health care in a publicly funded system
• Need to adequately measure meaningfully convey
  benefits of health research to policy-makers
  public
• Increasingly common view that health care /
  health research) is a cost-driver consuming an
  ever greater share of resources at expense of
  other sectors
• Concern about expenditure accountability in both
  the public and private sectors in Canada and
  abroad

9
10

Why ROI in Health Research? (2)

• Lack of consensus on how and when to best
  evaluate return on research expenditures
• Questions from policy makers about tangible
  results attributable to recent increases in
  public investment in health research e.g. CIHR,
  CFI, CRC programs
• Uncertainty about appropriateness of Canadas
  health research expenditures versus those of
  analogous contributions in other industrialized
  countries
• Need to acquire appropriate evidence to strike
  right funding balance between investigator-initiat
  ed discovery targeted strategic health
  research

10
11
Mobilizing Science and Technology to Canadas
Advantage 2007
The Science and Technology Framework
Vision We will build a sustainable national
competitive advantage based on science and
technology and the skilled workers whose
aspirations, ambitions, and talents bring
innovations to life. To achieve this vision, we
will create three ST Advantages for Canada
Entrepreneurial Advantage Canada must translate
knowledge into practical applications to improve
our wealth, wellness, and well-being.
Knowledge Advantage Canada must build upon our
research and engineering strengths, generate new
ideas and innovations, and achieve excellence by
global standards.
People Advantage Canada must grow its base of
knowledge workers by developing, attracting, and
retaining the highly skilled people we need to
thrive in the modern global economy.
12
Mobilizing Science and Technology to Canadas
Advantage 2007
Policy Commitments
Canadas federal government will increase its
accountability to Canadians by
Improving its ability to measure and report on
the impact of ST expenditures. The government
will improve its understanding of Canadian ST
developments and the impact of federally
performed ST, and will work with the OCED and
other countries to develop metrics that will
enable comparisons against international
benchmarks of success.
12
13
CAHS Standing Committee on Assessments

• Andreas Laupacis, MD (Chair), Executive Director,
  Li Ka Shing Knowledge Institute of St.
  Michael'sHospital Professor, Faculty of
  Medicine, University of Toronto
• John A. Cairns, MD, Professor of Medicine and
  Dean Emeritus, UBC
• Timothy Caulfield, LLM, CRC in Health Law and
  Policy Professor, Faculty of Law and School of
  Public Health Senior Health Scholar AHFMR and
  Research Director, Health Law Institute,
  University of Alberta
• André-Pierre Contandriopoulos, PhD, Professeur
  Titulaire, Département d'Administration de la
  santé, Université de Montréal
• Alastair Cribb, DVM, PhD, Dean, Faculty of
  Veterinary Medicine, U of Calgary
• Jean Gray, CM,LLD, DSc, Professor Emeritus,
  Dalhousie University
• Pavel Hamet, MD, PhD, CRC, Predictive Genomics
  Chief, Gene Medicine Services, Centre de
  recherche Centre hospitalier de lUniversité de
  Montreal
• Dorothy Pringle, OC, RN, PhD, Professor
  Emeritus, Faculty of Nursing, U of Toronto
• Matthew Spence, OC, MD, PhD, Retired President
  and CEO, AHFMR
• Peter S. L. Tugwell, MD, CRC in Health Equity
  Director, Centre for Global Health (Institute of
  Population Health) Professor of Medicine
  Epidemiology, U Ottawa
• Sharon L. Wood Dauphinee, PhD, PT, Professor,
  McGill University, Montreal, QC

13
14
Steps in a CAHS Assessment

• Choice of topic
• Choice of chair
• Choice of assessment panel
• Panels independent work
• Review of draft report by external reviewers and
  CAHS Assessment Committee
• Revision of report
• Approval by CAHS Board

14
15
External Reviewers

• Joseph B. Martin. MD, PhD, Edward R. and Anne G.
  Lefler Professor of Neurobiology and former Dean
  of the Harvard Faculty of Medicine, Harvard
  Medical School, Boston, MA, USA
• The Honourable Michael J. L. Kirby, MA, PhD,
  LLD(Hon), Chair, Mental Health Commission of
  Canada, Senator (former), Ottawa, ON, Canada
• John W. Frank, MD, CCFP, MSc, FRCP(C ), Director,
  Scottish Collaboration for Public Health Research
  andPolicy, MRC Human Genetics Unit, Edinburgh,
  United Kingdom

15
16
The Report
scientific advice for a healthy Canada
16
17
The Panel (1)

• Cyril Frank, MD (Chair), McCaig Professor of
  Joint Injury and Arthritis Research Professor,
  Division of Orthopaedics University of Calgary
• Renaldo Battista, MD, MPH, ScD, Professor and
  Director of the Department of Health
  Administration, Université de Montréal
• Linda Butler, Fellow and Head, Research
  Evaluation and Policy Project, Australian
  National University
• Martin Buxton, BA, Professor, Health Economics,
  Brunel University, UK
• Neena Chappell, PhD, Canada Research Chair
  Social Gerontology, Professor of Sociology and
  Centre on Aging, University of Victoria
• Sally C. Davies, Director General, Research and
  Development, Department of Health and National
  Health Service, UK
• Aled Edwards, PhD, Banbury Professor, Banting and
  Best Department of Medical Research, University
  of Toronto
• Chris Henshall, PhD, Pro-Vice-Chancellor,
  External Relations, University of York, UK

17
18
The Panel (2)

• Yann Joly, LLB, LLM, project manager, Centre de
  recherche en droit public Université de Montréal
• Gretchen Jordan, PhD, Principal Member of
  Technical Staff, Science and Technology Strategic
  Management Unit, Department of Energy,
  Washington, DC
• Terence Kealey, MB, BS, PhD, Vice Chancellor
  Clinical Bio-Chemist, University of Buckingham,
  UK author The Economic Laws of Scientific
  Research
• Michael C. Wolfson, PhD, Assistant Chief
  Statistician, Analysis and Development,
  Statistics Canada
• Steven H. Woolf, MD, MPH, Professor, Departments
  of Family Medicine, Epidemiology, and Community
  Health, Virginia Commonwealth University, US

18
19
The Staff

• Eddy Nason Health Research Evaluation Analyst
  and Writer (Toronto)
• Larissa Sommerfeld Research Assistant (Calgary)
  
• Linda Marchuk Research Associate , Admin
  Support and Finance Officer (Calgary)
• Rhonda Kennedee Meeting and Events Coordinator
  (Calgary)

19
20
The Remit

• Is there a best way (method) to evaluate the
  impacts of health research in Canada are there
  best metrics for assessing those impacts (or
  improving them)?

• Useful to a full range of funders/research types
• Compatible with what is already in place in
  Canada
• Transferrable to international comparisons
• Able to identify the full spectrum of potential
  impacts

20
21
The Approach

• Sponsor interviews by panelists to be certain of
  needs
• Literature review
• Expert interviews
• Seven commissioned papers in areas of special
  interest and/or perceived gaps
• Public perspective
• Ethics
• International frameworks
• Pillar II, Pillar III, and Pillar IV research
• Meso-level metrics for impact
• Working Groups on panel
• Face-to-face meetings x 3

21
22
Sponsor Expectations
22
23
Different Evaluation Needs
23
24
Many ComplexitiesFor example the (nearly)
infinite number of potential evaluation
questions

• Evaluation questions from one funder and one
  program only
• Have we increased the skill set of Canadian
  health research?
• Have we increased the number of skilled
  researchers working in Canada?
• Are our trainees producing high quality research?
• Are our trainees disseminating their findings to
  a variety of appropriate stakeholders?

24
25
Experts Interviewed

• Ilse Treurnicht
• Sharon Manson Singer
• Teren Clarke
• Cheryl L. Koehn
• Greg Tassey
• Muhajarine Nazeem
• Carol Dahl

• Douglas Barber
• Stefan Ellenbroek
• Greg Webster
• Wendy Baldwin
• Egon Jonsson
• Noralou P.Roos
• John Cairns

25
26
Selected Literature gt260 articles, books, web
links, etc

• Buxton MJ, Hanney SR. How can payback from health
  services research be assessed? J Health Serv Res
  Pol. 19961(1)35-43.
• US Senate. Joint Economic Committee. The benefits
  of medical research and the role of the NIH.
  Washington, D.C. 2000.
• Nason E, Janta B, Hastings G, Hanney S,
  O'Driscoll M, Wooding S. 2008. Health research
  Making an Impact. The Economic and Social
  Benefits of HRB Funded Research.DublinIreland.
• Oortwijn W, Hanney S, Ligtvoet A, Hoorens S,
  Wooding S, Grant J, et al. Assessing the impact
  of health technology assessment in the
  Netherlands. Int J Technol Assess Health Care.
  200824(3)259-69.
• Lavis J, Ross S, McLeod C, Gildiner A. Measuring
  the impact of health research. J Health Services
  Res Pol. 20038(3)165-170.
• Funding First. 2000. Exceptional Returns The
  economic value of America's investment in medical
  research. New York, New York The Lasker
  Foundation.
• Cutler DM, Kadiyala S. 1999. The Economics of
  Better Health The Case of Cardiovascular
  Disease. New York, New York The Lasker
  Foundation.
• Access Economics 2003. Exceptional Returns the
  value of investing in health RD in Australia.
  2003.
• Access Economics 2008.Exceptional returns The
  value of investing in health RD in Australia
  II.Canberra.AccessEconomics2008
• Buxton M, Hanney S, Morris S, Sundmacher L,
  Metre-Ferrandiz J, Garau M, et al. Medical
  Research Whats it worth? Estimating the
  economic benefits from medical research in the
  UK. Report to the UK evaluation forum 2008.
  London, UK.

26
27
Many Options for Methods

• Econometric approaches and models
• Performance measurement systems
• Logic models and frameworks
• Implementation evaluation models
• Balanced scorecards

27
28
Many Technical Complexities Issues in
Determining Health Research Impacts

• Attribution issues (effects of factors other than
  research) and the counterfactual (what would
  have happened without the research being done)?
• Need collaboration
• Need research on those topics
• Time lags to impact are very long
• Need indicators that can track longitudinally
• Double-counting of health research impacts
• Need to determine contributions wherever possible
• The Halo effect (only consider positive impacts
  of research)
• Need to consider negative impacts of research

28
29
Big Challenge

• To address the diverse needs of the sponsors
• Identify a method that can work for all
• To identify a method that can help resolve the
  technical challenges noted above
• To synthesize all available information on this
  topic
• To not reinvent the wheel as many other groups
  are also working in this area in the world right
  now
• Avoid simply advocating more research on the
  topic

29
30
Health Research Evaluation Frameworks An
International Comparison
RAND EUROPE

• Philipp-Bastian Brutscher, Steven Wooding,
  Jonathan Grant

30
31
Reviewed 8 evaluation frameworks
31
32
Reviewed 8 evaluation frameworks (Contd)
32
33
Payback Logic Model Flow
Adapted from Hanney S, Gonzalez-Block M, Buxton
M and Kogan M. The Utilisation of health research
in policy-making concepts, examples and methods
of assessment. Health Research Policy Systems
2003, 12
33
34
Five Categories of Health Research Impact
35
Payback Model Advantages (for Canada)

• Built on a logic model and framework
• Strong for accountability and management
• In use in Canada (CIHR and AHFMR)
• Similar to some others in use in Canada (eg FRSQ)
  and builds on elements of others (Lavis, etc)
• Can allow adequate time scale (longitudinal
  evaluation)
• Can track from individual level upward
• Can potentially track from inputs to outputs and
  onwards to outcomes
• Considers social and economic impacts

35
36
CAHS Logic Model Framework built on payback
logic model (across bottom)
Initiation and Diffusion of Health Research
Impacts
Global Research
Health Industry
Health care Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-Pillar Research

Other Industries
Interactions/Feedback
Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
Primary Outputs
36
37
Initiation and Diffusion of Health Research
Impacts
That influence decision making in
Research activity
That produces results
That affect healthcare, health risk factors, and
other health determinants
That contribute to changing health, well -being
and economic and social prosperity
Health Industry -Products/drugs -Services,
databases -Practitioners behaviour -Clinical/mana
gers guidelines -Institutional policies -Social
care practices
Knowledge Pool
Global Research

• Health care
• -Appropriateness
• -Acceptability
• -Accessibility
• -Competence
• -Continuity
• Effectiveness
• Safety
• Occur through prevention and treatment
• For disease, illness, injury, or progressive
  condition
• Prevention
• Diagnosis/prognosis
• Treatment/palliation
• Post-treatment

• Canadian Health Research
• Biomedical
• Clinical
• Health services
• Population and public health
• Cross-pillar research

• Other
• Industries
• Products/services
• Built infrastructure
• Work environment

Consultation/ Collaborations
Improvements in health and well-being (disease
prevalence and burden)
Health status and function, well-being, economic
conditions
Government (multiple levels) -Resource
allocation -Regulation -Policy -Intervention
programs -Taxes and subsidies

• Research Capacity
• Increased understanding
• Methodological advances
• Larger, more comprehensive data sets
• Human capital (absorptive capacity)
• Student and faculty career paths
• Reputation
• Research revenues
• Cross-fertilization of ideas/research
• Education curriculum

Research Decision Making -RD agendas/investment
(industry/govt/foundations) -Identify issues,
gaps -Evidence problems are being
addressed -Tackle harder problems
Economic and social prosperity

• Determinants of health
• Personal behaviour
• Social/cultural determinants
• Environmental determinants
• Living and working conditions

External Influences Interests, Traditions
Technical limitations, Political dynamics

• Public Information, Groups
• - Advocacy groups
• Media coverage
• General knowledge
• Confidence in data

Impacts feed back into inputs for future research
PAYBACK FRAMEWORK

• Topic Identification
• Selection
• Inputs
• Process

Primary Outputs
Secondary Outputs
Dissemination
Adoption
Final Outcomes
37
38
Impact Categories and Impacts
scientific advice for a healthy Canada
38
39
(No Transcript)
40
Biomedical Research
Initiation and Diffusion of Health Research
Impacts
Knowledge Pool
Health Industry
Research Results
Global Research
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Interactions/Feedback
Other Industries

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
40
41
Clinical Research
Initiation and Diffusion of Health Research
Impacts
Knowledge Pool
Global Research
Health Industry
Research Results
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Other Industries

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Interactions/Feedback
Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
41
42
Health Services Research
Initiation and Diffusion of Health Research
Impacts
Global Research
Knowledge Pool
Health Industry
Research Results
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Other Industries

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Interactions/Feedback
Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
42
43
Population and Public Health Research
Initiation and Diffusion of Health Research
Impacts
Knowledge Pool
Health Industry
Research Results
Global Research
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Interactions/Feedback
Other Industries

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
43
44
Cross Pillar Research
Initiation and Diffusion of Health Research
Impacts
Knowledge Pool
Health Industry
Research Results
Global Research
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Other Industries
Interactions/Feedback

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
44
45
Appropriate Use of the Frameworkunderstanding
the logic model and impact categories
45
46
Indicators vs Metrics

• Indicators indicate impact they do not attempt
  to quantify that impact
• Metrics are numeric indicators they allow
  putting some numbers on impact
• A combination of indicators and metrics are
  recommended

46
47
Table of CAHS Indicators and Metrics arranged
by impact category with advice about how to use
each one
REFER to MENU of INDICATORS

• Shows subcategories, indicators within each
  subcategory, indicator description, level of
  recommended application , comments and pillars

To begin a library with references on the web
47
48
Sets of indicators and metrics chosen should meet
FABRIC criteria

• Focussed on the organizations objectives that
  will use them
• Appropriate for the stakeholders who are likely
  to use the information
• Balanced to cover all significant areas of work
  performed by an organization
• Robust enough to cope with organizational changes
  (such as staff changes)
• Integrated into management processes
• Cost-effective (balancing the benefits of the
  information against collection costs)

48
49
Indicators proposed meet qualities of
attractiveness and feasibility

• Attractiveness validity, relevance, behavioural
  impact, transparency, coverage, recency,
  methodological soundness, replicability,
  comparability
• Feasibility data availability, cost of data,
  compliance costs, timeliness, attribution, avoids
  gamesmanship, interpretation, well-defined

49
50
Steps for Users of the Framework and Indicators

• Define and prioritize specific evaluation
  question(s).
• Use the framework to determine where to look for
  impacts
• Based on question(s) choose the impact categories
  (and subcategories) of interest advancing
  knowledge, capacity building, informing decision
  making, health impacts, and broad economic and
  social impacts.
• Be as specific as possible about where impacts
  are expected to occur and at what level
  (individual, group, institution, provincial,
  federal, international).
• Choose (or develop) attractive and feasible
  indicators and metrics from the appropriate
  categories of interest that will address the
  evaluation questions at the right level.
• Choose sets of indicators that are appropriate.

Avoid inappropriate uses attribution, Halo,
counterfactual, double-counting
50
51
Three Case Examples

• To demonstrate potential uses at different levels
  of complexity

scientific advice for a healthy Canada
51
52
Example 1 National funder Is our research
getting commercialized and can we improve that?

• Step 1
• Q Are our researchers commercializing their
  research effectively right now?
• Q What proportion of projects we fund lead to a
  commercialized product each year?
• Step 2
• Use the framework to determine where to look for
  impacts.

52
53
Biomedical Research
Initiation and Diffusion of Health Research
Impacts
Knowledge Pool
Health Industry
Research Results
Global Research
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Interactions/Feedback
Other Industries

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research

53
54
Example 1 National funder Is our research
getting commercialized and can we improve that?

• Step 3
• Choose Sets of Indicators for each category of
  interest.
• Informing decision making (health products
  industry vs other industries)
• patents licensed (per year)/ projects funded
  (or per program)
• funded researchers consulted by industry (per
  year)/ projects
• Economic Benefits
• Licensing returns ()
• Valuation of spin-out companies ()
• Product sales revenues ()
• Step 4
• Review evaluation results to identify potential
  enablers and barriers to commercialization and
  improve the system

54
55
Example 2Provincial Funder Are we building
Research Capacity in our province?

• Step 1
• Q1 Are we developing and retaining highly
  qualified research personnel in our province?
• Q2 Are researchers from more than one pillar of
  research being retained?
• Step 2
• Use the framework to identify where to look for
  impacts and clarify what you mean by research
  capacity
• e.g. You could use direct indicators (people/)
  and/or indirect indicators (impacts on decision
  makers)

55
56
Biomedical Research
Initiation and Diffusion of Health Research
Impacts
Knowledge Pool
Health Industry
Research Results
Global Research
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Interactions/Feedback
Other Industries

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
56
57
Health Services Research
Initiation and Diffusion of Health Research
Impacts
Global Research
Knowledge Pool
Health Industry
Research Results
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Other Industries

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Interactions/Feedback
Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
57
58
Cross Pillar Research
Initiation and Diffusion of Health Research
Impacts
Knowledge Pool
Health Industry
Research Results
Global Research
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Other Industries
Interactions/Feedback

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Health Status, Function, Well-being, Economic
Conditions
Government
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
58
59
Example 2Provincial Funder Are we building
Research Capacity in our province?

• Step 3
• Choose sets of indicators for categories of
  interest (Capacity Building)
• Infrastructure being built (/year)
• Infrastructure grant attracted (/year)
• Funding attracted (/year)
• Levels of additional funding attracted (/year)
• Personnel trained/attracted in the province
• Graduated students per year (MSc or PhD or
  MD-PhD)
• hospital staff with MSc or PhD or MD/PhD)
• provincial government staff with MSc or PhD

59
60
Example 2Provincial Funder Are we building
Research Capacity in our province?

• Step 3 (cont)
• Choose sets of indicators for categories of
  interest (Eg. Decision-Making and subcategories
  in domains of potential impact interest)
• Health related
• Use of research in provincial health care
  guidelines
• Survey of health policy makers
• Research
• Citation analysis of successful funding
  applications
• Health Products Industry
• Use of research in stage reports by provincial
  industries
• General Public
• Media citation analysis
• Step 4 Use results to improve capacity
  development

60
61
Example 3Federal Government Are we achieving
national health benefit from funding Canadian
health research?

• Step 1
• Q How much health benefit are we achieving per
  invested in (any area of) health research in
  Canada?
• e.g. Cardiovascular research
• Step 2
• Use the framework to define the potential
  outcomes and (if attribution to Canadian health
  research alone is to be determined) try to
  trace the stream that leads to Canadian health
  benefits

61
62
Cross Pillar Research
Initiation and Diffusion of Health Research
Impacts
Knowledge Pool
Health Industry
Research Results
Global Research
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Other Industries
Interactions/Feedback

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Health Status, Function, Well-being, Economic
Conditions

Government
all Cdn CV Research
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
The Gross Approach
62
63
Example 3Federal Government Are we achieving
national health benefit from funding
Canadian health research?

• Step 3 Choose sets of indicators for categories
  of interest (Economic and Social Benefits)
• Indicators of Health Status
• Mortality (potential years of life lost)
• Quality adjusted mortality (quality adjusted life
  years)
• Indicators of Determinants of Health
• Prevalence of modifiable risk factors (eg.
  hypertension)
• Indicators of Health Benefit
• QALY gain per health care dollar spent in CV
  research and/or CV research plus CV treatment
• Step 4 Improve by defining Canadian attribution

63
64
Cross Pillar Research
Initiation and Diffusion of Health Research
Impacts
Attribution
Knowledge Pool
Health Industry
Research Results
Global Research
Healthcare Appropriateness, Access, etc.
Prevention and Treatment
Improvements in Health and Well-being
Other Industries
Interactions/Feedback

• Canadian Health Research
• Biomedical
• Clinical
• Health Services
• Population and Public Health
• Cross-pillar Research

Health Status, Function, Well-being, Economic
Conditions

Government
Cdn CV Research
Economic and Social Prosperity
Research Agenda
Determinants of Health
Public Information, Groups
Research Capacity
Impacts feed back into inputs for future research
The Attribution Approach
64
65
What this framework can do

• With strategic selection of appropriate sets of
  indicators the framework can be used to trace
  impacts within any of the four pillars of health
  research or domains that cut across these
  pillars.
• It can also be used to describe impacts at
  various levels individual, institutional,
  provincial, national, or international and define
  the returns on investment of funders by
  (eventually) quantifying the values of impacts as
  a function of dollars put in (ROI).

65
66
What this framework can do

• The combination of breadth, depth, and
  flexibility suggests that the new framework
  fulfils our sponsor-requested criteria to be
  useful to a full range of funders/research types,
  compatible with what is already in place in
  Canada, transferable to international
  comparisons, and able to identify the full
  spectrum of potential impacts.

66
67
What this framework cannot do

• The framework can help guide evaluations but it
  does NOT provide the questions (or refine them).
• The framework cannot resolve all complexities
  easily or immediately
• If applied prospectively, it will take time and
  depending on the scope, it could become costly to
  apply it
• It is only as good as the selection of domains,
  indicator sets and data within indicators
• to progressively resolve attribution, to achieve
  a balanced perspective, to avoid the Halo effect
• The menu of 66 indicators provided is only a
  start more are required.
• Balanced perspectives may not provide black and
  white answers

67
68
Recommendations
scientific advice for a healthy Canada
68
69
Recommendation 1

• The framework and indicators identified by this
  assessment should be used by all funders of
  health research in Canada for evaluation of their
  health research impacts.

69
70
Recommendation 2

• Sets of indicators and metrics chosen from our
  menu should be used by all funders of health
  research in Canada for evaluation of their health
  research impacts.

70
71
Recommendation 3

• Canadian health research funders should begin
  collaborations immediately to advance the
  practical (methodological) prerequisites for
  measuring returns on investment in Canada.
• To standardize and refine methods, and to
  routinely collect high quality and appropriate
  data (eg. common CV end-of-grant reports, etc).
• A library of impact indicators and metrics should
  be created, beginning with the starting menu
  developed here.
• A core set of key health research impact
  questions based on what is practical and
  feasible should be developed.
• Strategic and ethically sound selection of
  indicator sets is required in order to avoid
  biasing future health research
• Evaluation questions and the choice of indicators
  and metrics to be used reflect political and
  social choices the motives underpinning such
  choices should be made transparent.

71
72
Recommendation 4

• Canada should immediately initiate a national
  collaborative effort to begin to measure the
  impacts of Canadian health research
• Leaders from national organizations, industry
  government should organize a comprehensive
  evaluation effort that engages research
  communities, other stakeholders and members of
  the public
• One option funders, led by one national
  organization, form a national council to lead
  planning and execution with a formal secretariat
  commissioned data collectors to begin this work

72
73
Recommendation 5

• Canadian health research funders should
  collaborate internationally to advance the basic
  science of health research impacts.
• an international funding stream Collaborative
  international funding would be beneficial in
  advancing the elements identified in
  Recommendation 3.
• a research plan to identify contribution
  indicators and close attribution gaps The
  distal indicators of the impacts of health
  research are very broad, and many factors other
  than funded health research can and do affect
  these outcomes. It is fundamental that
  methodologies be developed to separate the
  contribution of health research from other causal
  factors.
• a research plan to use the recommended framework
  for learning (impact improvement) purposes

73
74
How recommendations address 6 evaluation issues

• Attribution (things other than health research
  impacting health) solution R5 research on
  attribution
• Counterfactual (what would have happened
  without the research) solution R5 controls are
  required
• Using isolated indicators (distorts impressions)
  solution R2 indicator sets must be used
• Costs of Evaluation (hard to justify) solution
  R3-R5 collaboration nationally and
  internationally
• Double-Counting (two disciplines taking credit
  for same advance) solution R5 contributions
  defined
• Halo Effect (seeking only positive impacts)
  solution R5 consider negative impacts within
  indicator sets

74
75
Many Lessons from Process

• There are a number of ROI in health research
  projects in progress internationally right now
• we had many of the global experts involved in our
  process established excellent relationships
• There are common international needs
  opportunities
• many potential synergies (future potential)
• accurate content and a consistent approach are
  required to make informed decisions
• The principles of measuring ROI go far beyond
  health research
• Involving others (e.g. Energy, etc) can add
  value

75
76
Making an Impact A Preferred Framework and
Indicators to Measure Returns on Investment in
Health Research Full Report available at
http//www.cahs-acss.ca/e/assessments/completedpro
jects.php
Canadian Academy of Health Sciences Académie
canadienne des sciences de la santé
scientific advice for a healthy Canada