Making an Impact

1
Making an Impact A Preferred Framework and
Indicators to Measure Returns on Investment in
Health Research Approaches to the Social Impact
of Health Research. The challenge of Attribution
Can both approaches be integrated? The CAHS ROI
model and Future Applications C. Frank
Asociacion de Economia de la Salud - Malaga -
June 18, 2009
scientific advice for a healthy Canada
2
Outline

• CAHS Very Brief Background
• The rationale for ROI as first assessment
• The ROI Process and Summary of the Report
• The framework and indicators/metrics
• Three examples of how to use this approach
• Five Panel Recommendations

3
CAHS Background

• Created in 2004
• Non-profit charitable organization
• One of three founding member academies
• Unique collaboration of 6 health disciplines and
  the full spectrum of academic health sciences
• Elected Board (13) plus gt260 Fellows

4
The Foundation of CAHS
Dentistry
Rehab Sciences
Nursing
Medicine
Health Scientists
Veterinary Medicine
Pharmacy
5
Scientific Advice for a Healthy Canada

• Tackle urgent and complex problems
• Use knowledge to address apprehension
• Congregate the best minds
• Assemble the best science
• Listen, deliberate, debate
• Provide the best advice
• Unbiased
• Non-vested
• Balanced
• Feasible

6
6

Why ROI in Health Research?

• Lack of consensus on how and when to best
  evaluate return on research expenditures
• Questions from policy makers about tangible
  results attributable to recent increases in
  public investment in health research e.g. CIHR,
  CFI, CRC programs
• Uncertainty about appropriateness of Canadas
  health research expenditures versus those of
  analogous contributions in other industrialized
  countries
• Need to acquire appropriate evidence to strike
  right funding balance between investigator-initiat
  ed discovery targeted strategic health
  research

10
7
Evolution of Return on Investment Assessment
Meeting
Sponsors input
Invite panelists, refine question , comprehensive
literature and expertise searches, panel
meetings, draft assessment and related
information, interview experts, identify areas
where information is lacking, commission papers
to fill gaps, obtain consensus on
recommendations, address issues raised by
external reviewers, help disseminate report
Recommend Recruit External Reviewers
Refinement of prospectus, approval of question,
define expertise appoint panelists
7 commissioned papers
Reviewed recommended approval
Initial literature and expertise searches,
drafting prospectus, fundraising, communication
to sponsors, forum planning, financial
management Select appoint chair
Progress report sponsors
Approval, publication preparation,
translation, meeting planning, dissemination
7
8
ROI Sponsors

• Sponsors (continued)
• Manitoba Health Research Council (MHRC)
• Michael Smith Foundation for Health Research
  (MSFHR)
• National Cancer Institute of Canada (NCIC)
• Nova Scotia Health Research Foundation (NSHRF)
• Ontario Neurotrauma Foundation (ONF)
• Saskatchewan Health Research Foundation (SHRF)
• Western Economic Diversification Canada (WD)
• Contributors
• Canada Foundation for Innovation (CFI)
• Canadian Association of Schools of Nursing (CASN)
• Canadian Medical Association (CMA)
• Canadian Nurses Association (CNA)
• Canadian Nurses Foundation (CNF)
• Newfoundland Labrador Centre for Applied Health
  Research (NLCAHR)
• Research Canada

• Major Sponsors
• Canadian Institutes of Health Research (CIHR)
• Canadian Health Services Research Foundation
  (CHSRF)
• Canadas Research-Based Pharmaceutical Companies
  (RxD)
• Public Health Agency of Canada (PHAC)
• Sponsors
• Alberta Heritage Foundation for Medical Research
  (AHFMR)
• Association of Canadian Academic Healthcare
  Organizations (ACAHO)
• Association of Faculties of Medicine of Canada
  (AFMC)
• BIOTECanada
• Canadian Agency for Drugs and Technologies in
  Health (CADTH)
• Fonds de la recherche en santé du Québec (FRSQ)
• Government of Ontario, Ministry of Research and
  Innovation Ministry of Health and Long-Term Care
• Heart Stroke Foundation of Canada (HSFC)

8
9
The Panel

• Cy Frank U. Calgary
• Renaldo Battista U. Montreal
• Linda Butler Australia Nat. U.
• Martin Buxton Brunel U. (UK)
• Neena Chappell U. Victoria
• Sally Davies Dept Health, UK
• Aled Edwards U. of Toronto

• Chris Henshall U. York (UK)
• Yann Joly U. Montreal
• Gretchen Jordan Sandia Labs, U.S. Department
  of Energy
• Terence Kealey U. Buckingham (UK)
• Michael Wolfson Statistics Canada
• Steven Woolf, Virginia Commonwealth U.

10
Our Approach

• Sponsor Interviews
• Literature review
• Expert interviews
• Seven commissioned papers in areas of special
  interest and/or perceived gaps
• Public perspective
• Ethics
• International frameworks
• Pillar II, Pillar III, and Pillar IV research
• Meso-level metrics for impact
• Working Groups on panel
• Face-to-face meetings x 3

11
Sponsor Expectations
12
Different Evaluation Needs
13
LITERATURE REVIEW gt260 articles, books, web
links, etc

• Buxton MJ, Hanney SR. How can payback from health
  services research be assessed? J Health Serv Res
  Pol. 19961(1)35-43.
• US Senate. Joint Economic Committee. The benefits
  of medical research and the role of the NIH.
  Washington, D.C. 2000.
• Nason E, Janta B, Hastings G, Hanney S,
  O'Driscoll M, Wooding S. 2008. Health research
  The Economic and Social Benefits of HRB Funded
  Research.DublinIreland.
• Oortwijn W, Hanney S, Ligtvoet A, Hoorens S,
  Wooding S, Grant J, et al. Assessing the impact
  of health technology assessment in the
  Netherlands. Int J Technol Assess Health Care.
  200824(3)259-69.
• Lavis J, Ross S, McLeod C, Gildiner A. Measuring
  the impact of health research. J Health Services
  Res Pol. 20038(3)165-170.
• Funding First. 2000. Exceptional Returns The
  economic value of America's investment in medical
  research. New York, New York The Lasker
  Foundation.
• Cutler DM, Kadiyala S. 1999. The Economics of
  Better Health The Case of Cardiovascular
  Disease. New York, New York The Lasker
  Foundation.
• Access Economics 2003. Exceptional Returns the
  value of investing in health RD in Australia.
  2003.
• Access Economics 2008.Exceptional returns The
  value of investing in health RD in Australia
  II.Canberra.AccessEconomics2008
• Buxton M, Hanney S, Morris S, Sundmacher L,
  Metre-Ferrandiz J, Garau M, et al. Medical
  Research Whats it worth? Estimating the
  economic benefits from medical research in the
  UK. Report to the UK evaluation forum 2008.
  London, UK.

14
Many Technical Complexities Issues in
Determining Health Research Impacts

• Attribution issues (effects of things other than
  research) and the counterfactual (what would
  have happened without the research being done)?
• Need collaboration
• Need research on those topics
• Time lags to impact are very long
• Need indicators that can track longitudinally
• Double-counting of health research impacts
• Need to determine contributions wherever possible
• The Halo effect (only consider positive impacts
  of research)
• Need to consider negative impacts of research

15
Health Research Evaluation Frameworks An
International Comparison
RAND EUROPE

• Philipp-Bastian Brutscher, Steven Wooding,
  Jonathan Grant

16
Reviewed 8 evaluation frameworks
17
Contd
18
Payback Logic Model Flow
Adapted from Hanney S, Gonzalez-Block M, Buxton
M and Kogan M, The Utilisation of health
research in policy-making concepts, examples
and methods of assessment. Health Research
Policy Systems 2003, 12
19
CAHS Logic Model Framework built on payback
logic model (across bottom)
20
(No Transcript)
21
(No Transcript)
22
MAPPING IMPACT CATEGORIES by RESEARCH PILLARS
23
(No Transcript)
24
(No Transcript)
25
Appropriate Use of the Frameworkunderstanding
the logic model and impact categories
26
FINDING IMPACT

• INDICATORS of IMPACT

27
Indicators vs Metrics

• Indicators indicate impact they do not attempt
  to quantify that impact
• Metrics are numeric indicators they allow
  putting some numbers on impact
• A combination of indicators and metrics are
  recommended

28
MENU of 66 INDICATORS Table of CAHS Indicators
and Metrics arranged by impact category with
advice about how to use each one

• Shows subcategories, indicators within each
  subcategory, indicator description, level of
  recommended application , comments and pillars

To begin a library with references on the web
29
Indicators proposed meet qualities of
attractiveness and feasibility

• Attractiveness validity, relevance, behavioural
  impact, transparency, coverage, recency,
  methodological soundness, replicability,
  comparability
• Feasibility data availability, cost of data,
  compliance costs, timeliness, attribution, avoids
  gamesmanship, interpretation, well-defined

30
Sets of indicators and metrics chosen should meet
FABRIC criteria

• Focussed on the organizations objectives that
  will use them
• Appropriate for the stakeholders who are likely
  to use the information
• Balanced to cover all significant areas of work
  performed by an organization
• Robust enough to cope with organizational changes
  (such as staff changes)
• Integrated into management processes
• Cost-effective (balancing the benefits of the
  information against collection costs)

31
Steps for Users of the Framework and Indicators

• Define and prioritize specific evaluation
  question(s).
• Use the framework to determine where to look for
  impacts
• Based on question(s) choose the impact categories
  (and subcategories) of interest advancing
  knowledge, capacity building, informing decision
  making, health impacts, and broad economic and
  social impacts.
• Be as specific as possible about where impacts
  are expected to occur and at what level
  (individual, group, institution, provincial,
  federal, international).
• Choose (or develop) attractive and feasible
  indicators and metrics from the appropriate
  categories of interest that will address the
  evaluation questions at the right level.
• Choose sets of indicators that are appropriate.

Avoid inappropriate uses attribution, Halo,
counterfactual, double-counting
32
THREE EXAMPLES

• To demonstrate one potential use (each) for
  national and provincial funders and for
  government
• 2 bottom up and 1 top down

33
EXAMPLE 1- bottom up National funder is our
research getting commercialized and can we
improve that?

• Step 1
• Q Are our researchers commercializing their
  research effectively right now?
• Q What proportion of projects we fund lead to a
  commercialized product each year?
• Step 2
• Use the framework to determine where to look for
  impacts.

34
(No Transcript)
35
EXAMPLE 1- bottom up National funder is our
research getting commercialized and can we
improve that?

• Step 3
• Choose Sets of Indicators for each category of
  interest.
• Informing decision making (health products
  industry vs other industries)
• patents licensed (per year)/ projects funded
  (or per program)
• funded researchers consulted by industry (per
  year)/ projects
• Economic Benefits
• Licensing returns ()
• Valuation of spin-out companies ()
• Product sales revenues ()
• Step 4
• Review evaluation results to identify potential
  enablers and barriers to commercialization and
  improve the system

36
EXAMPLE 2 bottom upProvincial Funder Are we
building Research Capacity in our province?

• Step 1
• Q1 Are we developing and retaining highly
  qualified research personnel in our province?
• Q2 Are researchers from more than one pillar of
  research being retained?
• Step 2
• Use the framework to identify where to look for
  impacts and clarify what you mean by research
  capacity
• Eg. You could use direct indicators (people/)
  and/or indirect indicators (impacts on decision
  makers)

37
(No Transcript)
38
(No Transcript)
39
All Pillars of Health Research
40
EXAMPLE 2 bottom upProvincial Funder Are we
building Research Capacity in our province?

• Step 3
• Choose sets of indicators for categories of
  interest - Eg. Direct Evidence of Capacity
  Building
• Infrastructure being built (/year)
• Infrastructure grant attracted (/year)
• Funding attracted (/year)
• Levels of additional funding attracted (/year)
• Personnel trained/attracted in the province
• Graduated students per year (MSc or PhD or
  MD-PhD)
• hospital staff with MSc or PhD or MD/PhD)
• provincial government staff with MSc or PhD

41
EXAMPLE 2 bottom upProvincial Funder Are we
building Research Capacity in our province?

• Step 3 (cont)
• Indirect Evidence of Capacity Building Impacts
  on Decision-Making
• Health related
• Use of research in provincial health care
  guidelines
• Survey of health policy makers
• Research
• Citation analysis of successful funding
  applications
• Health Products Industry
• Use of research in stage reports by provincial
  industries
• General Public
• Media citation analysis
• Step 4 Use results to improve capacity
  development

42
EXAMPLE 3 top downFederal Government Are we
achieving national health benefit from funding
Canadian health research?

• Step 1
• Q How much health benefit are we achieving per
  invested in (any area of) health research in
  Canada?
• Eg. Cardiovascular research
• Step 2
• Use the framework to define the potential
  outcomes and (if attribution to Canadian health
  research alone is to be determined) try to
  trace the stream that leads to Canadian health
  benefits

43
All Pillars of Health Research

all Cdn CV Research
The Gross Econometric Approach
44
EXAMPLE 3 top downFederal Government Are we
achieving national health benefit from funding
Canadian health research?

• Step 3 Choose sets of indicators for categories
  of interest (Economic and Social Benefits)
• Indicators of Health Status
• Mortality (potential years of life lost)
• Quality adjusted mortality (quality adjusted life
  years)
• Indicators of Determinants of Health
• Prevalence of modifiable risk factors (eg.
  hypertension)
• Indicators of Health Benefit
• QALY gain per health care dollar spent in CV
  research and/or CV research plus CV treatment
• Step 4 Improve by defining Canadian
  contributions

45
All Pillars of Health Research

Cdn CV Research
The
The Detailed Attribution Approach
46
What this framework can do

• With strategic selection of appropriate sets of
  indicators the framework can be used to trace
  impacts within any of the four pillars of health
  research or domains that cut across these
  pillars.
• It can also be used to describe impacts at
  various levels individual, institutional,
  provincial, national, or international and define
  the returns on investment of funders by
  (eventually) quantifying the values of impacts as
  a function of dollars put in (ROI).

47
What this framework can do

• The combination of breadth, depth, and
  flexibility suggests that the new framework
  fulfils our sponsor-requested criteria to be
  useful to a full range of funders/research types,
  compatible with what is already in place in
  Canada, transferable to international
  comparisons, and able to identify the full
  spectrum of potential impacts.

48
What this framework cannot do

• The framework can help guide evaluations but it
  does NOT provide the questions (or refine them).
• The framework cannot resolve all complexities
  easily or immediately
• If applied prospectively, it will take time and
  depending on the scope, it could become costly to
  apply it
• It is only as good as the selection of domains,
  indicator sets and data within indicators
• to progressively resolve attribution, to achieve
  a balanced perspective, to avoid the Halo effect
• The menu of 66 indicators provided is only a
  start more are required.
• Balanced perspectives may not provide black and
  white answers

49
Recommendation

• Canadian health research funders should begin
  collaborations immediately to advance the
  practical (methodological) prerequisites for
  measuring returns on investment in Canada.
• To standardize and refine methods, and to
  routinely collect high quality and appropriate
  data (eg. common CV end-of-grant reports, etc).
• A library of impact indicators and metrics should
  be created, beginning with the starting menu
  developed here.
• A core set of key health research impact
  questions based on what is practical and
  feasible should be developed.
• Strategic and ethically sound selection of
  indicator sets is required in order to avoid
  biasing future health research
• Evaluation questions and the choice of indicators
  and metrics to be used reflect political and
  social choices the motives underpinning such
  choices should be made transparent.

50
Recommendation

• Canada should immediately initiate a national
  collaborative effort to begin to measure the
  impacts of Canadian health research
• Leaders from national organizations, industry and
  government should organize a comprehensive
  evaluation effort that engages research
  communities, other stakeholders and members of
  the public
• One option funders, led by one national
  organization, form a national council to lead
  planning and execution with a formal secretariat
  and commissioned data collectors to begin this
  work

51
Recommendation

• Canadian health research funders should
  collaborate internationally to advance the basic
  science of health research impacts.
• an international funding stream Collaborative
  international funding would be beneficial in
  advancing the elements identified in
  Recommendation 3.
• a research plan to identify contribution
  indicators and close attribution gaps The
  distal indicators of the impacts of health
  research are very broad, and many factors other
  than funded health research can and do affect
  these outcomes. It is fundamental that
  methodologies be developed to separate the
  contribution of health research from other causal
  factors.
• a research plan to use the recommended framework
  for learning (impact improvement) purposes

52
QUESTIONS, COMMENTSandNEXT STEPS