Conducting Formative Evaluations Using a Consolidated Framework for Implementation Research

1
Conducting Formative Evaluations Using a
Consolidated Framework for Implementation
Research
Laura J. Damschroder, MS, MPH Julie Lowery, PhD

• Diabetes QUERI
• Ann Arbor Center for Practice Management
  Outcomes Research

2
Introduction

• What questions do we ask when conducting
  implementation research?
• Why didnt the intervention work everywhere?
• What can we do to ensure success?
• How can we predict success or failure?
• Etc.
• So many models, so little time
• Comprehensive framework needed with clear
  definitions of constructs

3
Purpose

• Present the Consolidated Framework for
  Implementation Research (CFIR)
• Describe the constructs
• Show how we applied CFIR in a macro formative
  evaluation of the MOVE! Program
• Question Why was there such a wide variation in
  uptake 1.5 years after dissemination?
• This presentation will combine the theoretical
  development of CFIR and application to illustrate
  constructs

4
Methods Theoretical Development

• Literature review of models applicable to
  implementation research
• Targeted and non-systematic
• 11 models were included
• Constructs were identified along with evidence of
  their role in implementation
• Theoretical and/or empirical
• Theme saturation
• When new models failed to produce new constructs,
  we stopped
• Goal build on what was already developed

5
Methods Application in a Study

• Qualitative study of barriers and facilitators of
  MOVE! Program uptake
• MOVE! Weight Management Program in the VA
• Purposive sample of 5 low high uptake sites
• Semi-structured interviews with 24 key
  stakeholders
• 83 of those contacted and invited
• Qualitative analysis
• Deductive, using CFIR
• Inductive, open to new themes
• Team-based analysis
• Strength of multiple perspectives
• Test Face Validity of CFIR

6
Consolidated Framework for Implementation
Research (CFIR)
Intervention at Time0
Intervention at Time1
Internal Context
Soft Periphery
Soft Periphery
Hard Core
Hard Core
Intervention
Intervention
External Context
7
Side Note Dependent Variable

• Implementation
• The process of putting an intervention into use
  in an organization
• The vehicle by which a new practice is
  assimilated into an organization
• Implementation Effectiveness
• Three general categories
• Widespread avoidance (non-use)
• Meager and unenthusiastic use (compliant use)
• Skilled, enthusiastic, consistent use (committed
  use)

8
Consolidated Framework for Implementation
Research (CFIR)
Intervention at Time0
Intervention at Time1
Process over Time
Internal Context
Soft Periphery
Process
Soft Periphery
Hard Core
Hard Core
Intervention
Intervention
External Context
9
Consolidated Framework for Implementation
Research (CFIR)
Intervention at Time0
Intervention at Time1
Process over Time
Internal Context
Internal Context
Process
Soft Periphery
Process
Soft Periphery
Hard Core
Hard Core
Intervention
Intervention
External Context
External Context
10
Flow

• Support for each construct in the literature
• Empirical support from MOVE! qualitative study
• Begin with Intervention
• Evidence Building
• Intervention Attributes

11
INTERVENTION Evidence BuildingLiterature
Support
12
INTERVENTION Evidence BuildingFace Validity
13
Intervention Evidence BuildingHigh Uptake Sites

• Strong positive influence of Evidence Strength
• "...providers understand that a lot of problems
  are related to obesity. Absolutely everyone
  understands.
• One VISN used pilot study results to make a
  business case to obtain 9 dedicated FTEs for the
  program at VISN sites
• Strong positive influence of Relative advantage
  
• with the help of MOVE information, MOVE
  literature, MOVE whateverit sort of boosted our
  existing program more and we were able to expand
  more

14
INTERVENTION Evidence BuildingFace Validity
15
Intervention Evidence BuildingLow Uptake Sites

• Source of the Intervention
• Wow. Well theres nothing like an unfunded
  mandate...to get...their blood boiling around
  here where workloads are so high everywhere
  else.
• The same pilot results were seen as weak evidence
  supporting MOVE!
• unfortunately, with the pilot study data not
  being very robust, we had a difficult time
  selling chief of staff and chief of medicine on
  the efficacy of the pilot study
• One low uptake site referred patients to an
  external program which they felt had greater
  relative advantage

16
INTERVENTION AttributesLiterature Support
17
INTERVENTION AttributesFace Validity
18
Intervention AttributesHigh Uptake Sites

• Adaptability Both high-performing sites
  perceived the MOVE! program as flexible and
  designed programs that fit within their
  particular context
• Trialability One site learned from the national
  pilot experience
• Complexity
• One site simplified a challenging implementation
  by taking an incremental approach
• The other site viewed MOVE! as a relatively
  simple incremental change from their current
  program

19
INTERVENTION AttributesFace Validity
20
Intervention AttributesLow Uptake Sites

• MOVE! as not sufficiently adaptable for one site
  
• all of our patients would need to be referred
  for an EKG prior to starting the program. There
  were some barriers that we didnt understand
• Implementation was particularly complex because
  of barriers throughout their organizations
• One site was particularly troubled by packaging
  of MOVE!
• the initial start up manual had very not
  positive pictures on it. Depressed looking, heavy
  sailors in stretched out white tee-shirts....
• The other site was impressed with quality of
  materials and suggested it be implemented outside
  the VA

21
EXTERNAL CONTEXTLiterature Support
22
EXTERNAL CONTEXTFace Validity
23
EXTERNAL CONTEXTHigh Uptake Sites

• Both sites put an emphasis on tailoring
  programming based on patient needs and requests
• Many anecdotal stories about patients
• one of our patients that is enrolled in
  MOVEgoes and talks to other patients in the
  waiting rooms saying what a great program it is,
  takes their names andleaves it in my boxesand
  we actually have a tremendously long waiting
  list

24
EXTERNAL CONTEXTFace Validity
25
EXTERNAL CONTEXTLow Uptake Sites

• The general atmosphere at the two low-uptake
  sites is a belief that it is a challenge to
  present MOVE as a viable alternative
• our system is geared to paying people to be
  disabled.I think the commitment of the patients
  has to be up there among the top three
  difficulties...becausewe live in a society of
  quick fix and if the medicine wont do it you
  dont expect me to starve for 10 weeks, do you?
• and speaking very frankly, when Im at the
  community program kickoff which is a very
  positive high paced environmentIm out there
  with the managing obese veterans everywhere. I
  franklyhad a hard time selling some of our
  veterans

26
INTERNAL CONTEXTLiterature Support
27
INTERNAL CONTEXTFace Validity
28
INTERNAL CONTEXTHigh Uptake Sites

• Both sites had a high degree of teamness
• Very amicable, very very good, pleasant, very
  professional. I mean there isnt a week that
  doesnt go by that you know, were not
  communicating with each other and not really,
  were having a good time too with the group
  sessionswere all there to make the patients
  really change the way theyre eating and their
  activity habits
• Meet regularly
• we do this through ourlunch time. We keep it
  very shortIts very difficult to and we have our
  other assignments and you dont have the free
  time to do itto discuss obstacles, to discuss
  problems, to discuss you know, things that need
  to be discussed for us to be able to run this
  program properly.

29
INTERNAL CONTEXTFace Validity
30
INTERNAL CONTEXTLow Uptake Sites

• Did not have regular team meetings
• Though one did have a multi-disciplinary team
  that takes turns leading the group visit
• Meet through email
• Lack of effective communication
• Patients confused about what MOVE was
• a movie?
• Dance class?
• Bariatric surgery?

31
Another Side Note

• Confusion between
• Culture
• Climate
• Other terms
• Greenhalgh, et al
• Receptive context for change
• System readiness for change
• Absorptive capacity (for new knowledge)
• PARiHS
• Readiness for change
• Klein Sorra
• Implementation Climate

32
CFIR Terms

• Culture
• Implementation Climate
• 6 constructs
• Readiness for implementation
• 4 constructs

33
INTERNAL CONTEXTImplementation
ClimateLiterature Support
34
INTERNAL CONTEXTImplementation ClimateFace
Validity
35
INTERNAL CONTEXTImplementation ClimateTension
for change

• Particularly clear for transition site
• all of the group was pretty excited you know,
  because they had sat stagnant for a yearthey had
  put upposters for the MOVE program and all
  that stuff and they didnt have anything set up
  so people were consulting to the MOVE program
  when there wasnt even a program set up
• Closely related to Relative Advantage for high
  uptake sites
• Low tension for change at low uptake sites

36
INTERNAL CONTEXTImplementation ClimateFace
Validity
37
INTERNAL CONTEXTImplementation
ClimateCompatibility

• Compatible at high uptake sites in terms of
  meaning (difficult to disentangle from other
  constructs, however)
• PCPs acknowledge the clear connection between
  MOVE!s ability to induce weight loss which
  impacts important performance measures like blood
  pressure
• and in terms of fit with pre-existing program
• At one low uptake site, some providers did not
  see this connection. However, MOVE! was
  compatible with physician champion values
• my just natural interest in this type of a
  program which is a more holisticapproach to
  managing some problems

38
INTERNAL CONTEXTImplementation ClimateRelative
Priority
39
INTERNAL CONTEXTImplementation ClimateRelative
Priority

• At one high-uptake site, tying MOVE! into their
  bariatric surgery program increased priority
• we were approved to start a bariatric surgery
  program bam, right away and unfortunately all
  our doctors and administrative people are
  enormously interested a bariatric surgeryand all
  resources and interests funneled into bariatric
  surgerywe did everything backwardsin hindsight,
  it probably was a good way to do that because our
  criteria for eligibility for people to have
  bariatric surgery is that they must be enrolled
  in MOVE! for one year
• One low-uptake site has to contend with many
  competing priorities that overshadow their MOVE!
  activities.
• Staff work weekends and lunches to get through
  backlog of patients in primary care
• MOVE! competes for space with other group classes

40
INTERNAL CONTEXTImplementation ClimateRelative
Priority
41
INTERNAL CONTEXTImplementation
ClimateIncentives Rewards

• One high-uptake site rewarded a clerk with a
  prize of recognition for stellar work on MOVE!
• A physician champion at one low-uptake site
  shared that
• we had no incentive, you know, we didnt get
  our boxes checked for getting this program
  implemented. You know, I didnt get a raise, I
  didnt get a bonus, nobody was patting me on the
  back
• particularly notable with lack of dedicated time

42
INTERNAL CONTEXTImplementation ClimateRelative
Priority
43
INTERNAL CONTEXTImplementation ClimateGoals
Feedback

• At one high-uptake site, anecdotal stories from
  successful patients motivate MOVE! team
• The other high-uptake site reports weight loss
  and goals
• Both sites emphasize outcomes other than weight
  loss
• At one low-uptake site, VHA performance measures
  are emphasized but
• Detriment to MOVE! because weight loss is not a
  performance measure
• One MOVE! team member works to track weight loss
  data from home because of lack of time at work

44
INTERNAL CONTEXTImplementation ClimateRelative
Priority
45
INTERNAL CONTEXTImplementation ClimateLearning
Climate

• At high-uptake sites coordinators not afraid to
  test new strategies and share dreams for
  improvement
• Act on lessons learned from VISN
• I did express that to my Chief of Staffthat I
  would like to see here what I call a MOVE
  Suite
• Weak Learning climate coordinator talked to
  potential physician champion informally to avoid
  getting arrows in her back

46
INTERNAL CONTEXTImplementation Climate

• In Summary, Implementation Climate comprises
• Tension for change
• Compatibility
• Relative Priority
• Incentives Rewards
• Goals Feedback
• Learning Climate

47
INTERNAL CONTEXTReadiness for Impl.Literature
Support
48
INTERNAL CONTEXTReadiness for Impl.Face Validity
49
INTERNAL CONTEXTReadiness for Impl.Leadership
Engagement

• High-uptake sites
• VISN MOVE! Coordinators actively involved with
  local facilities and with VISN leadership
• Help with data reporting and problem-solving
• Help ensure MOVE! is visible with leadership
• Local supervisors dedicate staff time for MOVE!
• Low-uptake sites
• In the last year, there has been a change of
  leadership in primary care and that has made a
  big difference...theyre giving it all they can
  possibly give it, given the overall constraints

50
INTERNAL CONTEXTReadiness for Impl.Face Validity
51
INTERNAL CONTEXTReadiness for Impl.Available
Resources

• All sites struggled with constrained resources
• High-uptake sites
• Tended to see lack of resources as a challenge
  worth solving
• Already existing weight management program
  attenuated influence of constrained resources
• Low-uptake sites
• Tended to see lack of resources as a reason that
  MOVE! couldn't possibly be fully successful
• Available resources is closely associated with
  leadership engagement and relative priority

52
INTERNAL CONTEXTReadiness for Impl.Face Validity
53
INTERNAL CONTEXTReadiness for Impl.Access to
Knowledge Information

• High quality and comprehensive patient and class
  materials, checklists, and guidelines were
  available to all sites
• High-uptake sites
• I think this is the only program that has a
  dedicated staff of experts that are there to help
  you when you need it. You can call them, you can
  email them, theyre there. They provided us with
  teaching materials, resources, equipment and
  supplies, it was just marvelous.
• Low-uptake sites talked about lack of information

54
INTERNAL CONTEXTReadiness for Impl.Face Validity
55
INTERNAL CONTEXTReadiness for Impl.Implementatio
n Leaders

• MOVE! Coordinators at 4 of 5 sites went above and
  beyond their assigned position
• Work from home and over lunch
• Buying patient incentives (e.g., t-shirts) out of
  own pocket
• Making healthy meals for reunion classes
• Physician champions played a much smaller role
• Only 2 agreed to talk to us for this study
• Did not actively participate at the low-uptake
  sites
• Played a supporting role at high-uptake sites
• One low-uptake site was exception with active
  physician champion helping teach classes

56
INTERNAL CONTEXTReadiness for Impl.

• In Summary, Implementation Readiness comprises
• Leadership engagement
• Available Resources
• Access to knowledge and information
• Implementation leaders

57
INTERNAL CONTEXTStakeholder AttributesLiterature
Support
58
INTERNAL CONTEXTStakeholder AttributesFace
Validity
59
PROCESSLiterature Support
60
PROCESSFace Validity
61
PROCESSEngage

• Coordinators at high up-take sites willingly
  volunteered for the role
• The staff thats gotten involved in the MOVE
  program was not appointed. We all had interest in
  this. We got involvedwe gotadministration to
  free this time up so that we can continue what
  becamea mandate
• Other team members challenging at all sites
  because of differences in priorities and
  supervisor support across departments

62
Consolidated Framework for Implementation
Research (CFIR)
Intervention at Time0
Intervention at Time1
Process over Time
Internal Context
Internal Context
Process
Soft Periphery
Process
Soft Periphery
Hard Core
Hard Core
Intervention
Intervention
External Context
External Context
63
Application of the CFIR

• Consists of 31 individual constructs
• Cannot use them all in every study
• And not all will apply
• A priori assessment of which constructs to
  include
• Only modifiable constructs?
• Determine levels at which each construct may
  apply
• E.g., teams, departments, clinics, regions

64
Conclusions CFIR

• Embraces, consolidates, and standardizes key
  constructs from other models
• Agnostic to models and theories
• Provides a pragmatic structure for approaching
  complex, interacting, and transient states of
  constructs in the real world
• Will help organize findings across disparate
  implementations and pave the way for multi-study
  analyses

65
Next Steps

• Work with other QUERI groups to use CFIR
• Gradually add to an database of evidence for
  constructs
• Shared Wikipedia of definitions and evidence
• Factor analysis of data in strength of evidence
  databaseto consolidate constructs and facilitate
  subsequent analyses (fewer variables, greater
  power)
• Continue to evaluate usefulness of the CFIR

66
Thank You!

• To see far is one thing,
• getting there is another
• Draft Manuscript Available
• Laura.Damschroder_at_va.gov

67
Models Included in CFIR

1. Conceptual Model for Considering the Determinants
   of Diffusion, Dissemination, and Implementation
   of Innovations in Health Service Delivery and
   Organization Greenhalgh T, Robert G, Macfarlane
   F, Bate P, Kyriakidou O Diffusion of innovations
   in service organizations systematic review and
   recommendations. Milbank Q 2004, 82581-629.
2. Implementation Model Klein KJ, Sorra JS The
   Challenge of Innovation Implementation. The
   Academy of Management Review 1996, 211055-1080
3. Theory-based Taxonomy in Nursing Leeman J,
   Baernholdt M, Sandelowski M Developing a
   theory-based taxonomy of methods for implementing
   change in practice. J Adv Nurs 2007, 58191-200
4. PARiHS Framework Rycroft-Malone J, Harvey G,
   Kitson A, McCormack B, Seers K, Titchen A
   Getting evidence into practice ingredients for
   change. Nurs Stand 2002, 1638-43.
5. Ottowa Model for Research Use Graham ID, Logan
   J Innovations in knowledge transfer and
   continuity of care. Can J Nurs Res 2004,
   3689-103
6. TCU Treatment Systems Simpson DD A conceptual
   framework for transferring research to practice.
   J Subst Abuse Treat 2002, 22171-182
7. Diagnosis Needs Assessment (DN/A) Kochevar LK,
   Yano EM Understanding health care organization
   needs and context. Beyond performance gaps. J Gen
   Intern Med 2006, 21 Suppl 2S25-29
8. Stetler Model of Research Use Stetler CB
   Updating the Stetler Model of research
   utilization to facilitate evidence-based
   practice. Nurs Outlook 2001, 49272-279
9. Process Model for Implementation Edmondson AC,
   Bohmer RM, Pisana GP Disrupted routines Team
   learning and new technology implementation in
   hospitals. Adm Sci Q 2001, 46685-716
10. Replicating Effective Programs Framework
    Kilbourne AM, Neumann MS, Pincus HA, Bauer MS,
    Stall R Implementing evidence-based
    interventions in health care Application of the
    replicating effective programs framework.
    Implement Sci 2007, 242
11. Framework for Organizational Transformation. Grol
    RP, Bosch MC, Hulscher ME, Eccles MP, Wensing M
    Planning and studying improvement in patient
    care the use of theoretical perspectives.
    Milbank Q 2007, 8593-138 .