Introduction To Evaluating Process And Outcomes In Research Design

1
Introduction To Evaluating Process And Outcomes
In Research Design

• Sarita Bhalotra
• Donald Shepard

August 18, 2004
2
Human Services Evaluation
1960s

• Johnsons Great Society proliferation of
  experimental and pragmatic H.S. programs

• Results in need for evaluation

3
Human Services Evaluation
1970s

• Experimental approach of programs inadequate to
  understand human services programs

• Traditional approach of evaluation over-reliant
  on bio-medical model

4
Human Services Evaluation
1980s

• Post modernist approach emphasizes
• practical problem-solving orientation
• to program evaluation

• Outcomes are personal or organizational
• changes or benefits as a result of activity,
• intervention, or service

5
Lifestyle Modification Program Demonstration
Evaluation
1. Outcomes Evaluation a. Clinical b.
Cost/Utilization c. Cost-effectiveness
2. a. MBMI and LA/PMRI b. Program Sites c. Del
Marva d. CMS
6
PROGRAM THEORY FOR MEDICARE LIFESTYLE
MODIFICATION DEMONSTRATION
Population Age 65 years Clinically high-risk
for progression of cardiovascular disease
Additional Services Nutrition Exercise Stress
Management Group Support
Health Status Reduced Cardiovascular
Events Improved Q.O.L. Reversal of
Cardiovascular Disease (Cost-saving/cost-neutral)
Appropriate
Effective
Efficient
Resources Additional Medicare Funds Additional
Demonstration Site funds In-Kind Contributions
7
ContextLMPD is testing two lifestyle
interventions nationwide1. Mind/Body Medical
Institutes Cardiac Wellness Program2. Dr.
Dean Ornish Program for Reversing Heart
Disease
Research QuestionWhat organizational
characteristics explain the number of Medicare
beneficiaries who participate in the Medicare
LMPD?

• Specific Aims
• Determine the critical stakeholders and their
  roles in the Medicare LMPD
• Establish the critical steps and their sequencing
  needed for the successful participation of
  Medicare beneficiaries in the Medicare LMPD.
• Analyze the characteristics of, and interactions
  between and among stakeholders that impacts the
  critical steps and the extent to which these
  affect the participation of Medicare
  beneficiaries in the Medicare LMPD.
• Develop a program model of the structure,
  processes, and intermediate outcomes of a
  successful organization in terms of Medicare
  beneficiary participation.

8

• Hypotheses
• Successful sites will have identified and
  cooperated with critical stakeholders, especially
  referral sources, in the planning phase.
• Successful sites will have designed an effective
  senior management team, and selected and train
  motivated staff.
• Successful sites will have developed and
  implemented participant-focused marketing,
  recruitment, enrollment and retention techniques.
• Successful sites will have developed and
  implemented procedures for tracking and
  maintaining relationships with stakeholders

9
(No Transcript)
10
(No Transcript)
11
Cumulative Enrollment
in Lifestyle demonstration by Program
12
Medicare Lifestyle Modification
DemonstrationEnrollment Review and Participation
Status
13
Enrollment Barriers For Patients Contacted by
Nurse Recruiter, First Ten Months (excluding
those in Process), n 1, 387
Note Information was updated on slightly
different dates. Minor discrepancies in totals
reflect differences in dates of data collection.
14
Results of Nurse Recruiter, First 10.5 Months (n
1706)
15
Nurse Recruiter Outcomes, First 10.5 Months
(n1706
16
Selected Stakeholder Groups and Key Steps Leading
to Participation(Specific Aims 1 and 2)

17
Barriers to Enrollment Across Programs

• Participants
• Out of pocket cost
• Time of day (e.g., ends after dark)
• Time demands
• Rigorous diet too time-consuming in preparation
• Program is too stringent to follow
• Lifestyle changes are not supported by
  environment
• Extensive documentation needed for enrollment
  
• Time consuming screening required
• Program staff
• General mailing is expensive
• Limited time available for some targeted
  recruitment despite its value (e.g. inpatient
  nurses)
• No time for other targeted recruitment (e.g., via
  cath lab)
• Documentation is labor intensive
• No benefits to program staff apparent from
  documentation

18
Barriers to Enrollment Across Programs

• Host Facility
• Not convinced of clinical efficacy
• Financial loss leader
• Takes away from other programs (e.g. cardiac
  rehab)
• Impinges on turf of other physicians
• Highly politicized in some cases (e.g.
  champions cause)

19
Barriers to Enrollment Across Programs

• Community physicians/ Referral sources
• Not convinced of efficacy of programs
• Not sure patients will comply even if efficacious
• Easier to prescribe drugs, other medical or
  surgical intervention
• Concerned about loss of revenue/patients
• Do not want to refer patients who they think will
  fail
• Frustrated when referred patients face long
  delays or are not found to be eligible
• Frustrated by amount of paperwork required to
  document a patients eligibility
• Concerned by the time of the doctors and nurses
  to explain the program to a patient
• Information available is incomplete and outdated

20
Key Activities, Interactions, and Characteristics
of Successful Organizations (Specific Aims 3 and
4)

• Marketing is multi-modal, but targeted especially
  to referring physicians
• Marketing anticipates physician resistance, and
  is convincing of the clinical efficacy,
  feasibility, and lack of threat to current
  practice
• Institutional supports financially, culturally,
  and clinically
• Eligibility criteria do not pose either a
  substantive or logistic barrier
• Reimbursement is by all payors or majority
• Program is a substitution for or complementary
  with traditional cardiac rehab
• Program Leadership is conducive to trust and
  credibility
• HQ Leadership is conducive to trust and
  credibility

21
Examples From Site Visits
On a scale of 1 to 5, from ineffective to
effective.
22

• Major Issues
• The target community opposes the change being
  advocated
• The adoption costs exceed tangible benefits
• Effective structural and motivational systems
  designed to engage and reward are needed at every
  level
• Consumer needs, preferences and lifestyles are
  paramount

• Conclusions
• Enrollment in LMPD, as in other preventive
  programs, is a challenge
• Environmental scans help identify barriers and
  ways to overcome them
• Interventions, such as the nurse recruiter, are
  achieving partial success
• Lifestyle Modification enrollment remains below
  program expectations