Strengths

1
Strengths Weaknesses of a Pre-Post Controlled
Randomized Trial

• Michael E. Matheny, MD MS
• Decision Systems Group
• Brigham Womens Hospital, Boston, MA

2
Trial Design
3
Trial Design
4
Pre-Post Randomized DesignStrengths

• Allows the study design to account for changes in
  measured outcomes by unmeasured factors over time
• Provides a single p value for the effect of the
  intervention that incorporates all of the data
• Used primarily for
• Studies that occur over a longer time period
• Studies in an environment where other changes
  taking place could affect the measured outcomes

5
Pre-Post Randomized DesignWeaknesses

• Requires allocation of significant time and
  resources prior to introduction of the
  intervention
• Doubles the cost of data collection
• Very little power gained from doubling the
  sample size
• calculations are still based on two arm design
• Unable to provide a single odds ratio result for
  the intervention

6
Data AnalysisSAS Code
7
Data Analysis Pre-Post Controlled Design

• Use of interaction term in this study design
• There are 4 study groups
• Multiplicative interaction term of (Pre/Post)
  (Control/Intervention) can be interpreted as the
  relative change in outcome between comparison
  groups from the baseline to the follow-up
  evaluation
• Pre/Post
• Control/Intervention
• PostIntervention
• Reported as a p value only
• Odds Ratios are reported for both the Control arm
  and for the Intervention arm

8
Example Results150 Per Arm
Group Pre Post OR 95 CI Interaction p
Outcome of Interest Control 33 33 1.00 0.62-1.61 0.042
Intervention 33 50 2.00 1.25-3.19
9
Example Results150 Per Arm
Group Pre Post OR 95 CI Interaction p
Outcome of Interest Control 33 40 1.33 0.83-2.14 0.047
Intervention 33 57 2.61 1.64-4.18
10
Example Results150 Per Arm
Group Pre Post OR 95 CI Interaction p
Outcome of Interest Control 33 17 0.40 0.23-0.70 0.014
Intervention 33 33 1.00 0.62-1.61
11
Impact of an Automated Test Results Management
System on Patients Satisfaction of Test Result
Communication

• Michael E. Matheny,
• Tejal K. Gandhi, E. John Orav,
• Zahra Ladak-Merchant, David W. Bates,
• Gilad J. Kuperman, Eric G. Poon
• Decision Systems Group / Division of General
  Medicine
• Brigham Womens Hospital, Boston, MA

12
BackgroundTest Result Communication

• Test result communication between patients and
  physicians is a critical part of the diagnostic
  and therapeutic process
• However, follow-up of test results in the primary
  care setting is often challenging
• High volume of test results
• Test results arrive when physician not focused on
  the patient
• Lack of systems to ensure reliability and
  efficiency

13
BackgroundTest Result Communication Problems

• 31 of women with abnormal mammograms did not
  receive care consistent with established
  guidelines
• 39 of abnormal TSH at BWH were not followed up
  within 60 days
• 36 of abnormal pap smear were lost to follow-up

14
BackgroundPhysician Workflow

• 33 of physicians reported they did not always
  notify patients of abnormal test results
• 30 of physicians reported they did not have a
  reliable method of test result communication
• 59 of physicians were dissatisfied with how well
  they managed test results despite spending over
  an hour a day in this activity

15
BackgroundPatient Expectations

• Patients do not normally discuss their
  preferences for test result notification with
  their providers
• Patients preferred telephone notification to
  regular mail, and found electronic notification
  to be uncomfortable due to security issues
• Patients wanted to be notified of all test
  results, regardless of whether the results were
  abnormal

16
BackgroundPatient Satisfaction

• These problems reduce patient satisfaction with
  their medical care, and impair future
  patient-physician interactions
• Improving patient satisfaction has been
  identified as one of the most important issues
  currently facing healthcare

17
Objective

• To evaluate the impact of an automated test
  results notification system imbedded into an
  electronic health record on patient satisfaction
  regarding test results communication

18
MethodsStudy Setting

• Partners HealthCare System
• Brigham Womens Hospital
• Massachusetts General Hospital
• Faulkner Hospital
• McLean Hospital
• Newton-Wellesley Hospital
• Free Standing Outpatient Clinics
• Longitudinal Medical Record (LMR)
• Released July 2000
• Scheduling
• Medication lists
• Problem lists
• Health maintenance record
• Clinic notes (free form templates)

19
MethodsLMR Summary Screen
20
MethodsStudy Setting

• Usual care regarding test results management
• Test results were embedded directly into the
  patients electronic health record
• No automated test results tracking
• All test results were mailed to the physicians
  clinic office
• Physicians were paged directly for critical
  results

21
MethodsPatient Test Results Screen
22
MethodsIntervention

• Results Manager - an electronic test results
  management system embedded into the LMR
• Tracks and displays all test results associated
  with an ordering physician
• Prioritizes by degree of test result abnormality
• Facilitates review of test results in context of
  patients history
• Generates test result letters
• Allows clinicians to set reminders for future
  testing

23
MethodsResults Manager Summary Screen
24
MethodsResults Manager Letter Generation Screen
25
MethodsStudy Design
26
MethodsRandomization

• Stratified randomization of 26 primary care
  clinics based on
• Primary Hospital Affiliation (BWH / MGH)
• Academic or Community Based
• Average Patient Socioeconomic Status
• Rolling implementation of Results Manager for
  intervention clinics was conducted from July,
  2003 to March, 2004.

27
MethodsStudy Criteria

• Patients were randomly sampled in both control
  and intervention clinics during
• Pre-Intervention 12/2002 06/2003
• Post-Intervention 09/2003 04/2005
• Inclusion Criteria All patients who had any
  laboratory, pathology, microbiology, or radiology
  tests from participating clinics.
• Exclusion Criteria Primary care physician
  determined that patient should not be contacted
  or patient did not speak English.

28
MethodsSurvey Instrument

• Internally developed
• Outcomes were measured on dichotomized Likert
  scale
• Administered by trained research assistants 5 to
  7 weeks after test results were posted
• Up to three attempts were made to contact each
  patient

29
MethodsPrimary Outcome Measure

• Overall satisfaction with test result
  communication
• I am satisfied with the way test results are
  communicated to me

30
MethodsSecondary Outcome Measures

• Satisfaction with PCP listening skills
• My primary care doctor always listens to my
  concerns
• Satisfaction with information given about
  treatment and condition
• My primary care doctor gives me as much
  information about my condition and treatment as I
  wanted
• Satisfaction with general PCP communication
• My primary care doctor and I communicate very
  well

31
MethodsOutcomes

• Whether a patients expectations were met by the
  method of test result communication was
  determined by
• Test result type normal / abnormal
• Defined as requiring follow-up or a management
  plan change
• Method of test result receipt
• Patients expected delivery method for test
• Hierarchy of test result communication
• Same Visit gt Telephone gt Letter gt Email gt Next
  Visit gt Never
• If receipt was by a more desired method, it was
  counted

32
MethodsData Analysis

• Intention-to-treat analysis
• Multivariate logistic regression (GEE)
• Clustered by primary care physician
• Adjusted for Patient Age, Gender, Race, Insurance
  Status, and Self-Reported Health Status
• SAS v9.1

33
Results Response Rates
Contacted Eligible (Surveyed) Contacted Eligible (Refused) Unable to Determine Eligibility p
Patient Age (Years) 57.4 58.6 50.0 lt0.001
Patient Sex ( Female) 399 (70.0) 124 (62.6) 495 (65.9) 0.109
Patient Race
White 374 (65.6) 119 (60.1) 426 (52.7) 0.005
Black 109 (19.1) 44 (22.2) 161 (21.4) 0.498
Other 87 (15.3) 35 (17.7) 164 (21.9) 0.009
Patient Insurance
Commercial 257 (45.1) 88 (44.9) 444 (59.1) lt0.001
Medicare 246 (43.2) 90 (45.5) 189 (25.2) lt0.001
Medicaid 56 (9.8) 16 (8.1) 95 (12.7) 0.099
Self Pay 4 (0.7) 3 (1.5) 23 (3.1) 0.008
Totals 570 198 751
34
Results Demographics
Sample
Patient Age (Mean Years) 57.3
Patient Sex ( Female) 70.0
Patient Race ()
White 65.6
Black 19.1
Other 15.3
Patient Insurance ()
Commercial 45.1
Medicare 43.1
Medicaid 9.6
Self Pay 1.1
Totals 570
35
Results Demographics
Control Intervention p
Patient Age (Mean Years) 57.1 57.7 0.648
Patient Sex ( Female) 64.6 76.3 0.003
Patient Race ()
White 65.9 65.3 0.929
African American 19.1 19.1 1.000
Other 14.9 15.7 0.816
Patient Insurance ()
Commercial 50.0 42.0 0.064
Medicare 40.9 45.8 0.270
Medicaid 8.4 11.4 0.260
Self Pay 0.7 0.8 1.000
Totals 313 257
36
Results Primary Outcome
Group Pre Post OR 95 CI Interaction p
Satisfaction with Test Control 90 82 0.682 0.345-1.347 0.012
Result Communication Intervention 85 93 2.380 1.068-5.306
37
Results Secondary Outcome
Group Pre Post OR 95 CI Interaction p
Expectation Met Regarding Control 61 47 0.469 0.326-0.674 0.001
Method of Communication Intervention 44 58 1.394 0.827-2.350
38
Results Secondary Outcome
Group Pre Post OR 95 CI Interaction p
Satisfaction with Information Control 95 87 0.711 0.261-1.937 0.034
Given for Treatment Intervention 94 96 3.671 1.300-10.37
And Condition
39
Results Secondary Outcome
Group Pre Post OR 95 CI Interaction p
Satisfaction with PCP Control 99 93 1.024 0.062-16.79 0.501
Listening Skills Intervention 99 98 2.962 0.760-11.54
40
Results Secondary Outcome
Group Pre Post OR 95 CI Interaction p
Satisfaction with General Control 96 91 1.373 0.404-4.668 0.367
PCP Communication Intervention 97 96 2.860 0.957-8.547
41
Discussion

• These findings could be related to a number of
  potential workflow improvements in RM
• Provided a concise summary page for the
  management of test results ordered by a provider
• Template-based results letter generator
• Can imbed actual test results into letter
• Improve patient-friendly interpretations of
  results
• One-click patient reference information

42
DiscussionLimitations

• Generalizability
• Imbedded in non-commercial EHR
• English speaking only
• Telephone Survey Bias
• Responders vs. Non-Responders
• Patient Recall

43
Conclusions

• An automated management system that provides
  centralized test result tracking and facilitates
  contact with patients
• improved overall patient satisfaction with
  communication of test results
• Increased patient satisfaction with the
  discussion of treatments/conditions
• Improved receipt of results by an expected method
  of communication

44
Acknowledgements

• Co-Authors
• Tejal K. Gandhi, MD MPH
• John Orav, PhD
• Zahra Ladak-Merchant, BDS MPH
• David W. Bates, MD MS
• Gilad J. Kuperman, MD PhD
• Eric G. Poon, MD MPH
• Funding
• AHRQ U18-HS-11046
• NLM T15-LM-07092

45
Michael Matheny, MD MS mmatheny_at_partners.org
Brigham Womens HospitalThorn 30975 Francis
StreetBoston, MA 02115
The End
46
MethodsSurvey Response Data