Educational Outcomes Measurement: What We

1
Educational Outcomes MeasurementWhat Weve
Learned, Whats Ahead
Derek T. Dietze, MAPresident
Presentation given at the 2006 Alliance for CME
Annual Meeting as
Executive Director of CME
Veritas Institute for Medical Education, Inc.
Hasbrouck Heights, New Jersey
2
Session Objectives
You will be better able to

• Evaluate you organizations educational outcomes
  measurement (EOM) in the context of others
  efforts
• Identify opportunities to advance EOM
• Describe the potential impact of EOM on the
  future of CME

3
Who Is in the Audience Today?

• What is your affiliation?

• Medical education/communication company (MECC)
• Hospital
• Medical school
• Medical specialty society
• Health system
• Commercial interest
• Other

4
Who Is in the Audience Today?
2. I have been involved in CME for _____ 1
less than 1 year 2 1 to 3 years 3 3 to 5
years 4 5 to 10 years 5 more than 10 years
5
Audience Opinion Poll
3. To what extent do you agree with the
following statement? Compared with other
organizations, my organization is doing a great
job of educational outcomes measurement.

• 1 Strongly disagree
• 2
• 3
• 4
• 5
• 6 Strongly agree
• 7 Not applicable

6
Audience Opinion Poll
4. To what extent do you agree with the following
statement?Educational outcomes measurement is a
fad that will pass with time. 1 Strongly
Disagree 2 3 4 5 6 Strongly Agree
7
Audience Opinion Poll
5. To what extent do you agree with the
following statement? Implementing EOM is
essential to the long-term success of my
organization.

• 1 Strongly disagree
• 2
• 3
• 4
• 5
• 6 Strongly agree
• 7 Not applicable

8
Audience Opinion Poll
6. To what extent do you agree with the
following statement? I need to learn a lot more
about EOM.

• 1 Strongly disagree
• 2
• 3
• 4
• 5
• 6 Strongly agree
• 7 Not applicable

9
CME should be a strategic asset for improving
performance and enhancing patient care.
CME in the public interest.
10
From the CEO of the ACCME (Dec. 2005)

• On the research front, primary studies and
  syntheses no longer need to ask if CE, in
  general, improves practice or other outcomes
  because there is so much evidence that many kinds
  and combinations can do so.
• From Robertson, Umble and Cervero, Journal for
  Continuing Education in the Health Professions
  23, 146, 2003

11
EOM The Past

• Early-mid 1990s discussion focused on the nature
  and definitions of educational outcomes
• Alliance for CME conference on educational
  outcomes
• Early-adopters and innovators
• Since 2000 we need to do EOM!

12
The Value of CME

• A CME evaluation should focus on identifying,
  measuring, and describing the value provided by
  an intervention that leads to enhanced
  performance, improved healthcare quality, and
  reduced costs.

Moore DE. A framework for outcomes evaluation in
the continuing professional development of
physicians. In Davis D, Barnes BE, Fox R, eds.
The Continuing Professional Development of
Physicians From Research to Practice. Chicago,
Ill American Medical Association 2003.
13
Levels of Educational Outcomes
Adapted from Moore DE. A framework for outcomes
evaluation in the continuing professional
development of physicians. In Davis D, Barnes
BE, Fox R, eds. The Continuing Professional
Development of Physicians From Research to
Practice. Chicago, Ill The American Medical
Association 2003.
14
Measurement Alliance for CME Competencies
Survey

• Competency 2.12, offer educational consultation
  that emphasizes measurement in physician
  organizations to identify goals for education
  that are specific to the practice and
  measurable,
• Rated the lowest in terms of responsibility and
  attainment by CME leaders, managers, and
  coordinators (2.6/2.6).

King R, North S. Alliance for CME Competency
Areas Research Project Report. Alliance for CME
2004.
15
Forces Driving EOM

• Accountability for effectiveness, impact, and use
  of resources
• Desire for exemplary status
• Performance improvement movement
• MOC and MOL requirements/initiatives
• Pay for performance movement
• Increasingly a requirement of commercial support
• ACCME is raising the bar for providers
• CME communitys desire for continuous improvement

16
Performance Improvement Movement
American Medical Association
October 6, 2004Performance Improvement
Activities approved for credit Stage A Learning
from current practice performance
assessment Stage B Learning from the application
of PI to patient care Stage C Learning from the
evaluation of the PI effort
May 23, 2005Internet Point of Care now eligible
for creditgt Review original clinical
question(s). gt Identify the relevant sources from
among those consulted. gt Describe the application
of their findings to practice.
17
Performance Improvement Movement (cont.)

• American Academy of Family Physicians
  Performance Improvement in Practice

18
MOC and MOL Initiatives
American Board of Medical Specialties Part
IV Maintenance of Certification Evidence of
evaluation of performance in practice

• July 2003 ABFM introduced new MOC requirements
  for family physicians.
• Part IV requirement replaced the traditional
  clinical chart review. Focus on improving quality
  of care, rather than on record-keeping.
• Each FP must complete 1 QI project during his or
  her recertification cycle to comply with Part IV.

19
Board of Directors Action Plan Fiscal
2004 Assure ACCME accreditation remains relevant
by addressing the needs of medical boards to
quantify learning and practice outcomes of
individual physician learners.
20
Pay for Performance Movement
Phase in of Medicare Pay for Performance
Q. Does the Academy (AAFP) have anything to help
me see what P4P might be like?A. Yes. METRIC --
Measuring, Evaluating and Translating Research
Into Care -- lets you earn 20 Prescribed CME
credits in your office as you complete
performance measurement and improvement projects.
21
Commercial Interests Expectations

• Increasingly, commercial interests are
  requiring CME providers to
• Include a sound outcomes measurement plan in
  grant requests
• Become more familiar with EOM methodologies and
  tools
• Report results for evaluation and future planning

22
Current Status of EOM

• Increasing understanding of EOM among some CME
  providers and MECCs
• Some measurement methodologies gaining acceptance
  and popularity
• Few standards or best practices
• Some measurement beyond satisfaction into
  learning and performance
• Lack of scientific rigor
• Much talk
• Limited experimentation
• Fear

23
EOM Challenges

• EOM competence, skills, and resources
• Time
• High level support resources
• Practical EOM methodologies/tools
• Funding/support for EOM from commercial interests
• Actual/perceived lack of access to the data
• Rigor mortis and perfection paralysis
• Fear of the results

24
Current EOM Methodologies Tools

• Follow-up surveys
• Immediate pre- and post-activity surveys/tests
• Control group surveys
• Focus groups
• Clinical assertions
• Clinical/case vignettes
• Commitment to change
• Pre- and post-activity prescription measurement
• Chart audits
• QA data

25
Pre/Post Clinical Assertion with Control
Early aggressive therapy is treatment
initiated within the first 6 months of RA
onset.
Data courtesy of Veritas Institute for Medical
Education, Inc.
26
Pre/Post Case Vignette Patient With AS

• How would you treat a 28-year-old male patient
  exhibiting the following symptoms
• Morning stiffness
• X-ray shows bilateral (sacroiliitis)
• Swollen, painful knee
• ESR 10
• CBC normal

27
Case AS PatientPercent of Pre-/Post-Discussion
Responses
How would you treat?
NSAIDs
SSZ
MTX
Anti-TNF
Combination
Percent of Responses
Data courtesy of Veritas Institute for Medical
Education, Inc.
28
Case AS PatientPercent Change in
Pre-/Post-Discussion Response
How would you treat?
NSAIDs
SSZ
MTX
Anti-TNF
Combination
Percent Change in Responses
Data courtesy of Veritas Institute for Medical
Education, Inc.
29
Case Vignette Post-activity with Control
N46 for both groups
P0.05
Outcomes, Inc. CME Metrics Report, July 25, 2005.
Challenging Cases in GastroenterologyThen Now,
Jobson Education Group. Satellite symposium at
Digestive Disease Week, May 15, 2005.
30
Pre/Post/Follow-up Clinical Assertions with
Control
Only patients with severe and/or refractory cases
of urticaria should be managed with steroids.
Strongly agree
ANOVA P lt 0.001
Mean Rating
Strongly disagree
n390
n1,753
n1,374
FU Follow-up survey EM Enduring material
n335
n229
n67
n372
Dietze DT, Magazine HI. Enhancing and Sustaining
Learning Report of a Comprehensive CME
Initiative and Educational Outcomes Measurement
Plan (forum). Alliance for CME 31st Annual
Conference January 24, 2006 San Francisco,
California.
31
Performance Improvement Project

• Purpose
• Decrease the risk of recurrent vascular events
  in patients with AMI and CAD by initiating
  therapy with lipid-lowering agents during
  hospitalization.
• Measurement Point
• Percentage of defined AMI and CAD patients who
  received a lipid lowering agent charge during
  stay.
• Multidisciplinary Team on PI Project
• Cardiology Physicians, Telemetry Nursing Staff,
  Quality Services, CME Department, Pharmacy
  Decision Support (data specialist)

Information courtesy of Susan G. McAlexander,
MPA, CHES, CME Coordinator, Providence St. Peter
Hospital, Olympia, WA, February 2006.
32
Measurement, Data Collection Analysis

• Lipid use rates compared monthly and quarterly
  for changes after interventions.
• Education CME and other
• Forms adjustments
• In-services
• Monthly project task force meetings to project
  planning, discuss interventions and results.
• Aggregate and specific rate collection performed
  in the Quality Department utilizing charge data
  and chart review.
• Assessment of knowledge post CME single activity
  intervention performed by CME department.
• Monthly tends reported on graphs.

33
Results and Application to Future CME

• Goal percentage attained in 3rd quarter for AMI
  patients and 4th quarter for ICS patients.
• Greatest change noted in the cardiology and
  internal medicine sections. Family medicine N
  too low to have statistical significance.

• Final report to CIC included a recommendation to
  tie CME into future PI projects for maximal
  impact.
• CME was seen to have a positive and reinforcing
  effect on physician learning and practice change.

34
Measurement Is in Your Future!
35
From the CEO of the ACCME (Dec. 2005)

• The Board wants to make it clear that the ACCME
  values an accreditation system that rewards
  providers
• for establishing a mission that identifies
  improvements in physician performance and/or
  patient or population health as organizational
  goals, and
• b) for having a process in place to measure and
  increase its effectiveness in meeting this
  mission.

36
From the CEO of the ACCME (Dec. 2005)

• Over the next few monthspresentation of draft
  new compliance criteria and policy through which
  ACCME will reward providers for having processes
  in place
• To facilitate changes in knowledge, strategy,
  performance-in-practice, healthcare delivery or
  patient health.
• To facilitate life-long learning based on
  individualized needs assessment where
  effectiveness is measured by the learning or
  improvements that occur.
• To deliver education in support of the
  ACGME/ABMS Core Competencies, around
  specialty-specific curricula and in the context
  of the IOM Competencies.

37
Audience Opinion Poll
7. To what extent do you agree with the
following statement? New ACCME compliance
criteria for measurement of CME effectiveness
would be a positive step forward for the CME
community.

• 1 Strongly disagree
• 2
• 3
• 4
• 5
• 6 Strongly agree

38
Audience Opinion Poll
8. To what extent do you agree with the
following statement? Compared to other
organizations, my organization is doing a great
job of EOM.

• 1 Strongly disagree
• 2
• 3
• 4
• 5
• 6 Strongly agree
• 7 Not applicable

39
Audience Opinion Poll
9. To what extent do you agree with the
following statement? Implementing EOM is
essential to the long-term success of my
organization.

• 1 Strongly disagree
• 2
• 3
• 4
• 5
• 6 Strongly agree
• 7 Not applicable

40
Vision for EOM

• EOM standards and benchmarks
• EOM best practices
• Proven methodologies and tools
• More efficient and effective CME
• Improved patient care
• Improved patient outcomes
• Improved population health

41
Thank You for Your Participation!

• Derek T. Dietze, MAPresident
• derek.dietze_at_improvecme.com

Queen Creek, Arizona