Reactions of Surgical Program Directors to a WebBased Interactive Educational Program Focusing on Co

1
Reactions of Surgical Program Directors to a
Web-Based Interactive Educational Program
Focusing on Cognitive Skills Donald A. Risucci,
PhD1, Patricia J. Numann, MD2, Richard Welling,
MD3, Marsha F. Pfingsten, MEd4, Patrice Gabler
Blair, MPH4, Ajit K. Sachdeva, MD4 1New York
Medical College, Valhalla NY 2SUNY Upstate
Medical University, Syracuse, NY 3Good Samaritan
Hospital, Cincinnati, OH 4American College of
Surgeons, Chicago, IL
2
Quality Quantity of Teaching Learning
Opportunities
Traditional
Non-Traditional
3
Quality Quantity of Teaching Learning
Opportunities
Traditional
Skills Labs
Work hour restrictions
Non-Traditional
4
Quality Quantity of Teaching Learning
Opportunities
Traditional
Simulation Skills Labs
Work hour restrictions Patient Safety Concerns
Non-Traditional
5
Quality Quantity of Teaching Learning
Opportunities
Traditional
Computer-Based Learning Simulation Skills Labs
Work hour restrictions Patient Safety
Concerns Diminished Reimbursement
Non-Traditional
6
Quality Quantity of Teaching Learning
Opportunities
Traditional
Immersive technologies Computer-Based
Learning Simulation Skills Labs
Work hour restrictions Patient Safety
Concerns Diminished Reimbursement Increased
Specialization
Non-Traditional
7
Quality Quantity of Teaching Learning
Opportunities
Traditional
Web 2.0, 3.0.. Immersive technologies Computer-Ba
sed Learning Simulation Skills Labs
Work hour restrictions Patient Safety
Concerns Diminished Reimbursement Increased
Specialization Advanced Technologies
Non-Traditional
8
Seasonal Variation in Surgical Outcomes as
Measured by the American College of
Surgeons-National Surgical Quality Improvement
Program (ACS-NSQIP) Ann Surg. 2007 September
246(3) 456465. Michael J. Englesbe, MD,
Shawn J. Pelletier, MD, John C. Magee, MD, Paul
Gauger, MD, Tracy Schifftner, MS, William G.
Henderson, PhD, Shukri F. Khuri, MD, and
Darrell A. Campbell, MD From the Department
of Surgery, University of Michigan Health System,
Ann Arbor, MI Department of Veterans Affairs,
VANational Surgical Quality Improvement Program,
Aurora, CO and National Surgical Quality
Improvement Program, West Roxbury, MA.
9
(No Transcript)
10
(No Transcript)
11
Fundamentals of Surgery Curriculum American
College of Surgeons Division of
Education www.facs.org/education/fundamentalsofsu
rgery.html
12
Focus on Diagnosis
Diagnosis errors far outnumber medication errors
as a cause of claims lodged (26 v. 12 in one
study Sato, 2001) 32 v. 8 in another study
(Phillips et al 2004). Harris poll commissioned
by National Patient Safety Foundation one in six
people had personally experienced a medical error
related to misdiagnosis (Golodner 2004). Most
medical error studies find that 1030 percent
(range 0.656.8 percent) of errors are errors
in diagnosis. A recent review of 53 autopsy
studies found an average rate of 23.5 major
missed diagnoses (range 4.149.8 percent).
Disease-specific studies show that substantial
percentage of patients (range 2.1 61)
experienced missed or delayed diagnoses.
Schiff et al, 2005
13
Fundamentals of Surgery Curriculum

• Case-based interactive curriculum
• Delivered on-line to PGY-1 surgical residents
• Each scenario presents a patient resident can
  take a history, perform a physical
  examination, order tests, review records, and
  recommend therapeutic interventions.
• Toolbar allows access to the patients virtual
  chart, including appropriate reports, and
  archives the information gathered during the
  patient history, physical examination, and
  diagnostic testing.

14
Fundamentals of Surgery Curriculum

• Specific feedback for every action taken while
  interviewing, examining, diagnosing and managing
  the patient.
• If a residents actions in caring for a virtual
  patient result in dire consequences, the resident
  is required to start over.
• Upon completion of FSC, the resident will have
  diagnosed and recommended treatment for scores of
  patients

15
Fundamentals of Surgery Curriculum

• Learning exercise, not a summative assessment
  tool residents can explore various alternatives
  and receive feedback without concern about being
  formally evaluated.
• When complete, FSC will be composed of 11
  on-line modules with approximately 100
  interactive case-based scenarios representing the
  foundation of surgical cognitive content for
  PGY-1 residents across all surgical specialties.

16
A Scenario may open with a Learning Objective
Page, indicated by a Surgeon behind a Podium.
17
This is a Reference Page, indicated by a Surgeon
with an Open Book.
18
Scenario Authors provide a detailed description
of the patient. Images are created to exhibit
signs that the Resident is expected to notice and
to incorporate into the diagnostic process..
19
Question
Possible Choices
Progress Meter
key
When the Resident opens a work space, a list of
choices is presented, a key to the grade that
each choice can receive, and a Progress Meter..
20
When a selection is made, in this case a question
to be posed to the patient, the response appears
as illustrated above.
21
In this case, a critical question was selected,
as indicated by the green dot. The progress
meter is partially full, indicating that
additional critical choices must be made before
the Resident can continue.
grade
Progress Meter
22
After all critical choices have been made, the
Progress Meter registers 100, The Resident is
allowed to Continue to the next step in
evaluating the patient.
After all correct responses have been selected
23
The Resident can review the patients responses
to the history by selecting the Patient History
and Image Tab from the Patient Section of The
Toolbar. The Toolbar represents the patients
Virtual Chart.
24
The Resident can review the patients Vital signs
by selecting the Vital Signs Tab from the Patient
Section of The Toolbar.
25
Reports and test results which are in the
patients chart at the time of the encounter are
made available in the Records Section and the
Tests Section of the Toolbar at the start of the
branch. If a report or test is ordered within a
branch, it is made available at the appropriate
time.
26
Each selection receives a response that not only
indicates whether it is right or wrong, but
also provides feedback designed to enhance the
educational experience for the Resident.
27
After the Resident has successfully completed the
scenario, a Summary is presented. If the
Learning Objective was not presented at the
beginning of the scenario, it is presented here.
28
Key Learning Points that have been demonstrated
within the scenario are listed after the Summary,
to reinforce the learning
29
A Resident may choose to review the Scenario at
this time, or to return to the Menu for another
scenarios. All scenarios, as well as all
reference material, tables, and charts, remain
available to each Resident for the entire year.
Feedback
The Fundamentals of Surgery Curriculum is an
educational program, not a test. For that
reason, and to encourage each Resident to
investigate each scenario completely, personal
feedback is provided to the Resident however
this is not tracked in any way.
30
Current Study

• Evaluates Program Directors reactions to a
  preview of a scenario from the Fundamentals of
  Surgery Curriculum.

31
Methods

• Program Directors were invited via the Listserv
  to preview the Fundamentals of Surgery Curriculum
  during the October 2007 ACS Clinical Congress.
• Previews were scheduled for 31 Program Directors
  on a first-come first-serve basis.
• Each Program Director spent 1 hour engaged in
  the scenario

32
Methods

• After engaging in a scenario each Program
  Director completed a questionnaire requesting
• Age range
• Rating of comfort using computers
• Ratings of the scenarios utility in addressing 9
  broadly defined educational goals (e.g., Provides
  a Solid Foundation for Future Learning)
• Ratings of 6 separate features of the scenario
  (e.g., Ease of Use, Feasibility).
• Informal debriefings were also conducted and
  comments were collected.

33
(No Transcript)
34
(No Transcript)
35
Program Directors with higher levels of comfort
using computers tended to perceive the modules as
easier to use (p lt 0.01)
36
Ratings of Features

• Ease of use (logical flow and methods of
  navigation)
• Very difficult to use Extremely easy to use

Usefulness of multimedia components to present
material and enhance understanding Does not
enhance presentation Extremely useful
Quality of presentation (appealing color schemes,
format, and graphics) Very Poor Exceptional
37
Ratings of Features

• Extent to which interactivity within the scenario
  promotes
• engagement in solving the problem presented
• Not at all engaging Made me want to solve
  the problem

Extent to which immersion in this virtual
experience adequately recreates the environment
in which the scenario takes place Felt very
artificial Felt real
Feasibility of incorporating scenarios like this
one into the required PGY-1 curriculum Will not
be feasible Should be part of required
curriculum
38
Ratings of FSC Utility in Addressing Educational
Goals

• Prepares residents to apply critical thinking
  and judgment essential in clinical situations
  during the PGY-1 year.
• Improves residents patient care skills and
  confidence during the PGY-1 year.
• Delivers content consistent with current
  practices and/or evidence (when available).

39
Ratings of FSC Utility in Addressing Educational
Goals

• Provides a solid foundation for future learning.
  
• Challenges residents.
• Encourages self-assessment and reflection.
• Motivates residents to voluntarily complete
  other scenarios.
• Improves residents teamwork and
  professionalism.
• Increases resident satisfaction with the
  educational process.

40
Ratings of the Scenarios Utility for Addressing
9 Broadly Defined Educational Goals
41
Comments Informal Debriefing

• Scenarios were fun and interesting.
• Felt as if they were diagnosing a real patient
  very involved in scenario.
• Several asked about the availability of other
  modules.
• Enables delivery of useful information
  consistently and efficiently.
• Helps to address educational objectives within
  the 80 hour work.
• Exposes residents to cases they may not actually
  encounter.

42
Comments Informal Debriefing

• Difficulties with the original physical
  examination model were recorded and have led to
  enhancements in that component of the scenarios.
• Most Program Directors indicated interest in
  participating in future content reviews, beta
  testing, and research initiatives.
• Some Program Directors indicated that they would
  like to require residents to complete FSC prior
  to entering their program.

43
Evaluation Framework

• Reactions Perceptions of stakeholders users
• Learning Cognitive gains from engaging in the
  curriculum
• Behavior Changes in clinical performance
  resulting from exposure to curriculum
• Outcomes Effects on patient outcomes, quality
  improvement

44
Conclusions

• Program Directors generally perceived the
  Fundamentals of Surgery Curriculum as a very
  feasible and effective tool addressing important
  educational needs of entering PGY-1 residents.
• Positive reactions from Program Directors suggest
  that the Fundamentals of Surgery Curriculum are
  likely to be implemented and evaluated in a
  significant number of programs.