Feedback

1
Feedback Evaluation

• Eva Aagaard, MD
• Vice Chair for Education
• Department of Medicine

2
Overview

• Feedback
• Why should you care about feedback?
• What is feedback?
• Whats the evidence that feedback works?
• How do you give feedback effectively?
• Case Discussion

3
Feedback
4
Why Should You Care?

• Trainees WANT feedback
• In a study of over 1500 residents, 96 believed
  feedback was important for learning
• Schultz, BMC Central 2004
• Trainees feel they NEVER get enough feedback
• Gil, J Med Educ 1984
• Isaacson, J Gen Int Med 1995

5
Feedback is ESSENTIAL to Teaching

• Giving high-quality feedback is strongly
  associated with teaching ratings
• Torre, Acad Med 2003
• Constructive and specific feedback can improve
  learner knowledge and skills (More on this
  later)
• Boehler, Med Ed 2006
• Clay, Critical Care Med 2007

6
Definitions

• Or What is Feedback?

7
Evaluation VS. Feedback

• Evaluation
• Summative
• Higher stakes
• Generally standardized
• Goal is to grade relative to peers or a gold
  standard
• Feedback
• Formative
• Goal is to help people improve
• Can be formal or informal

8
Categories of Feedback

• Brief feedback
• Brief, specific notation of a positive or
  negative behavior
• Generally provided for directive teaching
• Ex
• I think that what you noted to be distension is
  actually ascites. Let me show you some techniques
  to distinguish these things.
• Branch, Acad Med 2002

9
Categories of Feedback

• Formal Feedback
• Set aside a period of time (5-20 min) to discuss
  performance on a specific issue
• Can be event or encounter specific
• Exs
• Mistakes
• Handling of a specific patient case
• Feedback following a case presentation or a
  teaching encounter
• Still very specific and generally limited in
  scope
• Branch, Acad Med 2002

10
Categories of Feedback

• Major Feedback
• Scheduled sessions to review overall performance
  to date (15-30 min)
• Must be given in a timely enough fashion that
  trainee has the opportunity to improve/ grow/
  respond
• Ex
• Midpoint feedback on a rotation
• Branch, Acad Med 2002

11
The Evidence
12
Procedures TrainingBoehler, Med ED 2006

• Does Provision of Specific Procedural Skill
  Feedback Improve Performance in that Skill
  Compared to General Comments?

13
Design

• 33 2nd and 3rd year Med Students at Southern
  Illinois University
• All given same instruction on 2-handed surgical
  knot-tying
• Randomized to specific feedback or general
  compliments
• Videotaped performing the skill before and after
  feedback
• Videotapes scored by 3 blinded observers
• Student satisfaction with instruction assessed
  via 7-point global rating scale

14
Mean Performance Scores
Plt0.001
P0.2
Total Possible Ratings 0-32 IRR gt0.8
15
Mean Satisfaction Scores
P0.005
7-Point Likert Scale (1Very Poor 7 Truly
Exceptional)
16
Take Home Points

• Specific feedback can improve performance on
  procedural skills
• FEEDBACK ? LEARNING
• Satisfaction with feedback is a poor marker of
  the QUALITY of feedback
• SATISFACTION ? QUALITY
• Combination of specific feedback and compliments
  may be best option

17
Nontechnical SkillsSavoldelli, Anesthesiology
2006

• Does Feedback Improve Resident Performance in
  Nontechnical Simulator Skills?
• Is Video-Assisted Oral Feedback Better than Oral
  Feedback Alone?

18
Design

• Forty-two 1st, 2nd and 4th year anesthesia
  residents at the University of Toronto
• Videotaped managing an intraoperative cardiac
  arrest on a simulator with scripted operating
  room personnel
• Randomized to 3 arms
• No feedback
• Oral feedback
• Video-assisted oral feedback
• Videotaped on a second (different) intraoperative
  cardiac arrest scenario

19
Design Cont

• Videos scored by 2 blinded reviewers using a
  validated behavioral marker system (ANTS)
• Task management
• Team working
• Situational awareness
• Decision making
• Scenario order randomized
• Reviewers blinded to chronological order of
  scenarios
• OUTCOMEMean change in ANTS score

20
Key Feedback Points

• Focused on nontechnical (cognitive and
  behavioral) skills
• Resident reflection on encounter elicited
• Specific and constructive comments provided
• Both positive and negative
• Feedback limited to 4-6 main points

21
of ANTS Score Change
Plt0.05
Plt0.01
22
Take-Home Points

• Oral feedback improves nontechnical skills in a
  simulated crisis environment
• Specific, limited feedback along with resident
  self-reflection were used in this study
• Videotape review was not better than oral
  feedback alone

23
How Do I Do It Well?

• Giving Feedback 101
• Adapted from Ende, JAMA 1983

24
Key Principles

• The Groundwork
• The Meeting
• Things to Remember

25
The Groundwork

• Set clear objectives and goals upfront
• What does the trainee hope to get out of your
  time together?
• What specific behaviors do you expect?
• When will you give the trainee feedback?
• SET AN APPOINTMENT NOW
• When will you reassess their performance and
  reset goals?

26
Preparation

• Organize your thoughts and observations ahead of
  time
• Setting
• Positive brief feedback can be given almost
  anywhere
• Negative or major feedback should always be given
  in private and without interruptions
• Timing
• As close to event as possible
• Enough time to make a change
• Make an appointment
• Make sure learner is ready to hear it
• Not being paged
• Not distracted
• Not distraught

27
Opening the Encounter

• Describe the Purpose
• Label it FEEDBACK
• Sostock, Acad Med 2002

28
Conducting the Meeting

• ASK
• Elicit self-reflection
• TELL
• Give both positive and constructive feedback
  ALWAYS
• Be SPECIFIC and use non-judgmental language
• Behaviors not personality
• Objective, observable and modifiable
• Provide suggestions for how to improve
• ASK
• Elicit trainee understanding of feedback
• Allow trainee to develop own suggestions for
  improvement plan

29
Closing the Meeting

• SUMMARIZE
• Positives
• Areas for improvement
• Plan for improvement
• Plan for when to reassess/ meet again

30
REMEMBER.

• Limit constructive feedback to NO MORE than 4
  (and probably 2) areas

31
REMEMBER

• Listen to the persons perspective and feelings

32
PRACTICE
33
BRIEF FEEDBACK
34
The Consult Call

• You ask the MS4 to call a consult on a patient
  with brisk UGI bleeding. You listen as she speaks
  to the fellow on call. The presentation includes
  all pertinent material and is concise and
  organized. She asks specifically if the fellow
  would be willing to perform an emergent EGD, and
  states that this is the reason for her call.

35
Your Feedback?

• When?
• Where?
• What?

36
BRIEF FEEDBACK

• Timely- in the moment
• Generally can be given anywhere (unless of
  sensitive nature)
• Specific
• Objective

37
An Example

• I wanted to give you some quick feedback on
  your consult call. You provided the patient
  information clearly and concisely. More
  importantly, you clearly stated the reason for
  the consult. Nice work.

38
FORMAL FEEDBACK
39
The Medical Student Case Presentation

• Mr. Jones is a 50 year old man with cirrhosis
  related to hep C and alcohol abuse who is
  admitted with altered mental status. He was
  brought in by his friend who found him confused
  and lethargic after not showing up to work as
  expected. The patient is unable to give a history
  of recent events due to marked lethargy and
  confusion.
• Past medical is as already noted.
• Meds are unknown. He was discharged on
  spironolactone and atenolol at his last admission.

40
Presentation Cont

• He works construction and lives alone. The
  friend believes that he has quit drinking and
  drug use, although he is not sure.
• On PE he has normal vital signs.
• Neuro exam is nonfocal.
• Abdomen is distended with tense ascites,
  diffusely tender without rebound.
• Rectal is hemoccult negative.

41
Your Feedback?

• When?
• Where?
• What?

42
FORMAL FEEDBACK

• Label It- Let me give you some feedback on your
  presentation on Mr. Jones
• Set an appropriate time and place
• Timely
• Private place
• Include positive and negative
• Be specific
• Provide suggestions for improvement

43
An Example

• Your identification of Mr. Jones as a cirrhotic
  due to alcohol and Hep C helped give me an idea
  of what the most likely causes of his altered
  mental status are
• - It would have been very helpful to have a
  better understanding of the severity of his
  underlying illness. It sounds as if that
  information may have been available in the most
  recent discharge summary

44
Suggest

• Please make sure to include anything you think
  might be important information from the medical
  records in your oral presentation next time.

45
MAJOR FEEDBACK
46
Mid-Point Feedback

• You have been working with your medical student
  for 2 weeks. His presentations are excellent and
  his clinical acumen is strong.
• Today one of your nurses pulled you aside. Per
  your instructions, she had been calling him with
  the results of labs and x-rays on the patients
  that he has seen in clinic. She is upset because
  yesterday he took a long time to return pages and
  when he finally did call her back, he was abrupt.
• Today is the day you are scheduled to give
  midpoint feedback.

47
Your Feedback?

• When?
• Where?
• What?

48
MAJOR FEEDBACK

• Set the Stage
• Private place
• Enough time free of interruptions
• ASKTELL---ASK
• Give both positive and constructive feedback
• Be SPECIFIC and use non-judgmental language
• Behaviors not personality
• Objective, observable and modifiable
• Provide suggestions for how to improve, and
  elicit a self-improvement plan from the learner
• Summarize

49
An Example

• Set the Stage
• Its time for our feedback session.
• Lets go to my office
• Is now a good time?
• ASK
• How do you think you are doing so far?
• What things have gone particularly well?
• Are there any areas that you think you need to
  improve on?

50
Example Cont

• Tell
• Your oral presentations and written notes are
  clear, concise, and do an excellent job of
  depicting our assessment and plan.
• Your clinical decision-making has been
  consistently right on such that I feel I can
  completely count on your recommendations. This
  has been true even in complicated cases such as.

51
Constructive Feedback

• One of the nurses approached me today with some
  concerns..
• That really surprised me. Is something going on
  that I should know about?

52
Example Cont.

• Suggest
• If there are ever issues going on for you that
  make it hard for you to do your work, it is
  extremely important for you to tell someone. In
  this case, me or someone else you feel
  comfortable talking to. We are here to help you.
• ASK
• What do you want to do at this point to rectify
  things with Nancy?
• Do you feel comfortable calling me or someone
  else if something like this arises again?

53
Example Cont.

• SUMMARIZE
• Overall your performance to date has been
  excellent
• Specifically..
• Oral presentations
• Written Notes
• Clinical judgment
• You made a significant error in judgment by not
  letting me know that you needed my help.
• From now on you will call on me, or someone else
  when there are other issues that interfere with
  your duties
• Lets go talk to Nancy now to fix things.

54
Key Points

• Set expectations UPFRONT
• Make the time and space to do it
• Be specific and describe observable behaviors
• Give suggestions for ways to accomplish your
  expectations
• Practice, practice, practice!!!

55
QUESTIONS