Standardized Letters of Recommendation

1
Standardized Letters of Recommendation

• Anna H. Messner, MD
• Professor Vice Chair
• Program Director
• Otolaryngology/Head Neck Surgery
• Stanford University

2
Application season
3
Why have letters?

• Review of 966 LOR to General surgery program
• 24 were helpful in evaluation process
• Fortune JB. The content and value of letters of
  recommendation in the resident candidate
  evaluative process. Curr Surg 2002

4
Why have letters? The theory.

• Information on past performance which may be a
  good predictor of future success
• Unique features
• Insight into typical performance
• (as opposed to maximal performance)
• Opportunity to explain a less than perfect
  application
• Opportunity to promote students he/she envisions
  as becoming future leaders in field.

5

• Medicine seems to have taken up permanent
  residence along the shores of Lake Wobegon. All
  of the applicants are above averageway above.

All are outstanding
Schneider A. Why you cant trust letters of
recommendation. Chron High Educ 2000
6
Writing letters- The basics

• 1. Introduction
• 2. Relationship to applicant
• Length of time he/she has known subject
• Context of relationship
• Right to view waived
• 3. Describe subjects record,
• Brief summary of subjects academic performance
• Summary of clinical performance
• 4. Subjects traits and performance are
  evaluated and compared to peers ( unique
  features).
• 5. Summary sentence

7
OHNS Words/letter

• 763 letters
• Mean 329 (82-1297)
• Chair of dept (290) vs others (350) (plt.001)

Letters
Messner A, Shimahara E Laryngoscope 2009
8
What we currently do

• Look for Standout adjectives (average 2.6 in OHNS
  letters)
• star, excellent, superb, gifted, extraordinary,
  outstanding, unique, exceptional, unparalleled,
  superstar, finest, exquisite, stellar, sterling,
  phenomenal, ideal, exemplary
• Grindstone adjectives
• Hardworking, resolve, conscientious, dependable,
  meticulous, thorough, diligent, dedicated,
  careful, responsible, reliable, tireless, solid
• Gets along with others, Team player
• Compassionate, Caring, Kind, Empathy
• Intelligent

9
Features of letters

• Leadership potential
• High match rank
• Sense of humor
• Final sentence Intentionally ambiguous
• Recommend him without hesitation vs recommend
  highly vs highest recommendation vs absolute
  highest recommendation
• Strong recommendation vs very strong
  recommendation vs strongest recommendation
• Compare letters from same letter writer

10
Watch for doubt raisers

• Technical skills were average.
• I have discussed alternative career options
• Made an effort to be an effective team member.
• Average fund of knowledge
• His score of 210 accurately reflects his
  knowledge level.

11
Doubt raisers
Doubt Raisers

• It absolutely amazes me how he can work with
  others.
• Hes always trying.
• I assure you that no person would be better for
  the job.

12
Typical Letter analysis beginning end

• Dear Program Director
•  
• I have been asked to submit a letter of
  recommendation on behalf of Jane Doe who has
  waived her right to see this letter. I have known
  Jane for the past 2 years since she first became
  interested in Otolaryngology.
•  
• Jane is a fourth year medical student at X School
  of Medicine. She graduated from Y University in
  2002 with a Bachelor of Arts and History. She
  has achieved several awards during her medical
  school tenure including Outstanding Service in
  April 2006. This is an award elected annually by
  current X medical students to the one third year
  student who contributes significantly to
  improving the lives of fellow students.
  Additionally, Jane obtained honors in pediatrics,
  a high pass in surgery, a high pass in obstetrics
  and gynecology.
•  
• Jane rotated on the Head and Neck Surgery Service
  in the Department of Otolaryngology Head and Neck
  Surgery at S School of Medicine. During her
  rotation Jane performed in an outstanding manner.
  She is a bright and intelligent student who has
  an excellent knowledge of the head and neck
  anatomy and diseases that afflict the head and
  neck.
•  
• She was actively engaged throughout her rotation
  in caring for the head and neck cancer patients
  in the ward. Additionally, she played an active
  role in the operating room where she clearly
  demonstrated an affinity for surgery. She has
  excellent hand-eye coordination and an excellent
  sense of the anatomy. Jane was well liked by
  her peers, the residents and the attending staff.
  She is a pleasant person to work with. She is
  hard working and while somewhat shy Jane is
  clearly well informed and well read.
•  
• In general I was extremely impressed with this
  young student who I feel will make an outstanding
  contribution to any otolaryngology head and neck
  surgery training program.
•  
• As a result, I would like to submit a very strong
  letter of recommendation on behalf of Jane Doe.
  If I can be of further assistance please feel
  free to contact me.
•  
• Sincerely,

13
Typical Letter- ignore duplication

• Dear Program Director
•  
• I have been asked to submit a letter of
  recommendation on behalf of Jane Doe who has
  waived her right to see this letter. I have known
  Jane for the past 2 years since she first became
  interested in Otolaryngology.
•  
• Jane is a fourth year medical student at X School
  of Medicine. She graduated from Y University in
  2002 with a Bachelor of Arts and History. She
  has achieved several awards during her medical
  school tenure including Outstanding Service in
  April 2006. This is an award elected annually by
  current X medical students to the one third year
  student who contributes significantly to
  improving the lives of fellow students.
  Additionally, Jane obtained honors in pediatrics,
  a high pass in surgery, a high pass in obstetrics
  and gynecology.
•  
• Jane rotated on the Head and Neck Surgery Service
  in the Department of Otolaryngology Head and Neck
  Surgery at S School of Medicine. During her
  rotation Jane performed in an outstanding manner.
  She is a bright and intelligent student who has
  an excellent knowledge of the head and neck
  anatomy and diseases that afflict the head and
  neck.
•  
• She was actively engaged throughout her rotation
  in caring for the head and neck cancer patients
  in the ward. Additionally, she played an active
  role in the operating room where she clearly
  demonstrated an affinity for surgery. She has
  excellent hand-eye coordination and an excellent
  sense of the anatomy. Jane was well liked by
  her peers, the residents and the attending staff.
  She is a pleasant person to work with. She is
  hard working and while somewhat shy Jane is
  clearly well informed and well read.
•  
• In general I was extremely impressed with this
  young student who I feel will make an outstanding
  contribution to any otolaryngology head and neck
  surgery training program.
•  
• As a result, I would like to submit a very strong
  letter of recommendation on behalf of Jane Doe.
  If I can be of further assistance please feel
  free to contact me.
•  
• Sincerely,

14
Typical Letter analysis standouts, grindstone,
team player

• Dear Program Director
•  
• I have been asked to submit a letter of
  recommendation on behalf of Jane Doe who has
  waived her right to see this letter. I have known
  Jane for the past 2 years since she first became
  interested in Otolaryngology.
•  
• Jane is a fourth year medical student at X School
  of Medicine. She graduated from Y University in
  2002 with a Bachelor of Arts and History. She
  has achieved several awards during her medical
  school tenure including Outstanding Service in
  April 2006. This is an award elected annually by
  current X medical students to the one third year
  student who contributes significantly to
  improving the lives of fellow students.
  Additionally, Jane obtained honors in pediatrics,
  a high pass in surgery, a high pass in obstetrics
  and gynecology.
•  
• Jane rotated on the Head and Neck Surgery Service
  in the Department of Otolaryngology Head and Neck
  Surgery at S School of Medicine. During her
  rotation Jane performed in an outstanding manner.
  She is a bright and intelligent student who has
  an excellent knowledge of the head and neck
  anatomy and diseases that afflict the head and
  neck.
•  
• She was actively engaged throughout her rotation
  in caring for the head and neck cancer patients
  in the ward. Additionally, she played an active
  role in the operating room where she clearly
  demonstrated an affinity for surgery. She has
  excellent hand-eye coordination and an excellent
  sense of the anatomy. Jane was well liked by
  her peers, the residents and the attending staff.
  She is a pleasant person to work with. She is
  hard working and while somewhat shy Jane is
  clearly well informed and well read.
•  
• In general I was extremely impressed with this
  young student who I feel will make an outstanding
  contribution to any otolaryngology head and neck
  surgery training program.
•  
• As a result, I would like to submit a very strong
  letter of recommendation on behalf of Jane Doe.
  If I can be of further assistance please feel
  free to contact me.
•  
• Sincerely,

15
Typical Letter analysis doubt raisers

• Dear Program Director
•  
• I have been asked to submit a letter of
  recommendation on behalf of Jane Doe who has
  waived her right to see this letter. I have known
  Jane for the past 2 years since she first became
  interested in Otolaryngology.
•  
• Jane is a fourth year medical student at X School
  of Medicine. She graduated from Y University in
  2002 with a Bachelor of Arts and History. She
  has achieved several awards during her medical
  school tenure including Outstanding Service in
  April 2006. This is an award elected annually by
  current X medical students to the one third year
  student who contributes significantly to
  improving the lives of fellow students.
  Additionally, Jane obtained honors in pediatrics,
  a high pass in surgery, a high pass in obstetrics
  and gynecology.
•  
• Jane rotated on the Head and Neck Surgery Service
  in the Department of Otolaryngology Head and Neck
  Surgery at S School of Medicine. During her
  rotation Jane performed in an outstanding manner.
  She is a bright and intelligent student who has
  an excellent knowledge of the head and neck
  anatomy and diseases that afflict the head and
  neck.
•  
• She was actively engaged throughout her rotation
  in caring for the head and neck cancer patients
  in the ward. Additionally, she played an active
  role in the operating room where she clearly
  demonstrated an affinity for surgery. She has
  excellent hand-eye coordination and an excellent
  sense of the anatomy. Jane was well liked by
  her peers, the residents and the attending staff.
  She is a pleasant person to work with. She is
  hard working and while somewhat shy Jane is
  clearly well informed and well read.
•  
• In general I was extremely impressed with this
  young student who I feel will make an outstanding
  contribution to any otolaryngology head and neck
  surgery training program.
•  
• As a result, I would like to submit a very strong
  letter of recommendation on behalf of Jane Doe.
  If I can be of further assistance please feel
  free to contact me.
•  
• Sincerely,

16
Typical Letter

• Dear Program Director
•  
• I have been asked to submit a letter of
  recommendation on behalf of Jane Doe who has
  waived her right to see this letter. I have known
  Jane for the past 2 years since she first became
  interested in Otolaryngology.
•  
• Jane is a fourth year medical student at X School
  of Medicine. She graduated from Y University in
  2002 with a Bachelor of Arts and History. She
  has achieved several awards during her medical
  school tenure including Outstanding Service in
  April 2006. This is an award elected annually by
  current X medical students to the one third year
  student who contributes significantly to
  improving the lives of fellow students.
  Additionally, Jane obtained honors in pediatrics,
  a high pass in surgery, a high pass in obstetrics
  and gynecology.
•  
• Jane rotated on the Head and Neck Surgery Service
  in the Department of Otolaryngology Head and Neck
  Surgery at S School of Medicine. During her
  rotation Jane performed in an outstanding manner.
  She is a bright and intelligent student who has
  an excellent knowledge of the head and neck
  anatomy and diseases that afflict the head and
  neck.
•  
• She was actively engaged throughout her rotation
  in caring for the head and neck cancer patients
  in the ward. Additionally, she played an active
  role in the operating room where she clearly
  demonstrated an affinity for surgery. She has
  excellent hand-eye coordination and an excellent
  sense of the anatomy. Jane was well liked by
  her peers, the residents and the attending staff.
  She is a pleasant person to work with. She is
  hard working and while somewhat shy Jane is
  clearly well informed and well read.
•  
• In general I was extremely impressed with this
  young student who I feel will make an outstanding
  contribution to any otolaryngology head and neck
  surgery training program.
•  
• As a result, I would like to submit a very strong
  letter of recommendation on behalf of Jane Doe.
  If I can be of further assistance please feel
  free to contact me.
•  
• Sincerely,
• NOTE nothing unique, interesting, no comparison
  to peers, no mention leadership potential, no
  humor, no match potential

17
Letter of Minimal Assurance

• I believe that Joe is a strong candidate for a
  residency position and that he will ultimately be
  an excellent otolaryngologist and contributor to
  medicine. He is passionately committed to this
  goal and he is driven by that commitment. He is
  a unique person, and one whom you will enjoy
  meeting and interviewing.
• I would of course welcome a direct phone call
  regarding this letter.
• Thank you for considering Joe for an interview
  position.

18
Current Status

• Average program 250 applications x 4 letters
  1000 letters
• Average letter 329 words
• Time 1 min/letter (without analysis)
• 1000 min 17 hours.
• Result Letters often not used to make 1st cut
• Increased reliance on USMLE scores
• And many not read at all.

19
Standardized letter of recommendation (SLOR)

• Council of Emergency Medicine Residency Directors
  created SLOR
• Successfully used since 1996

20
Comparison of SLOR NLOR

• Part I Compared 20 SLOR and 20 NLOR
• 7- point Likert-type scale
• 4 raters (2 experienced, 2 inexperienced)
• Results
• Interrater reliability of SLOR 0.97, NLOR 0.78
• Average time to interpret SLOR 16 sec, NLOR
  90 sec
• Girzadas DV Jr, Harwood RC, Dearie J, Garrett
  S. Acad Emerg Med. 1998 Nov5(11)1101-4

21
Comparison of SLOR NLOR

• Part I Compared 20 SLOR and 20 NLOR
• 7- point Likert-type scale
• 4 raters (2 experienced, 2 inexperienced)
• Results
• Interrater reliability of SLOR 0.97, NLOR 0.78
• Average time to interpret SLOR 16 sec, NLOR
  90 sec
• Part II 207 SLOR/NLOR pairs
• Results
• 54 given same numeric value, 39 differed by one
  point, 6 differed by 2 points. Overall
  correlation was 0.58.
• Girzadas DV Jr, Harwood RC, Dearie J, Garrett
  S. Acad Emerg Med. 1998 Nov5(11)1101-4

22
EM SLOR

• 173 surveys of PD organization (70 response
  rate)
• Compared with the narrative LOR, please rate
  the SLOR in terms of
• Ability to discriminate differences between
  candidates
• Better 75, Same 18, Worse 7
• Ease of reading and incorporating into ranking
  scheme
• Better 84, Same 12 Worse 4
• Ease of completion
• Better 83, Same 9, Worse 4
• Would you like to continue using the SLOR?
• Yes 90, No 7
• Keim SM, et al. Acad Emerg Med. 1999
  Nov6(11)1141-6

23
Proposed Standardized Letter of Recommendation

24
Standardized LOR
25
Standardized LOR
26
Compare to other medical students
27
Standardized LOR
28
Standardized LOR
29
Standardized LOR
30
Standardized LOR
31
Standardized LOR
32
Standardized LOR
33
Advantages of SLOR

• More consistent information
• Less duplication- of contents of application and
  letters
• Less time to complete interpret
• Information less dependent on letter writer (more
  fair)
• (A superb letter writer may portray a good
  candidate in stratospheric terms, whereas a
  mediocre letter writer may portray a superb
  candidate less positively)
• Still have opportunity to tell about unique
  features of applicant

34
Disadvantages of SLOR

• Potential loss of nuances
• Much harder to make the average student look
  stupendous
• Potential attending MD guilt
• Dont get to show off fancy Department stationary
• But OK to add Logo, or put form on stationary

35
Next Steps

• Open for comments until February 1, 2012
• Send to amessner_at_ohns.stanford.edu
• OPDO council will review all comments and revise
  proposed SLOR
• Assuming response is positive
• Will post SLOR on SUO-AADO website
• Email distribution to PDs and Chairs
• Need Program director assistance to get the word
  out to faculty and applicants
• Investigate information re SLOR in Oto journal
• Start using in spring/summer 2012
• Survey SUO/AADO/PD re use, then revise.

36
Conclusions

• Traditional letters have the potential to help
  with identifying top residency candidates but
  often do not.
• A SLOR has the potential to increase the value of
  LOR while saving time.
• Please send all comments to
• amessner_at_ohns.stanford.edu