Portfolio Assessment in Clerkship

1
Portfolio Assessment in Clerkship

• Michelle Gibson - Geriatrics
• (thanks to Chris Frank and Melissa Andrew too)

2
Objective

• Participants will discuss the use of portfolios
  in assessing medical trainees (after a brief
  review of some basic principles)

3
Background

• Change to the Internal Medicine clerkship in
  January 2007
• 4 weeks to 6 weeks
• Opportune time to review the assessment process
  in geriatrics
• Previously, 8 assessment cards filled out by MD
  supervisors

4
Challenges with old system

• Cards were often filled out the last week of the
  rotation
• Depended solely on supervisor impression
• No direct observation was incorporated
• Students were assessed using only one modality

5
Reasons for change

• Desire for more formative assessment
  opportunities
• Desire for more robust assessment
• (Need to meet course requirements!)

6
Formative assessment

• Assessment that occurs during instruction to
  provide feedback to teachers and students
• Describes needs for future learning
• Assessment for learning

7
Summative Assessment

• Assessment that occurs at the end of an
  instructional unit to document student learning
• Certifies learning
• Assessment of learning
• Is this portfolio formative or summative?

8
High Quality Assessment

• Clear appropriate learning targets
• Appropriateness of assessment methods
• Validity
• Reliability
• Fairness
• Positive consequences
• Alignment
• Practicality efficiency

9
Reliability Validity

• Reliability concerned with the consistency,
  stability, and dependability of the scores
• Validity the appropriateness of the inferences,
  uses, and consequences of the assessment
• Is it assessing what you want it to assess?

10
Portfolio

• A systematic collection of student products to
  assess progress
• Structured portfolio
• The assessor determines the mandatory content of
  the portfolio

11
Literature review

• Assessment of housestaff in clinical rotations
• Portfolio literature reviewed
• Specific instruments examined
• Portfolio components were selected to provide
  multiple methods of assessment for each clerkship
  objective

12
Components by Methodology
13
Mini-CEX
14
TAB
15
Card
16
Medication Review - written

• Based on an actual patient, preferably one for
  whom the student has responsibility
• brief summary of patient,
• meds he/she is taking including whether or not
  they are appropriate
• suggestion of other medications he/she should be
  taking

17
Self-assessment of write-up

• H P from early in rotation
• Students asked to read and identify
• 2 things done well
• 2 things that could be improved
• Reviewed with undergrad coordinator on the last
  day

18
Components CPE Categories
19
Process

• Orientation with students on Day 1
• Team, faculty, residents are oriented to the
  portfolio
• Students meet with undergrad coordinator at
  mid-term, review progress, identify any gaps,
  etc.
• Last day - review portfolio with students,
  completion of CPE

20
Evaluation of Pilot

• Focus on acceptability
• Students and assessors
• Student evaluations ongoing - not seen by the
  undergrad coordinator until collated in April
  2008
• Will redo faculty portion in December 2007

21
Challenges noted - 1st time

• First rotation ever for clerks
• Time spent learning how to be a clerk
• Couldnt assemble half the portfolio in 3 weeks
• Exam stress detracted from our learning
• Supervisors were absent more than usual
• all except exam stress have resolved

22
Strengths noted

• Multidisciplinary team members provided
  high-quality feedback
• Direct observation occurred
• Forms were felt to be user-friendly, acceptable
• Students appreciated discussing their medication
  review and being observed

23
Assessor ratings of forms

• TAB
• 5/5 for ease of use 5/5 for usefulness
• Mini-CEX
• 5/5 for ease of use 4/5 for usefulness
• Time spent by assessor 13.5 minutes
• Encounter card
• 4.5/5 for ease of use 4.5/5 for usefulness
• Time spent by assessor 15 minutes

24
My time?

• 5-10 minutes on day 1 for orientation
• Plan to develop talking power point
• 15 min per student at the end of week 3
• Mid-term review
• Could be any faculty
• 15 min per student on the last day
• Final evaluation
• Could be any faculty

25
My impressions

• Students receive more feedback in a timely
  fashion
• Direct observation doesnt take long, and is very
  valuable
• TAB form is very useful, and team members are
  pleased to be filling them out
• Mid-term meetings are very useful

26
Future directions

• Continue with student and assessor feedback
• Incorporate other tools?
• Peer assessment?
• Patient assessment?
• Study it more
• Implement it for residents

27
Factors to note

• Small unit, discrete group of nursing and other
  staff
• Small division
• Very workable for us but for others?

28
Questions to consider

• Is this high quality assessment? How could it be
  improved?
• Can it be extended into other areas?
• Is there a role for a whole clerkship
  portfolio? Pre-clerkship portfolio?
• Residents non-MDs as assessors?
• Electronic?

29
Comments about our students

• .. Is very attentive he listens well to pt.
  staff, then goes off evaluates what hes been
  told
• excellent verbal communication observed with all
  team members
• better H P than most residents