Item Dependency in an Objective Structured Clinical Examination

1
Item Dependency in an Objective Structured
Clinical Examination

• Cherdsak Iramaneerat
• Carol M. Myford
• Rachel Yudkowsky

University of Illinois at Chicago
2
Objective Structured Clinical Examination

• Objective structured clinical examination (OSCE)
• An assessment approach used in medical education
  in which the clinical competence of residents is
  evaluated using multiple stations of standardized
  clinical tasks
• Standardized patients (SP)
• Lay persons trained to portray a scripted patient
  presentation in a standardized fashion

3
Conditional Item Independence

• A basic assumption of the Rasch model
• After accounting for the latent trait, item
  responses on a test are independent.

• Item dependence can lead to inaccurate estimation
  of item parameters, test statistics, and resident
  competency.
• Item dependence can lead to overestimation of
  reliability and test information.

4
Items in OSCE

• In each OSCE station, items are linked to the
  same clinical task and are rated by the same SP.
• A residents level of performance on one item may
  be dependent on his/her level of performance on
  other items in the same OSCE station.

5
Purposes

• To check for the existence of item dependency in
  an OSCE
• To outline an alternative approach for analyzing
  rating data using a MFRM model to ameliorate the
  problem of item dependency
• To compare reliability estimates, parameter
  estimates, and fit statistics obtained from MFRM
  analyses when local dependence is present/absent

6
Participants

• 79 residents from one Midwestern medical school
• 68 internal medicine residents
• 66 Male
• 34 Female
• 11 family medicine residents
• 45 Male
• 55 Female

7
Tasks (OSCE Stations)

• A communication skills assessment
• Six OSCE stations of simulated clinical scenarios
  
• Patient education
• Informed consent
• Treatment refusal
• Elderly abuse
• Giving bad news
• Physical examination

8
Rating Scale

• A modification of the communication skills rating
  form of the American Board of Internal Medicine
• 18 items asking for agreement ratings
• Five-point Likert Scale
• 1 (Strongly disagree) to 5 (Strongly agree)

9
Items

• 1. You greeted me warmly...
• 2. You were friendly...
• 3. You treated me like we were on the same
  level...
• 4. You let me tell my story without
  interruption...
• 5. You were truthful...
• 6. You never ignored what I had to say...
• 7. You discussed options with me...
• 8. You made sure that I understood the options...
• 9. You allowed me to make my own decision...
• 10. You encouraged me to ask questions...
• 11. You were patient...
• 12. You never avoided my questions...
• 13. You clearly explained the problem...
• 14. You clearly explained what should be
  expected...
• 15. You used plain language, not medical
  jargon...
• 16. You were careful in approaching sensitive
  issues...
• 17. You displayed a positive attitude...
• 18. I will choose this physician as my personal
  physician.

10
Analyses

• Pnijk Probability of resident n receiving a
  rating of k on item i in station j
• Pnij(k-1) Probability of resident n receiving a
  rating of k-1 on item i in station j
• Bn Level of communication competence of
  resident n
• Di Difficulty of item i
• Cj Difficulty of OSCE station j
• Fik Difficulty of receiving a rating of k
  relative to k-1 for item i

11
Local Independence

• Yens Q3 statistic (Yen, 1984, 1993)
• the correlation of the residuals for a pair of
  items after partialling out the latent trait
  estimate
• Fishers Z approach (Shen, 1996)
• A modification of Yens Q3 statistic
• Adjusting residuals by the accuracy of the
  resident communication competence measure
• Establishing a practical significance level

12
Fishers Z Index

• Calculate the standardized residuals for each
  rating of resident n on item i
• dni (observed rating expected rating)/SEn
• Correlate the standardized residuals for all
  pairs of item i, j in each OSCE station
• Compute Fishers Z statistic

13
Alternative Approach

• Treating each OSCE station as a scoring unit
• Average the ratings from all items in one station
  and multiply by 10 to produce a station score.
• Integers ranging from 10 (poor performance) to 50
  (excellent performance)

14
Alternative Analysis

• Pnjk Probability of resident n receiving a
  station score of k in station j
• Pnj(k-1) Probability of resident n receiving a
  station score of k-1 in station j
• Bn Level of communication competence of
  resident n
• Cj Difficulty of OSCE station j
• Fjk Difficulty of receiving a station score
  of k relative to k-1 for station j

15
Item Dependency
16
Resident Separation Reliability

• Using items as scoring units
• A resident separation reliability 0.94
• Using stations as scoring units
• A resident separation reliability 0.74

17
Resident Communication Competence Measures
18
Resident Communication Competence Measures
19
Misfitting Residents
20
Station Difficulty Measures
21
Misfitting Stations
22
Item Dependency in MFRM Analyses

• A violation of a basic assumption of the model
• Results
• Overestimation of separation reliability
  estimates
• Poorer fit of resident communication competency
  measures (according to standardized fit
  statistics)
• Poorer fit of station difficulty measures
  (according to both standardized and
  unstandardized fit statistics)

23
Suggestions

• When conducting a MFRM analysis of a data set
  that has items linked to the same task or raters
• Check for the violation of local independence
  assumption
• If item dependency is a problem combine ratings
  from multiple items into a station score
• Alleviate item dependency problem
• Loss of information and decrease in resident
  separation reliability

24
Questions and Comments

• Cherdsak Iramaneerat
• Department of Educational Psychology
• College of Education
• University of Illinois at Chicago
• cirama1_at_uic.edu