Closing the loop: Assessing the effectiveness of competency measures

1
Closing the loopAssessing the effectiveness of
competency measures

• David Bienenfeld, M.D.

2
Outline

• Why should we measure?
• What do we measure?
• How can we measure?

3
The competency context

• External agencies
• ACGME general competencies
• Specialty RRCs specific competencies
• ABMS certification
• Hospitals and practices
• Courts
• Internal processes
• Self-evaluation
• Performance evaluation
• Advancement criteria
• Program effectiveness evaluation

4
Construct validity
Content validity
Predictive validity
Concurrent validity
5
Construct validity
The degree to which the test results measure the
trainees ability to perform the required task
6
Content validity
The degree to which completion of the required
task reflects the objective of the program
7
Predictive validity
The ability of the test to predict the likelihood
that the trainee will be able to perform a task
in the future
8
Concurrent validity
The degree to which a test measures skills
possessed by those who have a degree of expertise
that is not itself directly measurable
9
Wright State University Psychiatry skills
competencies

• 65 specific skills
• Done / not done
• Developed over 2 years by Residency Training
  Committee
• Implemented 1998, evaluated 2002

10
Process evaluation
11
Resident faculty survey

• How easy has it been to determine what is
  expected for each of the requirements?
• How easy has it been to obtain/provide patients
  or procedures?
• What influence have the requirements had on
  clarifying training objectives?
• To what extent have the requirements affected
  feedback?
• How well do the criteria reflect the educational
  objectives of the program?

12
Ease of determining expectations
13
Ease of providing/obtaining patients
14
Clarification of training objectives
15
Effect on performance feedback
16
Predictive validity

• Survey
• Recent graduates
• Medical directors of AMBHA and best hospitals
• Reflection of current practice
• Reflection of future needs
• Identify over- or under-emphasis

17
Reflection of currently needed skills
18
Reflection of skills needed in the future
19
Other means of evaluation

• Correlation with ABMS performance
• Correlation with patient outcomes
• Clinical
• Satisfaction
• Comparison with master practitioners
• .Your ideas?