nMRCGP The Clinical Skills Assessment : an evolving process

1
nMRCGPThe Clinical Skills Assessment an
evolving process

• Prepared by the CSA Operations Group
• nMRCGP

2
Health Warning

• Contemporary
• as of January 2007

3
(No Transcript)
4
Principles of Assessment (from PMETB)

• Principle 1
• The assessment system must be fit for a range of
  purposes
• Principle 2
• The content of the assessment will be based on
  curricula for postgraduate training which
  themselves are referenced to all of the areas of
  Good Medical Practice
• Principle 3
• The methods used within the programme will be
  selected in the light of the purpose and content
  of that component of the assessment framework.
• Principle 4
• The methods used to set standards for
  classification of trainees performance /
  competence must be transparent and in the public
  domain

5
Principles of Assessment (from PMETB) continued

• Principle 5
• Assessments must provide relevant feedback
• Principle 6
• Assessors / examiners will be recruited against
  criteria for performing the tasks they undertake
• Principle 7
• There will be Lay input in the development of
  assessment
• Principle 8
• Documentation will be standardised and accessible
  nationally.
• Principle 9
• There will be resources sufficient to support
  assessment

6
The MRCGP Curriculum Statements (Principle 2)

• Where to find them
• RCGP website
• What are they?
• Series of statements, each covering different
  clinical and practice management areas, based on
  European Academy of Teachers in General Practice
  (EURACT) framework and Good Medical Practice (GMC
  document)
• Written by a variety of experts in their field,
  coordinated by RCGP

7
The MRCGP Curriculum Statements (Principle 2)
continued

• How they are being used
• Curriculum statements subdivided into intended
  learning outcomes. Cases linked to specified
  learning outcomes within specified curriculum
  statements.
• This enables sampling from across the curriculum,
  as cases can be mapped to the curriculum
  statements (or nMRCGP blueprint)

8
Components of the nMRCGP

• The Applied Knowledge Test
• Computer-based test of knowledge using
  multiple-choice questions, completed on-screen in
  dedicated test centres.
• The Workplace Based Assessment
• Formative and summative variety of measures
  based on a series of reviews electronic
  portfolio. Tests trainee in his/her place of
  work, doing what he/she actually does.
• The Clinical Skills Assessment
• Clinical consulting skills examination, based on
  cases from general practice, with role players as
  patients, and experienced assessors provides a
  pre-determined, standardised level of challenge
  to candidates.

9
Why a Clinical Skills Assessment?

• Criticism of current MRCGP that there is no
  clinical consulting skills component
• Provides external validation / triangulation with
  the other testing methods used
• Using simulated patients is a validated and
  reliable method for testing clinical skills, so
  long as quality assurance of case production,
  role player and assessor training is carried out.
• Able to offer a standardised, pre-determined
  level of challenge to candidates and to vary this
  level of challenge as needed by the assessment
  requirements

10
Purpose of the CSA

• An assessment of a doctors ability to integrate
  and apply appropriate clinical, professional,
  communication and practical skills in general
  practice
• Integrative skills assessment - tests a doctors
  abilities to gather information and apply learned
  understanding of disease processes and
  person-centred care appropriately in a
  standardised context, making evidence-based
  decisions, and communicating effectively with
  patients and colleagues.

11
Series of Integrated Developments

• Curriculum statements form the basis for the
  nMRCGP blueprint
• Intended learning outcomes - track from
  curriculum statements to cases via CSA blueprint
  and case selection blueprint
• Competency progression
• CSA Blueprint based on nMRCGP blueprint, those
  sections that can be tested by this methodology

12
Series of Integrated Developments continued

• Case writing proforma that guides case writers
  through complexities of linking their case,
  focussing on the nub of the case, and writing a
  marking schedule that reflects this nub.
• Linking the case with searchable keywords so that
  a defined selection of cases can be found for
  each time the assessment is run.

13
CSA Blueprint derived from the Curriculum
14
Case Selection Blueprint
15
How does the CSA differ from the Simulated
Surgery?

• Not just a test of communication skills in a
  clinical setting
• Based on the nMRCGP blueprint, and samples across
  this blueprint.
• Will be taken by many more candidates (3,000 -
  4,000 per year)
• Assesses integrative clinical skills in primary
  care settings

16
What is the CSA likely to look like?

• Starts October 2007
• Temporary assessment centre to be used initially
• Dedicated assessment centre within new College
  build planned within next 3-5 years
• Will take place for a number of weeks, several
  times a year
• Will use multiple circuits
• Candidate stays in surgery and patient and
  examiner move around circuit

17
What is the CSA likely to look like? continued

• Will consist mostly of simulated patient cases.
• 13 stations, probably each of 10 minutes
• Marks collected by Opscan techniques
• Some triangulation with Workplace Based
  Assessment competencies
• Stations picked from intended learning outcomes
  (ILOs) across the nMRCGP blueprint with clear
  derivation

18
(No Transcript)
19
The Marking Schedule

• Each case is marked in 3 domains
• Data gathering, examination and clinical
  assessment skills
• Clinical management skills
• Interpersonal skills
• All domains have equal weighting
• 4 grades awardable
• Clear pass
• Marginal pass
• Marginal fail
• Clear fail

20
The Marking Schedule continued

• Assessor uses word pictures to help decide grade
  for each domain, then uses this information to
  make a judgement on the grade for the case
  overall (4 decisions)
• Feedback to candidates
• Serious concerns box

21
Three domains for each case
22
Grades

• Clear Pass (CP)
• Marginal Pass (MP)
• Marginal Fail (MF)
• Clear Fail (CF)

23
How the CSA is aiming to meet PMETB assessment
criteria

• PMETB ASSESSMENT CRITERIA
• 1. This methodology is judged to be the best way
  to test Clinical Skills in general practice
  currently.
• 2. Cases are based on the nMRCGP curriculum.
• 3. Assessment methodology chosen is fit for
  purpose validated and reliable, elsewhere and
  our main pilot.
• 4. Standard setting will be transparent and in
  the public domain with wide consultation.
• 5. Feedback will be given to all candidates.

24
How the CSA is aiming to meet PMETB assessment
criteria continued

• PMETB ASSESSMENT CRITERIA
• 6. Recruitment of assessors will be on ability
  to rank order, mark reliably, knowledge.
• 7. Lay input has been consistently sought.
• 8. Documentation will be accessible nationally
  through the College website and publication in
  peer reviewed journals and GP rag mags.
• 9. Resources? Continually under review

25
Acknowledgements

• This presentation written by
• Kamila Hawthorne
• Neil Munro
• On behalf of the CSA Operations group, nMRCGP
• Updated January 2007

26
Members of the CSA Operations Group

• Neil Munro GP, MRCGP examiner, chair of the
  Operations Group
• Mei Ling Denney GP, MRCGP examiner and course
  organiser, deputy convenor MRCGP Simulated
  Surgery
• Kamila Hawthorne GP and trainer, MRCGP examiner,
  Senior Lecturer in General Practice, Cardiff.
• Fiona Patterson Professor of Work Psychology at
  City University
• Sue Rendel GP, MRCGP examiner and Convenor MRCGP
  Simulated Surgery
• Amar Rughani GP, MRCGP examiner, nMRCGP
  blueprint guardian
• David Sales GP, Assessment Fellow, RCGP
• Richard Wakeford MA Cpsychol, FRCGP, Assessment
  Consultant, University of Cambridge.
• Faye Geoghegan nMRCGP Project Manager