The Royal College of Pathologists the overall assessment system

1
The Royal College of Pathologists the overall
assessment system

• Dr. Trevor Gray
• Director of Examinations and Assessment

2
The Assessment System

• Overview of PMETB and assessment strategy
• How WBA fits with other assessments
• Advantages and disadvantages of WBA compared with
  other assessments
• Local organisation of WBA and how it is
  administered by the College
• Future prospects

3
Overview

• PMETB (and the MMC process) has laid down
  significant challenges to medical postgraduate
  training (elegantly dissected in the Tooke
  report)
• PMETB principle is that training and assessment
  should be much more integrated an assessment
  system
• Workplace-based assessment is an essential
  element of the assessment of trainees undertaking
  specialist medical training
• The specialty curriculum, training, exams and
  assessments are core College business (in the
  charter)
• The Colleges aim is to develop a validated
  assessment system which is fit for purpose

4
PMETB strategy on assessment

• PMETB principles for assessment 2004
• Nine principles for assessment systems
• Now backed up with technical documentation
• Principles based on overarching strategy
• Workplace-based assessment
• Examinations of knowledge
• Examinations of Clinical Skills
• (Workplace-based assessment PMETB, Jan 2005)
• Specialty curriculum and assessment system now
  approved by PMETB
• (i.e. match strategy)

5
Workplace-based Assessment

• Assessment based on performance in the workplace
  and measured there
• Assessed locally by supervisors as well as staff
  of all grades in contact with the trainee
• Assessment occurs during normal work immediately
  followed by feedback so trainee receives
  continuous feedback on their routine performance
• Results put into trainee portfolio for their
  annual review and hence important for trainee
  development and ultimately their employment
  contract

6
College Assessment Systemapproved by PMETB

• Workplace-based assessment
• Year 1 assessment
• MRCPath Part 1 (test of knowledge)
• MRCPath Part 2 (test of skills and applied
  knowledge)
• Feedback on regular basis (WBA)
• Educational supervisors report
• ARCP (Annual Review of Competency Progression)

7
The College Assessment system Millers pyramid
WBA - MSF
Does
WBA e.g. Mini-CEX, DOPS
Shows
MRCPath Part 2 Practical
Knows how
MRCPath Part 1
Knows
8
Value of multiple assessments

• Key is several modes of assessment
  triangulation
• A single exam system (MRCPath) does not provide
  adequate coverage of clinical practice
• WBA is also not sufficient in itself

(Assessment System Guidance PMETB)
9
Examinations and Assessment

• Examinations are simply one particular form of
  assessment
• Assessments are
• formative (involve feedback and are intended to
  guide training) or
• summative (assess whether reach given standard)
  e.g. driving test, MRCPath
• All assessments should be blueprinted to the
  curriculum to demonstrate that the most
  appropriate method is being used

10
How Workplace-based Assessment fits with MRCPath

• Workplace-based assessment depends on assessment
  by local staff, examinations rely on assessment
  by those who have had no contact with the trainee
  
• Examinations provide an independent assessment of
  knowledge and practical skills to national
  standards
• Workplace-based assessment provides a local
  insight into candidates performance in routine
  tasks and interaction with staff and patients

11
How Workplace-based Assessment fits with MRCPath

• Workplace-based assessment is a local process
  governed by deaneries albeit to PMETB standards
• Examinations are organised nationally by the
  College
• Comments, complaints and appeals for WBA go to
  deanery, for exams go to College

12
Workplace-based Assessment Advantages

• Measurement occurs in more authentic environment
  than the simulated one of the examination room
• Trainees can take assessment in their own time
  and are not limited to exam timetable
• The assessment tools are set up to provide
  extensive feedback to the candidate (formative)
• Use of computerised forms makes them easy to
  record

13
Workplace-based Assessment Disadvantages

• To avoid time-consuming busy assessment forms,
  the assessments are often simplified if not
  simplistic
• A robust rating can only occur if there are a
  number of independent observers which can be
  difficult in small specialties
• They have only been validated in a number of
  clinical settings but not in all pathology
  settings
• Computerised forms can over-simplify complex
  judgements and may not reward excellence
  appropriately

14
Year 1 Assessment (OSPE)

• Developed to provide evidence of progression in
  run-through training in the absence of further
  recruitment and selection
• Designed to assess progress in first year and be
  taken after 7 months training
• It is a test to indicate the capability and
  potential of a trainee through a test of applied
  knowledge and skills relevant to the year 1
  curriculum

15
Training and Assessment Plan
ARCP
F1
F2
ST1
ST2
ST3
ST4
ST5
Foundation
Stage A
Stage B
Stage D
Stage C
WBA
Workplace-based assessment
MSF
Y1A
MRCPath CCT
Part 1
Part 2
16
The training environment

• Workplace-based assessment utilises the skills of
  a wide variety of professional groups as
  assessors medical, nursing, BMS, clinical
  scientist, ancillary staff
• The College cannot train or quality assure all
  assessors
• Training centres are recommended to develop a
  training environment where those with the skills
  needed for assessment are recognised and training
  is cascaded from those attending events such as
  this

17
Workplace-based Assessments and the College

• WBA is part of assessment system for Specialist
  Trainees from 1st August 2007
• Assessments need to be undertaken from early in
  training and recorded for ARCP
• 2007 Launch of tools and paper-based assessment
  forms
• College is researching web-based systems similar
  to the ones used in foundation

18
Workplace-based Assessments and the College

• April 2008 pilot web-based system
• By August 2008 launch of web-based system
• Use data generated by all assessments including
  MSF and OSPE for evaluation of assessment
  strategy
• Ensure compliance with MMC Gold Guide and planned
  ARCP

19
Vision for College Assessment system

• College will have integrated specialty curriculum
  and assessment system endorsed by PMETB
• College will have fully validated workplace-based
  assessment tools delivering feedback on progress
  to trainees throughout course
• Valid and reliable tests will underpin MRCPath
  examination Part 1 test of knowledge, Part 2
  test of applied knowledge and skills

20
Factors affecting future assessment strategy

• Selection into specialty may not be an annual
  process in future compromising ability to deliver
  year 1 assessment (except maybe in
  Histopathology)
• Multiple recruitment may require more flexibility
  in arrangements for WBA
• Selection into specialty may involve a national
  set of tests (Tooke and Academy) displacing Year
  1 assessment

21
Factors affecting future assessment strategy

• Workplace-based assessment is likely to continue
  but its use must be evidence-based
• Workplace-based assessment is not cast in stone
  work in progress and subject to improvement
  following evaluation
• Place of individual components of assessment in
  MRCPath, Year 1 assessment and workplace-based
  assessment may change

22
Summary

• The Colleges education and training, assessment
  and examinations departments have worked hard to
  produce curricula and assessments acceptable to
  PMETB
• This training day is the first for
  workplace-based assessment and is intended to
  assist training centres to develop assessment
  locally
• Workplace-based assessment will be subject to a
  full programme of validation and optimisation to
  ensure fitness for purpose
• The College is looking to source a web-based
  system to support workplace-based assessment

23
Glossary

• ARCP Annual Review of Competence Progression
• CbD Case-based Discussion
• DOPS Direct observation of practical skills
• ECE Evaluation of clinical events
• MSF Multi-source feedback (ePATH-SPRAT)
• Mini-CEX Mini clinical evaluation exercise
• OSPE Objective structured practical examination
• PMETB Postgraduate medical education and training
  board
• WBA Workplace-based assessment