Introduction to Workplacebased Assessments

1
Introduction to Workplace-based Assessments

• Dr Hani Zakhour
• Director of Training and Educational Standards
• The Royal College of Pathologists
• 19 October 2007

2
History I

• PMETB
• Standards for postgraduate training and
  assessment were published in 2005
• Training and assessment should be much more
  integrated an assessment system
• Clear view of PMETB and GMC (joint paper 2005)
  that these should include continuous monitoring
  not just periodic College examinations
• Workplace-based assessment PMETB Jan 2005

3
History II

• RCPath
• College has been fully engaged with reforms in
  Medical Education
• January 2005, College Council witnessed a
  presentation of RCP pilot of WBA, tools were
  commended by Council
• Director of Examinations and Assessment May 2005
• WBA Manager appointed October 2005
• WBA Working group March 2006

4
PMETB principle documents

• Standards for Curricula
• Janet Grant Chair Curriculum Sub-Committee Sue
  Fox, Namita Kumar, Ewen Sim, Members Curriculum
  Sub-Committee Revised March 2005
• Principles for an assessment system for
  postgraduate medical Training
• Lesley Southgate Chair Assessment Committee
  Janet Grant Chair Curriculum Sub-Committee 14
  September 2004

5
Test methods as part of the blueprint

• Test methods for assessment programmes can only
  be selected once the purpose and content are
  established
• What is the best trade-off between fidelity and
  efficiency.
• What is the programme for writing and refining
  test material?
• How are marking schedules developed and reviewed
  where relevant?
• Is new test material piloted before it is used?

6
How will the assessments be collated?

• Assessment tools will build up a profile as the
  year progresses
• Evidence of a response to feedback will be shown
  by subsequent assessment of the same area
• Overall record will form the basis for a
  submission to the deanery as evidence of
  satisfactory completion of the year ARCP

7
Standard Setting

• It is only possible to set a sensible standard if
  the purpose of the assessment is clearly set out.
• The most important thing about a standard is who
  sets it
• It is the standard expected of trainee at the end
  of a particular stage
• Understanding principles of the specialty and
  practices under direct supervision (Stage A)
• In-depth knowledge and understanding of
  principles and practices of the specialty (stage
  D)

8
Standardisation

• Commonality in the standards that are required of
  trainees.
• Grade descriptor, intended as a guide
• Grade scale
• 1,2 - Below expectation
• 3 - Borderline
• 4 - Meets expectations
• 5,6 - Above expectations

9
Tools

• Tools taken from F2 programme
• CbD Case-based Discussion (Dame Lesley Southgate)
• used to enable the documenting of conversations
  about, and presentations of, cases by trainees
• DOPS Directly Observed Procedural Skills (RCP)
• a method, similar to the Mini-CEX that has been
  designed specifically for the assessment of
  practical skills.
• MiniCEX Mini Clinical Evaluation Exercise (John
  Norcini)
• designed to provide feedback on skills essential
  to the provision of good clinical care by
  observing an actual clinical encounter

10
Tools

• Tools taken from F2 programme
• MSF Multi Source Feedback - ePath SPRAT (Dr
  Helena Davies)
• provides feedback from a range of co-workers
  across the domains of Good Medical Practice.
  These can be mapped to the core objectives of the
  curriculum
• ECE Evaluation of Clinical Events (WBA Group
  Histopathology)
• It replaces MiniCEX for Histopathologists

11
Purpose of Assessment Tools

• To assess what cannot be assessed by an
  examination.
• CbD
• Indicates trainees capability, potential and
  behaviour
• Provides immediate feedback on trainees learning
  needs highlighting strengths and identifying
  weaknesses
• DOPs
• Indicates trainees acquisition of practical
  skills and understanding of equipment
• Indicates trainee interacts appropriately with
  patients
• MiniCEX
• Indicates trainees acquisition of clinical
  skills

12
Pitfalls of WBA

• To avoid time-consuming busy assessment forms,
• the assessments are often simplified if not
  simplistic
• computerised forms can over-simplify complex
  judgements
• A robust rating can only occur if there are a
  number of independent observers which can be
  difficult in small specialties
• They have only been validated in a number of
  clinical settings but not in all pathology
  settings

13
Beware!

• Test methods can only be selected once the
  purpose and content are established
• What is the best trade-off between fidelity and
  efficiency.
• What is the programme for writing and refining
  test material?
• How are marking schedules developed and reviewed
  where relevant?
• Is new test material piloted before it is used?

14
Principles of conduct for WBA

• Assessment based on performance in the workplace
  and measured there
• Observations can be combined from a number of
  assessors to provide a robust rating for the
  candidate
• Assessment occurs during normal work immediately
  followed by feedback on their routine performance
• WBA is not cast in stone work in progress and
  subject to improvement following evaluation

15
Who can be an assessor?

• Assessors can be consultants who are supervisors
  as well as staff of all grades in contact with
  the trainee
• Clinical Scientists
• Nurses
• BMS
• Senior trainees
• Autopsy room staff and others
• No need for prior approval by RCPath
• Need to follow WBA guidelines and are willing to
  engage in the process

16
Who is it for?

• Mandatory for all StRs (Specialty Registrars)
  appointed from August 2007 to one of the
  following
• Specialty training programme with a National
  Training Number (NTN)
• Fixed Term Specialty Training Appointment (FTSTA)
• Locum Appointment for Training (LAT)

17
Feedback

• One of the most important aspects of any
  assessment programme is the effect it has on
  learning and preparation for individuals and the
  profession.
• How is feedback given to individuals and groups
  about the outcomes of an assessment?
• Finally the mechanisms for appeal about the
  outcome should be documented and reported.

18
(No Transcript)
19
Present and future

• 2007 Launch of tools and paper-based assessment
  forms.
• 2008 Launch of web-based system
• Use data generated by all assessments including
  MSF and OSPE for validation
• Ensure compliance with MMC Gold Guide and planned
  ARCP

20
Acknowledgments

• Sandra Dewar
• Dr Trevor Gray
• Dr Mary Thompson

21
Tribute

• Professor Dame Lesley Southgate
• Professor Janet Grant

22
(No Transcript)
23
(No Transcript)