Workplace Based Assessment The story of a journey

1
Workplace Based AssessmentThe story of a journey
2
Old v new MRCGP

• PMETB requires that all Royal Colleges meet their
  criteria for assessments for licensing.
• Fundamental redevelopment of the licensing for
  all the Royal Colleges
• RCGP only college with dual systems
• SA for CCT
• MRCGP as mark of excellence.

3
PMETB

• Required that there should be curricula for
  specialty training
• Assessments should map to the curriculum and Good
  Medical Practice
• Assessment systems should ensure that trainees
  are competent for independent practice and
  licensing CCT
• Assessment methodology should meet the 9 PMETB
  principles of assessment.

4
PMETB Principles of Assessment
5
PMETB Assessment Principles

• The assessment system should be fit for a range
  of purposes
• The content of assessments should be based on
  curricula and referenced to all areas of Good
  Medical Practice
• The methods used in the programme will be
  selected on the basis of the purpose and content
  of that component of the assessment framework
• The methods used to set standards should be
  transparent and in the public domain
• Assessments should provide relevant feedback
• Assessors/examiners will be recruited against
  criteria for performing the tasks they undertake
• There will be lay input into development of
  assessment
• Documentation will be standardised and accessible
  nationally
• There will be resources sufficient to support
  assessment

6
(No Transcript)
7
2003/04 Working groups set up

• Aim was to review the purpose WPBA
• Determine the most appropriate methodology
• Develop the tools
• Prepare the submission for PMETB to approve.

8
Workplace Based Assessment

• The evaluation of a doctors progress over time,
  in their performance in those areas of
  professional practice best tested in the
  workplace
• FORMATIVE
• Summative judgment made by panels informed by
  evidence from WPBA

9
Millers Pyramid
10
Psychomotor skills
11
RELATIONSHIP
DIAGNOSTICS
Professionalism
MANAGEMENT
12
Where do the competencies come from?

• Domains of GP Curriculum
• EURACT (European Academy of Teachers in General
  Practice)
• GMC Good Medical Practice

13
Some of the tools were there from foundation and
summative assessment.

• Video (COT)
• Audit
• Significant event analysis (SEA)
• Referral letter review
• Multi-source feedback (MSF)
• Patient satisfaction questionnaire (PSQ)
• Case based discussion (CBD)
• Mini CEX
• DOPs

14
Tools to be piloted

• Video
• Audit
• Significant event analysis (SEA)
• Referral letter review
• Multi-source feedback (MSF)
• Patient satisfaction questionnaire (PSQ)

• Combination SA/MRCGP video tools
• SA marking schedule but continuous rating.
• Proforma allows reflection
• Review of 10 consecutive letters
• 2 domain questionnaire using both clinical and
  non-clinical colleagues
• Empathy questionnaire (CARE)
• SA summative assessment

15
2005 External tools pilot

• Reviewed the tools
• Which would be fit for high stakes assessments.
• Originally
• PSQ
• MSF

16
Mapping WPBA to PMETB Principles

• The assessment system should be fit for a range
  of purposes
• Tools in the box should be fit for purpose and
  test against the competency framework.

17
6 tools in the box

• CBD
• Mini CEX
• COT
• DOPs
• MSF
• PSQ
• Each tool covers different competencies with some
  overlap.

18
The tools

• External review suggests PSQ MSF need revision
• COTs CBDs Mini CEX DOPs few changes but OK
• Under used is Naturally occurring evidence NOE
  needed for a balanced portfolio.

19
Principle 2

• The content of assessments should be based on
  curricula and referenced to all areas of Good
  Medical Practice
• Tools in the box should be fit for purpose and
  test against the competency framework.
• ?

20
Principle 3

• The methods used in the programme will be
  selected on the basis of the purpose and content
  of that component of the assessment framework
• ?

21
Principle 4

• The methods used to set standards should be
  transparent and in the public domain
• Difficult area, WPBA formative, most standard
  setting methodology applies to summmative
  process.
• Dilemma do we std set each tools or the summative
  judgement (ARCP panel)
• How ???

22
Principle 5

• Assessments should provide relevant feedback
• The formative nature of the assessments and the
  feedback given after each t is an essential
  component of the process.
• Good use of e-portfolio helpdesk support good.

23
Principle 6

• Assessors/examiners will be recruited against
  criteria for performing the tasks they undertake
• Trainers in primary care ?
• Trainers in secondary care need support,
  training in use of tools, the competency
  framework and meaning of competence.

24
Principle 7

• There will be lay input into development of
  assessment
• PSQ suggests lay involvement in the feedback
• Lay representation on assessment committee, but
  not on WPBA group.

25
Principle 8

• Documentation will be standardised and accessible
  nationally.
• E-portfolio fulfils this
• I have concerns about lack of standardisation of
  the evidence presented and the balance of
  portfolios.

26
Principle 9

• There will be resources sufficient to support
  assessment.
• EP well resourced
• Conflict between Deanery v RCGP, implementation
  and QA, training and support of assessors
  deanery.
• My belief is that RCGP has under resourced WPBA
  in particular supporting deaneries in training
  and QA.

27
What next?
28
Next steps

• 1 Review WPBA
• Tools and methodology fit for purpose
• 4 Standard setting review methodology, which is
  appropriate which component should have
  standards applied?
• 6 Assessors
• Supporting 2 care
• Understanding competency framework
• Congruent assessment practices
• 8 Documentation looking at balanced portfolios.
• 9 Resources ongoing political issue.

29
Critical Events Crashes
30
Truths myths about WPBA

• WPBA and e-portfolio are NOT synonymous
• WPBA is longitudinal assessment of performance in
  the workplace.
• E- portfolio is the personal education portfolio
  and this houses the evidence including WPBA for
  the nMRCGP, certification and licensing process.

31
Changing the mindset

• There is a risk of teaching to the assessment
  hurdles.
• WPBA is not just 12 COTs 12 CBDs
• Risk with e-portfolio that the richness of
  qualitative information from old ETR will be lost
  in the reductionist approach for recording in the
  EP.

32
Key to Quality Assurance

• Work in collaboration with deaneries.
• Congruent assessment practices.
• Provide training resources, cascaded to
  deaneries.
• Prompt and authoritative guidance.
• Adequate IT infrastructure support.

33
The Blueprint Dilemma

• Passed MRCGP video but tape would have failed
  summative assessment.
• Could someone pass CSA but fails in those same
  blueprint areas in WPBA
• Is it possible ?
• and what do we do if it happens?

34
Calm, considered reflection
35
(No Transcript)
36
The solution

• Each component covers different aspects but WPBA
  them all.
• Congruent assessment practices across the
  components and the assessors
• Nationally agreed guidance and concordance for
  assessment judgements made locally.
• Standardisation
• Agreed deanery policies for panel judgments

37
How much evidence is enough?

• The evidence gathered from WPBA builds up a
  picture of the competent GP.

38
(No Transcript)
39
The Future

• New tools and assessments.
• e-learning modules, for management evidence based
  medicine critical appraisal developed and
  standard set by trained RCGP assessors, reducing
  assessor variability.
• Appropriate training and ongoing support and
  appraisal of assessors(1000s)
• Shared ownership with deaneries.

40
Straight Narrow