Curriculum review meeting 20 December 2004

1
The nMRCGP
An opportunity to reconnect teaching and
learning with assessment'   
David Sales
UKCEA
14 June 2006
2
Objectives

• nMRCGP Why? What? When? How?
• Update on nMRCGP developments
• Consider opportunities for the integration of
  assessment with teaching and learning

3
Key messagesWPBA is

• Key to the new CCT
• The mechanism by which assessment material will
  be used to continually improve performance
  through the educational route
• The means by which the new GP curriculum will be
  brought to life

4
So why change? Political

• Standing still was not an option.neither current
  MRCGP or SA would be fit for purpose
• Organised medicine desire to improve/rationalise
  medical teaching and assessment with a coherent
  assessment strategy which
• is outcome based
• uses specified knowledge, skills, behaviours and
  attitudes defined by and mapped to the
  curriculum
• relates to the entire training period

5
But educational too

• Growing unease with competence (can do)
  assessment in controlled environments such as
  exams
• Recognition that learning is facilitated when
  tasks are integrated
• Moving way from reductionism (atomisation/triviali
  sation) to whole task integrated approach of
  assessment of performance (does do)

6
Fitness for purpose (PMETB)

• Validity
• Reliability
• Standards
• Cost-effectiveness (NB not necessarily cheap)
• Adequate feedback
• Impact on learning
• Art is to strike correct balance (utility)

7
A quote from Kolb

• In our over eager embrace of the rational
  scientific and technological our concept of the
  learning process itself has been distorted by the
  rationalist. We have lost touch.
• Assessment should aim to evaluate professional
  judgement, appreciation of complexity and
  uncertainty and powers of reflection.as well as
  factual knowledge and practical skills

8
Opportunities

• Create a vibrant and relevant integrated
  learning and assessment experience
• Maintain and enrich key values of current
  assessments, but reframe and refine test formats
• Replace current dual track assessments with
  single licensing process- reduce assessment burden

9
Opportunities

• Articulation with preceding (basic training and
  F2) and subsequent (HPE/revalidation) assessments
  
• Improve standards in partnership with patients-
  selection, training and shifting the mean
• Reconnect teaching and assessment with an
  integrated continuum of assessments within the
  entire training programme, especially with
  work-place based assessments

10
The chosen solution

• RCGP/NSAB Assessment group set up in response to
  the PMETB and proposed
• A single new licensing assessment process to be
  applied across UK (neither SA or MRCGP) the
  nMRCGP
• Assessments should fit in with an overarching and
  coherent strategy relating to the entire training
  period which encourage progression and future
  professional development

11
nMRCGP- an integrated assessment programme

• Workplace based assessments (WPBA)
• Applied knowledge test (AKT)
• Clinical skills assessment (CSA)

12
Key nMRCGP messages

• Each of these is independent, will test different
  skills but together will cover the GP curriculum
• Each component will contribute one third of the
  assessment programme
• RCGP will deliver the CSA and AKT and the
  deaneries will deliver the WPBA

13
Testing the curriculum

• Assessments must be pinned to the curriculum ILOs
  and integrated to avoid duplication of test
  developments
• Vanilla solution? Every test method is flawed but
  some are useful and some are more useful than
  others
• Each method may test different but nevertheless
  important competencies which must be cross linked
  (triangulation)

14
Current MCPgt AKT

• Already approved for purpose of SA
• Same length -200 item 3 hour test
• Provides sufficient content coverage and good
  reliability
• Same format SBA, EMQ
• More frequent testing February, May and October,
  timed to meet demand

15
AKT

• Defensible item bank management (Speedwell) being
  implemented
• October 06 MCP should be entirely constructed and
  marked using Speedwell
• Looking at optimal methods for standard setting
• Future computer based testing

16
AKT

• Content drift, shift , deeming
• 80 clinical medicine
• 10 administration
• 10 on research and statistics
• Standards are another matterSEM

17
Purpose of CSA

• An assessment of a doctor's ability to
  integrate and apply clinical, professional,
  communication and practical skills appropriate
  for general practice

18
Scope of CSA

• PRIMARY CARE MANAGEMENT
• Recognition management of common medical
  conditions in primary care
• PROBLEM-SOLVING SKILLS
• Gathering using data for clinical judgement,
  choice of examination, investigations their
  interpretation. Demonstration of a structured
  flexible approach to decision-making

19
Scope of CSA

• COMPREHENSIVE APPROACH
• Demonstration of proficiency in the management
  of co-morbidity risk
• PERSON-CENTRED APPROACH
• Communication with patient the use of
  recognised consultation techniques to promote a
  shared approach to managing problems

20
Scope of CSA

• ATTITUDINAL ASPECTS
• Practising ethically with respect for equality
  diversity, with accepted professional codes of
  conduct
• TECHNICAL SKILLS
• Demonstrating proficiency in performing physical
  examinations using diagnostic/ therapeutic
  instruments

21
CSA format

• Will be in a multi-station OSCE format and will
  use simulated patients
• Comprising 14 scoring stations, probably each
  lasting 10 minutes, to give an acceptable
  reliability
• When your work speaks for itself why interrupt?

22
Venue

• One purpose built clinical skills test centre to
  be set up nationally, probably in the new RCGP,
  possibly in 2009
• Interim plans likely to use single circuit at
  GMC PLAB test centre
• PLAB pre pilot, Warwick main pilot

23
When?

• After completing a minimum of 6 months of GP in
  ST and within 18 months of completion of the
  planned ST programme
• Probably a maximum of 4 attempts

24
CSA pre pilot

• Main intention was as a feasibility study, but
  felt authentic to 10 GPRs sitting 10 stations
• Important not to over-analyse or over-interpret
  data from a study of this size but.
• The overall case-based reliability looks good and
  based on this small sample confirms that one
  using 14 stations should achieve the target
  reliability (? gt 0.9)

25
WPBA- draft

• Is the evaluation of a doctors progress over
  time in their performance in those areas of
  professional practice best tested in the
  workplace

26
WPBA

• Teaching, learning and assessment will be closely
  integrated in the WPBA by
• Creating the opportunity for gathering evidence
  of actual performance in the workplace
• Enabling assessment of aspects of professional
  behaviour that proved impossible in traditional
  assessments

27
The Educational Model of WPBA for nMRCGP- draft
GP Trainee
Educational Supervisor
Structured Tools
Submission of Evidence
Local assessment
Objective Assessment and Evidence of Learning
Feedback
Specialist GP Trainees Progress in Performance
Training Record
Record of Assessment
The educational cycle to be repeated at agreed
intervals throughout 3 year training period
Deanery Decision Based on all Evidence
28
Link between training and independent practice

• Model is a neat and coherent transposition
  between the training context and the independent
  practice context
• Proposes how WPBA, appraisal and revalidation
  might come together
• TraineebecomesGP
• Educational supervisorbecomesappraiser
• External tools.become.for peer review
• Deanery decisionbecomesrevalidation decision

29
WPBA- some challenges

• To reconcile potentially conflicting
  institutional expectations 
• The need to be simple and to reduce the
  assessment burden on one hand and on the
  other, from the patients that it confirms safety
  and from PMETB that it is fit for purpose.

30
WPBA reliability?

• Sufficient evidence A plea for new
  psychometric models Med Educ 2006 40 296-300
• Remember triangulated assessments
• Are we trying to defend it with the wrong
  paradigm? .like doing quantitative research in
  situations where we need qualitative - it took us
  ages to accept the value of qualitative research

31
Enhanced training report (ETR)

• The competency based ETR effectively will
  comprise a portfolio which collects evidence
• The trainer will have a central and developmental
  role and give feedback on the progress of their
  registrar in 12 competency areas

32
ETR

• The judgement made by the GP trainer will be
  externally moderated by the deanery which will
  review all evidence, including that from the
  externally moderated assessments

33
ETR

• The evidence may be naturally occurring and so
  gathered informally or it may be gathered more
  formally using assessment tools.
• The assessment tools being piloted for use in the
  work place are
• Multi-source feedback
• Case-based discussion
• Consultation observation tool

34
External assessments

• External assessments are being piloted
• Audit
• Multi-source feedback (MSF)
• Patient satisfaction surveys
• Referrals analysis
• Significant event analysis (SEA)
• Video

35
Pilots

• The ETR pilots will be completed by September
  2006 following which the precise composition and
  blend of objective (external) assessments will be
  decided

36
WPBA

• Will occur throughout the entire GP specialist
  training period, including the hospital posts.
  The details of how this will be administered have
  yet to be determined
• Is a longitudinal process which cannot be taken
  before the other assessments and will be signed
  off in the final period of training

37
Psychomotor (technical) skills

• .those required to undertake examinations and
  practical skills appropriate for general
  practice
• What are they.? Skill retention, duplication,
  management of acute patient problems
• How best to assess them? DOPS
• Where? WPBA cf CSA? Sampling

38
Psychomotor skills

• Accept F2 signoff is sufficient for acute care
  purposes
• CSA provides the opportunity to assess any PMS
  which sends a clear signal to GPRs that they take
  a risk if they fail to maintain their skills

39
Candidate numbers

• 2320 trainees completed their training in the
  year April 05 - March 06 (cf 2190 in the previous
  year)
• We anticipate that the numbers are likely to be
  similar (max 3000) for the coming years as
  deanery budget cuts have meant that recruitment
  has been pretty static over the last couple of
  years

40
(No Transcript)
41
What next?

• Electronic portfolio, which would be capable of
  informing and guiding doctors as to the required
  competencies, contexts and priorities.
• Communication.accurate dissemination of user
  friendly information of the nMRCGP proposals
• Recruit and train assessors

42
What next?

• Evidence and fact based decisions
• Integrated approach to developments
• Robust data management
• Piloting
• Ensure smooth and orderly transition from
  current MRCGP and SA methods

43
Transition from current to new assessment

• Complex and demanding (old and new)
• RCGP and COGPED have requisite collective wisdom
  and expertise to deliver
• Must be supported especially with resources
• (business planning) and IMT
• Realistic timelines

44
What next?

• PMETB QA/QC template submission.
• The current transition proposals are all subject
  to PMETB approval
• Must be deliverable