Selection of Students for Health Professional Courses

1
Selection of Studentsfor Health Professional
Courses

• David Powis
• Faculty of Health
• The University of Newcastle
• New South Wales, Australia

2
The Problem
3
Why arent they choosing the right candidates for
medicine?

• LB Lockhart (1981)
• The Lancet 1 546-548

4
Medical Student Selection in the UK

• For some time there has been dissatisfaction
  over the way medical students are believed to be
  selected, and much inconclusive discussion
  continues
• The Lancet, 24 September 1984, p1190

5
Selection of Medical Students - A Burning Question

• Although mounting criticism and concern are
  expressed for the manner in which our medical
  students are selected, the status quo continues

6
Selection of Medical Students - A Burning Question

• Although mounting criticism and concern are
  expressed for the manner in which our medical
  students are selected, the status quo continues
• EF Campbell et al. The Medical Journal of
  Australia 1 (1974) 785-788

7
What was the status quo?

• In Australia and the UK
• students chosen predominantly on academic
  criteria
• likewise in N. America graduate students

8
Medical Student Selection in Australia

• nobody has any other solution which is strong
  enough to combat..the high enough mark method
  
• J Best. The Medical Journal of Australia, 150
  (1989) 158-161

9
Basis of high academic threshold

• some argue that A-level grades or HSC mark
  accurately reflect academic ability and
  academic potential
• ease of administration!

10
A survey at Adelaide University

• JE Marley I Carman, Medical Education
• 200 medical students, Years 1 2, 1995
• all had been selected on academic marks alone

11
A survey at Adelaide University

• 52 stated a personal motivation to be a doctor
• but only half indicated that they wanted to work
  with people
• 48 admitted personal motivation was not
  responsible for them being at medical school
• 22 parental pressure
• 22 dont waste marks
• 4 no idea

12
The Goal

• to match applicants with the course of study in
  order to maximise the chance of successful
  completion and to produce a good graduate

13
What should we be doing?

• Selecting individuals who
• have a good idea what your course is about
• are motivated to do your course for the right
  reasons
• are likely to be academically successful in your
  course
• have the qualities necessary to be a good
  professional as well as a good student

14
To be selected

• applicants should have appropriate
• motivation
• academic qualifications
• cognitive skills
• non-cognitive qualities
• positive attitudes
• communication skills
• written and verbal

15
Step 1

• Determine an explicit Admissions Policy, and
  advertise this to applicants
• Referring to both the Curriculum and the
  Mission statement of the school list the specific
  criteria that successful applicants need to meet
  in order to achieve congruence

16
Admissions Policy

• Influenced by a variety of factors

17
Pressures that shape the Admissions Policy
Admissions Committee
Admissions Policy
18
Admissions Committee
from within the university
Admissions Policy
19
Admissions Committee
from within the university
from the profession
Admissions Policy
20
Admissions Committee
from within the university
from the profession
from the community
Admissions Policy
21
Admissions Committee
from within the university
from the profession
from the community
political pressures
Admissions Policy
22
Pressures within the University

• compatibility with the curriculum
• examination success
• academic distinction
• social contribution to University community
• (professional competence)

23
Pressure from the Profession

• registration and legal certification
• maintenance of professional standards
• need to fill various specialties

24
Political Pressures

• widening access
• anti-discrimination legislation
• requirements of the healthcare system
• financial control
• resource efficiency

25
Pressures from the community

• Good (caring) practitioners
• (Community at large)
• technically competent practitioners
• (Patients)
• Practitioners of particular gender or ethnic
  background
• preference to local candidates

26
The Admissions Policy should state explicitly

• the level and breadth of prior knowledge and the
  range of cognitive skills and non-cognitive
  qualities essential or desirable
• for success in the course
• congruence between admissions policy and
  curriculum
• in a good professional

27
Step 2

• Specify what level of prior academic achievement
  is necessary
• Determine appropriate subject pre-requisites

28
What prior academic scores are required?

• does a higher score really imply a better
  student?
• or (ultimately) a more competent professional?

29
Biomedical Science, UDSC (1995,1996,1997 entry)
AAA
E
0
10
20
30
40
A level points at entry
30
Biomedical Science, UDSC (1995,1996,1997 entry)
12
I
10
2i
8
2ii
degree classification
6
III
4
Pass
2
0
0
10
20
30
40
A level points at entry
31
Biomedical Science, UDSC (1995,1996,1997 entry)
12
1
1
I
10
1
2i
1
1
1
2
2
3
3
3
3
4
4
7
8
1
1
1
2
3
5
5
6
6
6
8
2ii
degree classification
6
y 0.0242x 6.3493
III
1
1
1
1
1
1
1
2
4
r
0.1091
Pass
2
0
0
10
20
30
40
A level points at entry
32
Curriculum Congruence

• Does the course require more than an ability to
  memorise and recall facts?
• Do the academic demands of the course require
  other cognitive skills?

33
Cognitive skills

• literacy
• numeracy
• comprehension
• processing speed and efficiency
• application of knowledge/intellectual ability
• reasoning skills, logic, problem solving,
  critical skills

34
What is the relationship between academic
achievement and cognitive ability (i.e.,
intelligence)?
35
Cognitive Skills Score v TER
r 0.1934
36
Cognitive Skills Score v TER
r 0.1934
BBC
AAA
37
Cognitive Skills Score v TER
r 0.1934
n 1737
BBC
AAA
38
In Australian medical school applicants.
39
In Scottish medical school applicants.
40
Conclusion

• secondary school matriculation examination
  results dont necessarily reflect cognitive
  ability

41
Intelligence

• The intelligent person is far more easily
• spotted from his (sic) response to
• new problems
• not his knowledge of old solutions
• Paul Kline (1991)
• Intelligence The PsychometricView
• Routledge

42
Which prior studied subjects are required?

• Do your students need Chemistry, Physics and
  Mathematics, for example?

43
(No Transcript)
44
(No Transcript)
45
From HG Gough J Med Ed 53 (1978) 291-300
46
High achievers
Science GPA

• less adept in
• areas requiring
• personal skills
• less articulate
• narrower interests
• less adaptable

Preference for Science Subjects
Composite Index of Scientific Aptitude
From HG Gough J Med Ed 53 (1978) 291-300
47
What else is required?

• for success in the course
• to make a good graduate
• to make a professional practitioner

48
Step 3

• specify what other personal qualities are to be
  sought, and classify each as essential or
  desirable
• attitudes
• non-cognitive qualities

49
Survey of professionals

• in 1998 Newcastle (NSW) academic staff members
  and clinical colleagues were asked to rate the
  importance in medical students of 39 qualities
• 190 completed questionnaires were returned

50
Seven top ranked attributes

• high personal integrity (95)
• critical thinker (91)
• good communication skills (91)
• highly motivated (88)
• evident spirit of enquiry (87)
• caring ethos / good empathy (85)
• racially tolerant (80)

51
Another survey of professionals

• Clinical staff asked to list undesirable personal
  characteristics they had observed in medical
  students

52
Common descriptors of ethically inappropriate
medical student behaviours and attitudes

• arrogant
• power-seeking
• inflexible
• defensive
• dishonest
• patronising
• brash
• egocentric
• isolated
• insensitive
• self-centred
• uncaring

• indifferent
• selfish
• antisocial
• amoral
• devious
• prejudiced
• flippant
• rude
• aggressive
• condescending
• rigid attitudes
• judgemental

53
Step 4

• now we know what we are seeking, determine how it
  can be done objectively, fairly, efficiently and
  effectively

54
Digression - the Macnamara fallacy(Charles
Handy, quoted in BMJ, 325 (2002) S193)

• Measure whatever can be easily measured
• (OK as far as it goes)
• Disregard that which cant easily be measured, or
  give it an arbitrary quantitative value
• (artificial and misleading)
• Presume that what cant be measured easily isnt
  important
• (this is blindness!)
• Say that what cant easily be measured really
  doesnt exist
• (this is suicide!!)

55
Tools for Selection

• choose the most appropriate technique to obtain
  the information required

56
Guiding Principles

• use methods that allow direct observation/measurem
  ent of the quality in question
• use only objective, valid and reliable measuring
  tools

57
Sources of Informationfor making selection
decisions

• academic record
• report from School Principal
• referees reports, testimonials
• self-reports
• written psychometric tests
• structured task
• observed group activity
• personal interview

58
Sources of Informationfor making selection
decisions

• academic record
• report from School Principal
• referees reports, testimonials
• self-reports
• written psychometric tests
• structured task
• observed group activity
• personal interview

59
Assessing Cognitive Skills
Factual Knowledge
Literacy
Numeracy
Decision Making
Logical Reasoning
Problem Solving
Critical Reasoning
Comprehension
Processing speed
60
Assessing Cognitive Skills
Academic Record
Principals Report
Factual Knowledge
Literacy
Numeracy
Decision Making
Logical Reasoning
Problem Solving
Critical Reasoning
Comprehension
Processing speed
61
Assessing Cognitive Skills
Academic Record
Principals Report
?
Factual Knowledge
?
Literacy
probably
probably
Numeracy
probably
possibly
Decision Making
not usually
possibly
Logical Reasoning
not usually
possibly
Problem Solving
not usually
possibly
Critical Reasoning
not usually
possibly
Comprehension
not usually
possibly
possibly
Processing speed
not usually
62
Assessing Cognitive Skills
Academic Record
Principals Report
Psychometric Tests
?
?
Factual Knowledge
?
Literacy
probably
probably
?
Numeracy
probably
possibly
?
Decision Making
not usually
possibly
?
?
Logical Reasoning
not usually
possibly
?
Problem Solving
not usually
possibly
?
Critical Reasoning
not usually
possibly
?
Comprehension
not usually
possibly
possibly
?
Processing speed
not usually
63
Cognitive skills

• academic record
• school principals report
• written psychometric tests

64
Non-cognitive qualities

• autobiographical data
• school principals report
• written psychometric tests
• observed group exercise
• structured task
• interview

65
Communication Skills

• autobiographical data
• observed group exercise
• structured task
• interview

66
Psychometric tests

• objective
• efficient (in terms of numbers of applicants
  processed)
• can be made both reliable and valid
• can measure skills and attributes difficult to
  assess by other methods
• expensive (to set up, maintain and administer)

67
psychometric tests in the context of medical
student performance

• Many reports in the literature, e.g.,
• HG Gough WB Hall (1964)
• Prediction of performance in medical school from
  the California Psychological Inventory
• J Appl Psychol 48 218-226
• R Murden et al. (1978)
• Academic and personal predictors of clinical
  success in medical school
• J Med Ed 53 711-719

68
psychometric tests in the context of medical
student selection

• MCAT (N. America)
• Medical Colleges Admissions Test
• GAMSAT (Australia)
• Graduate Australian Medical Schools Admissions
  Test
• UMAT (Australia)
• Undergraduate Medicine and Health Sciences
  Admissions Test
• a development from the tests used by the
  University of Newcastle

69
Observed group exercise

• logistically demanding to set up
• labour intensive to run
• useful for observing actual behaviour and a range
  of skills that are necessary in a course
• scoring system must be predetermined and
  objective

70
Observed group exercise

• communication skills
• leadership qualities
• debating skills
• teamwork skills
• problem-solving abilities
• problem formulation
• hypothesis generation
• tolerance

71
Structured task

• can be used to observe and measure exactly how a
  candidate does something
• rather than how they say they would do it!
• Communication skill and style
• de-technicalising or paraphrasing complex prose
• interpreting a report
• dealing with an ethical dilemma
• hand-eye co-ordination

72
Interview

• labour intensive
• only useful if objective
• only valid if structured
• only reliable if interviewers trained for the
  task

73
Interview

• should only be used as a tool for data gathering
• best used only for qualities that can be assessed
  directly
• verbal communication skills
• verbal comprehension
• decision making
• critical thinking/reasoning process

74
Interview

• acceptable for purposes of evaluating probable
  future behaviour if care taken investigating past
  experiences and actual behaviour

75
Selection models

• Add the scores, and then rank
• e.g., academic score test score interview
• Philosophically flawed
• Good score in one can compensate for poor score
  in another
• If weightings applied how is the amount
  determined?
• Statistically flawed
• Adding apples to pears

76
Selection models

• Sequential model
• Should be criterion referenced, and
  non-compensatory

77
A possible selection sequence

• coarse filters at the outset
• subject pre-requisites
• threshold academic mark
• psychometric tests in MCQ format for quantifying
  desirable cognitive and non-cognitive skills
  objectively and efficiently in potentially large
  numbers of candidates

78
A possible selection sequence

• labour intensive procedures in the later stages
  when candidate numbers have been reduced
• interview
• structured tasks
• observed group exercise

79
A possible selection sequence

• set academic threshold
• subject pre-requisites
• mark
• psychometric tests for quantifying desirable
  cognitive and non-cognitive skills
• structured interview for identifying verbal
  communication skills, attitudes, motivation,
  decision making

80
Selection decision

• An administrative decision that accommodates the
  data collected
• preferably directed by a predetermined algorithm
• objective data should not be diluted by
  subjective judgements

81
applicants
entrants
82
Pre-determined criteria
first filter e.g., academic threshold
applicants
entrants
83
Pre-determined criteria
Pre-determined criteria
first filter e.g., academic threshold
third filter e.g., interview
second filter e.g., test
applicants
entrants
84
Pre-determined criteria
Pre-determined criteria
Pre-determined criteria
first filter e.g., academic threshold
third filter e.g., interview
second filter e.g., test
applicants
entrants
85
Pre-determined criteria
Pre-determined criteria
Pre-determined criteria
Pre-determined algorithm
applicants
entrants
86
Things we do wrong...

• Use the interview as a selection tool
• apply selection criteria that havent been
  defined
• apply exclusion criteria that havent been
  specified
• Subscribe to the Macnamara fallacy!

87
Useful References

• Medical Student Selection a tentative attempt to
  establish a code of practice
• Anderson et al. BMJ 280 (1980) 1216-1218
• Selecting Medical Students a rational approach
• J Lazarus JP deV van Niekirk Medical Teacher
  8 (1986) 343-357
• Selecting Medical Students
• DA Powis Medical Education 28 (1994) 443-469
• Choosing Tomorrows Doctors
• Isobel Allen, Philip Brown and Patricia Hughes
  (1997)
• Selecting our Doctors
• Report of BMA conference, December 1999

88
The Newcastle procedure

• academic threshold
• subject pre-requisites (none)
• mark (top 10, or top 15)
• psychometric tests for quantifying desirable
  cognitive and non-cognitive skills
• structured interview for identifying verbal
  communication skills, attitudes, motivation,
  decision making

89
To be selected

• applicants must have appropriate
• motivation
• academic qualifications
• cognitive skills
• non-cognitive qualities
• positive attitudes
• communication skills
• written and verbal

90
Pre-determined criteria
Pre-determined criteria
Pre-determined criteria
Pre-determined algorithm
applicants
entrants
2300
1800
300
60
Year 2001 admission