PSI Division of Work and Occupational Psychology

1

• PSI Division of Work and Occupational Psychology
• CPD Seminar February 2008
• Recruitment and Selection
• Implementing New Approaches
• Áine Gray
• Public Appointments Service

2
Our Customers

• Civil Service
• Entry to the civil service
• Promotions
• Local Government
• Senior Management
• Professional
• Technical
• Health Sector
• Senior Management
• Professional
• Technical
• An Garda Síochána

80 100 Clients
Handling approx. 80,000 applications p.a.
Filling approx. 5,000 vacancies p.a. Emerging
RD capability
3
Overview

• Realistic Job Simulations
• Situational Judgment exercises
• Video based tests
• Internet Based Testing
• Case Study Using the Internet to assess people
  with disabilities

4
Realistic Job Simulations

• Growing body of research evidence that realistic
  job simulations are effective in measuring job
  relevant skills
• Also known as situational tests
• Anastasi and Urbina (1997) define a situational
  test as one that places the test taker in a
  situation closely resembling or simulating a
  real-life criterion situation
• Generally fairer than traditional cognitive
  tests, except where the job has a heavy cognitive
  load
• Gives candidates a realistic job preview
• Credible with candidates and the media

5
Some jobs are harder to simulate than others..
6
(No Transcript)
7
(No Transcript)
8
What is a Situational Judgment Exercise (SJE)?

• Also known as Situational Judgement Test, Job
  Simulation or Situational Judgement Inventory
• First used in 1920s
• Present candidates with job related situations/
  dilemma
• Adopt a multiple choice format
• May be bespoke for a particular role or
  off-the-shelf for particular grade e.g.
  managerial role

9
Experience of Public Appointments Service

• Situational Judgement Exercises used since mid
  1990s
• Used for a range of roles including entry level
  posts (Garda Trainee, Executive Officer, Junior
  Diplomat) and middle to Senior Management
  Positions (Assistant Principal recently
  introduced at Principal Officer level)
• Used as part of a competency based approach to
  recruitment to assess a range of competencies
• Particularly valuable because of gender
  performance
• Almost all SJEs are bespoke for the role / grade
  and set in a Civil Service Context
• Designed in-house with job incumbents, line
  mangers and other SMEs, trialling and item level
  analysis of test performance

10

• You are on duty alone in a housing complex which
  has a reputation as a trouble spot.
• While you are on patrol, a youth who is one of a
  group throws an empty mineral can
• which hits you on the head. You are not
  physically injured, but are annoyed. The group
• jeers at you in a threatening way.
• What do you do?
• A Notify the Station by radio that a situation is
  developing and then approach the group yourself
  to apprehend the can-thrower.
• B Approach the group and warn them that they will
  be arrested if they continue being abusive.
• C Choose not to respond to this incident as you
  are not injured by the can and continue on your
  beat.
• D Approach the youths and calm the situation down
  by asking them to move on. Decide to deal with
  the issue of the can-throwing later on when the
  situation has calmed down.
• E Make a mental note of who threw the can, radio
  for assistance and wait for back-up.

11
What does an SJE measure?

• Job Knowledge

• Organising for Results

• Practical Intelligence

• Commitment to high Standards

• General Cognitive Ability

• Call Centre Performance

• Decision Making

• Team work Knowledge

• Negotiating/ Influencing

• Aviation Pilot Judgement

• Integrity

• Conscientiousness, Emotional Stability
  Agreeableness

12
What does an SJE measure? Contd.

• Best viewed as a measurement method and not a
  measure of a single construct (O Connell et al
  2007) outputs may be a single score or range of
  scores across constructs/ competencies
• Can assess the integration of cognitive and
  affective based behaviours Lewis et al 2006,
  cited in Creighton Scott 2006.

13
Recent developments Principal Officer

• Senior Grade within the Civil Service
• Exercise to be used for existing Civil Servants
  and open candidates
• Initial context setting brief with organisation
  chart to increase the fidelity of exercise
• Candidates presented with 12 scenarios and
  provided with 5 possible actions asked to rate
  each in terms of their appropriateness on a 6 pt.
  scale from highly appropriate to highly
  inappropriate
• 971 candidates sat the tests
• 87.7 found it relevant or very relevant
• 91.9 felt it was at the right level of
  difficulty
• 82.4 felt it was fair or very fair

14
Considerations in the use of SJEs

• Benefits
• Effective assessment methods
• Can balance out the potential adverse impact of
  cognitive tests
• Flexibility Can measure a range of required
  constructs
• High levels of face validity for candidates
• Can be administered efficiently to large numbers
  of candidates
• May be designed to replicate specialist jobs
  effectively useful realistic job preview

15
Considerations in the use of SJEs

• Challenges
• Challenges with Candidate feedback
• Significant investment required to devise
  exercises
• Balance needed when using exercises on a confined
  and open candidate pool
• In confined competitions, a neutral context is
  needed rather than one favouring candidates from
  policy/ operational areas
• Research suggests faking
• Is easier when items are more transparent
• Is more difficult when there is a heavier
  cognitive load
• Is more likely with behavioural instructions
  when asked what would you do, people will respond
  with what they feel is best answer

16
Example Fire-fighter campaign

• Online application only (over 4,300 applicants)
• Comprehensive assessment process
• Screening
• Job relevant selection tests (video based)
• Learning information
• Understanding Information
• Using numbers
• Observation
• Very positive feedback from candidates over 85
  
• describing the screening process as clear,
  relevant fair
• Follow up assessment
• Interview
• Team Exercise
• Practical Orientation Exercise

17
(No Transcript)
18
Example Fire-fighter campaign - Video based tests
Learning Information During this test candidates
were presented with a series of short talks/
presentations on video during which they were
encouraged to take notes. The talks/
presentations were on fire-related topics.
Candidates were assessed on how much they had
learned about the topic through a series of
multiple choice questions. Observation This
test is about the ability to identify key
information presented on video (involving a
series of short films of objects set out in a
number of rooms. Within these rooms are a range
of items such as boxes, planks, containers
electrical equipment and furniture After the
video candidates were shown a plan of the rooms
and asked a number of multiple choice questions
about the video
19
Recent developments Emergency Medical
Controller

• Emergency Medical Controllers take emergency
  medical calls and dispatch assistance
• Job Analysis demonstrated requirement for
  attention to detail and very specific
  interpersonal skills when dealing with callers
  and other health care professionals
• Sourced video based assessment
• Call taker test replicated a call-centre.
  Candidates required to take notes, and after an
  excerpt from the phone calls, choose which of a
  series of options the call taker should follow
• Dispatch test candidates had to dispatch
  emergency vehicles according to a set of
  priorities

20
(No Transcript)
21
Validity

• Sourced from IPMA HR
• Because of US legal context, very good validity
  evidence e.g
• Correlation between call-taker test and academy
  performance was .30
• Correlation with supervisors ratings was .20
• Correlation with overall pass/fail on probation
  was .28
• Number in sample 627

22
Case Study EO Disabilities Campaign 2007

• Stage 1
• On-line testing
• Verbal Reasoning test
• Skills and Experiences Questionnaire
• Equality Monitoring Form
• Stage 2
• Paper and Pencil testing
• Verbal Reasoning test
• Numerical Reasoning test
• Job Simulation Exercise
• Stage 3
• Competency-based interview and Work-sample test

23
Why Internet based testing?

• Evolution from supervised computer based testing
• Convenient for candidates
• Applicant population generally familiar with
  technology
• More efficient than face-to-face testing

24
Professional Concerns about Internet Based Testing

• Significant debate about the validity of
  Internet Based Testing (Tippens et al, 2006)
  Key concern threat of cheating to the integrity
  and validity of ability tests
• Concerns led Health Professions Council in
  South Africa to ban unsupervised Internet testing
  successfully challenged in the courts
• Cizek (1999) Propensity to cheat is
  negatively related to job performance (-.30)
• Burke and Wright (2008) SHL Verify range of
  tests
• 578 applicants re-tested
• Across the sample 5 flagged as aberrant about
  the rate expected by chance
• Applicants to graduate programmes 9 on the
  numerical test, 18 on the verbal test

25
Case Study EO Disabilities Campaign 2007

• No.
• Applied 575
• Sat Stage 1 on-line 311
• Sat Stage 1 off-line 81
• Passed Stage 1 281
• Sat Stage 2 250
• Passed Stage 2 124
• Stage 3 October 2007
• People currently being assigned to jobs

26
Stage 1 Feedback survey from candidates

• Telephone survey conducted with over 100
  candidates 45 responded
• 84 felt that doing the tests on line is a good
  idea for people with disabilities
• 78 preferred to do the tests on-line vs in a
  supervised environment
• more relaxed on-line
• the test was much easier on a PC
• 90 found it convenient to do the tests in their
  own time
• 100 of candidates use computers everyday

27
References

• Creighton, P Scott, N. (2006). An Introduction
  to Situational Judgement Inventories. Selection
  and Development Review 22, 3-6.
• O Connell et al (2007). Incremental Validity of
  Situational Judgement Tests for Task and
  Contextual Job Performance. International Journal
  of Selection and Assessment. 15, 19-29.
• McDaniel et al (2007) Situational Judgement
  Tests, Response Instructions and Validity A
  Meta-analysis. Personnel Psychology 60, 63-91.
• Lievens, F., Peeters, H Schollaert, E. (In
  Press) Situational Judgment Tests A Review of
  Recent Research. Personnel Review

28
References
International Guidelines on Computer-Based and
Internet Delivered Testing International Test
commission (2005) www.intestcom.org On-line
Testing and the Disability Discrimination Act
(SHL) www.shl.com Cizek. G. J (1999). Cheating on
tests How to do it, detect it, and prevent it.
New Jersey LEA Tippins, N. T., Beaty, J.,
Drasgow, F.,, Gibson, W.M., Pearlman, K., Segall,
D.O, and Shepherd, W. (2006) Unproctored
Internet Based Testing in employment settings.
Personnel Psychology, 59, 189-225.
29

• Thank You!

30
Chapter House, Abbey St, Dublin 1
aine.gray_at_publicjobs.ie