Utility Analysis

1
Utility Analysis Professional Qualifications in
Test Administration BPS Level A and Level B
(intermediate) courses

• Assessment in the Workplace

2
Overview

• What is Utility Analysis (UA) ?
• Traditional UA framework SDy and costs of
  selection
• Practical Example
• Issues in UA
• Alternative UA framework Boudreau et al (1997)
• Level A and Level B (Intermediate)
• M21 Seen Paper

3
What is Utility Analysis ?

• a family of theories and measures designed to
  describe, predict and/or explain what determines
  the usefulness or desirability of decision
  options and to examine how information affects
  decisions (Boudreau, 1991)
• Can be applied to selection processes, training,
  compensation, performance assessment, and
  internal staffing
• Evolved to provide tools for better describing
  and communicating impact of HRM and Occ. Psy.
  interventions on organisational goals.

4
Traditional UA framework for utility analysis of
new selection technique

• Top-down
• hiring
• Applicant Ability to Performance Value
  of
• Pool Predict Intervention
• New
• Intervention

5
Fundamental Processes in UA

• The relationship between predictors and criteria
  (represented by r)
• The nature of the criteria, represented by SDy
• The nature of the selection process, represented
  by Z
• The nature of the implementation process,
  represented by C.

6

• The expected increase in output is directly
  proportional to the predictive validity of the
  test (Brogden, 1946)
• Brogdens equation
• Yearly saving
• per employee (SDy x rv x Z) - costs of
  selection
• selected (/,etc)
• where
• SDy standard deviation of value in performance
• rv predictive validity coefficient of the test
• Z mean standard score of all applicants
  (selected and rejected) on the test
• Costs of Selection administration costs per
  selected employee multiplied by the number of
  employees selected
• Hence, for the amount saved for N selected
  employees, staying with the company for T years
• Total savings N x T x (SDy x rv x Z ) -
  (costs of selection)

7
How much more is a good worker worth than a poor
worker (i.e. SDy)

• Schmitt and Hunter (1982) 2 assumptions
• The value of a worker to a company is directly
  proportional to that workers performance
  (output) when the workers performance is
  measured on some criterion relevant to the
  company.
• If performance is normally distributed, then the
  value of performance should also be normally
  distributed.

8
When SDy is large vs when SDy is small

• If SDy is large (high) then employees vary a
  lot in how much they are worth to the company ...
• If SDy small (low) then investment in selection
  is of less value, as there is less discrimination
  possible amongst workers
• ( gtgtgt random selection ?!)

9
The costs of selection

• N.B. costs include costs of assessing those who
  are rejected as well as those that are selected
• Costs must include all costs (i.e. labour time,
  materials, selection programme development costs,
  etc.)
• If the cost of assessing each applicant is C and
  SR (selection ratio) is the proportion of
  applicants selected
• Costs per selected applicant C / SR.
• Costs of selection
• (costs per selected applicant) x (number of
  applicants selected)

10
Examples of Utility Analysis Results

• Schmitt Hunters estimates of predicted savings
  due to large scale implementation of improved
  selection methods
• Use of Computer Aptitude Battery by US Federal
  Government 64,000 per selectee, over 9 years.
• In US Federal Government selection, replacement
  of ability tests and work sample tests by less
  valid predictors would lead to losses of 3.12
  billion (job tryout) to 15.8 billion (age) per
  year.
• General improvements in the Philadelphia Police
  Force (5000 employees) 18 million/year
• General improvements in selection - US National
  Economy 87-152 billion/year.

11
Practical Exercise

• Old Selection Test
• SDy 30,000, rv 0.2, Z 1.0
• Costs per selected employee 1,000
• 10 people (N 10) selected who stay on average 4
  years (T 4)
• New Selection Test
• SDy 30,000, rv 0.4, Z 1.0, N 10, T 4
• Costs per selected employee 2,000
• Calculate the total savings (larger figure more
  saved) for the old test and the new test.
• Should the old or the new test be used ?

12
Special Issues

• To what extent can the validity reported for a
  test in a manual be applied to your selection
  context (Validity Generalisation synthetic
  validity)
• Real life issues not considered by
  traditional utility analysis
• Inflation (selection costs need to be adjusted)
• Variable costs of employees
• Taxation (of additional profits)
• Lost investment elsewhere
• Number of good candidates available?
• Do not overlook the old selection system
  (typically assume validity is zero).
• The real worth of improved selection

13
Boudreau et al (1997)

• Actual Hiring
• Method
• Applicant Ability to
• Pool Predict
• current aspects of VALUE
• interventions new performance OF
• intervention(s) INTERVENTION

14
Boudreau et al (1997)

• Multivariate validity over time (the Black Box
  of validity)
• temporal stability of validity
• multidimensional criteria (the nature of SDy)
• Using employee movement to represent multiple
  criteria
• The interaction between multiple predictors and
  multiple criteria
• The Black Box of Z
• Is maximising organisational performance (s)
  always an organisations main objective ?

15
Professional Qualifications in Test
AdministrationBPS Certificates of Competence in
Occupational Testing

• BPS website - for level A and level B (int.)
  General Information Packs (http//www.bps.org.uk
  )
• Is it worth me taking these additional courses?
• Course dates and costs - see MSc noticeboards
• Course content structure
• Level A Ability tests, aptitude tests, interest
  inventories
• Level B (Intermediate) Hogans Personality
  Inventory (HPI)
• Both 3 days (9am-5pm), with pre- post-course
  work