Assessing Employees

1
OPM Initiatives to Improve Personnel Assessment
in the Federal Government
U.S. OFFICE OF PERSONNEL MANAGEMENT
2
Context Reliance on Measures of Training
Experience

• Over half of all competitive new hires
  selected are referred on to the selecting
  officials based on a rating of their training and
  experience. (PPS, Asking the Wrong Questions A
  Look at How the Federal Government Assesses and
  Selects Its Workforce, October 2004)

The Government has gravitated toward the use of
training and experience (TE) assessments. 
MSPBs 2000 Merit Principles Survey asked
supervisors what information they use in
selection decisions.  Almost all96
percentreported using prior work experience to a
great or moderate extent, and 82 percent reported
using level of education to a great or moderate
extent. (MSPB, Reforming Federal Hiring Beyond
Faster and Cheaper, September 2006)
3
Impact of Automation

• Has drastically changed how agencies recruit,
  receive applications, assess and select
  candidates
• Easy for applicants to apply
• Agencies wanting to assess applicants in an
  on-line, unproctored setting
• Challenge
• Paring down candidates while making consistent,
  accurate and merit-based distinctions among them

4
Rating Training Experience Using an Automated
System

• Benefits
• Positions can be filled quickly
• Relatively inexpensive and efficient
• Low burden and high face validity for applicants
• Wide variety of KSAs can be assessed
• Ability for instantaneous scoring
• Test security is not an issue

5
Rating Training Experience Using an Automated
System

• Challenges/Limitations
• Score inflation
• Lower validity
• Lack of discrimination among candidates
• Less suitable for entry-level positions

Identifying top applicants requires a valid
assessment tool
6
Rating Training Experience Using an Automated
System

• Assumptions
• Past performance is a good predictor of future
  performance
• Not all assessment tools are developed equally
• KSAs, items, and scoring are based on job
  analysis
• Automation frees up time and resources to use a
  secondary assessment process (e.g., structured
  interview)

7
Recent Center for Talent Services (CTS)
Initiatives to Ensure Quality

• Review of internal and external best practices
• Interviews with HR Specialists across the Center
  for Talent Services (CTS)
• Preliminary development of a step-by-step
  Examiner Handbook
• Preliminary development of a 2-day training
  session

8
Program Objectives

• Identify best practices applicable to OPMs
  automated staffing system
• Further standardize practices across CTS network
• Provide HR Specialists with context and training
  on best practices
• Offer agencies implementing automated staffing
  systems training in developing questionnaires
• Provide a forum for continued discussion and
  improvement

9
Overview of Topics Covered in CTS Examiners
Handbook

• Researching the Position (Job Analysis)
• Assessing Minimum Qualifications and Eligibility
• Identifying Competencies/KSAs
• Developing Items to Measure Each Competency
• Selecting a Rating Scale
• Using Narrative Text Boxes
• Developing Scoring Procedures
• Adding Warnings and Verification Statements
• Reviewing and Testing the Automated Questionnaire
  
• Documenting the Development Process
• Conducting a Quality Review
• Evaluating the Assessment Process

10
Key Issues Difficult Questions 1

• Which types of items and rating scales are most
  effective?
• Default rating scales or customized rating
  scales? Pros and cons of each?
• Which have the greatest predictive validity?
• Which are most effective in terms of
  meaningfully distinguishing between applicants?

11
Key Issues Difficult Questions 2

• How do you develop effective items and rating
  scales?
• What information do you need?
• What type of expertise is required to develop
  these measures?
• What procedures should be established or training
  should be delivered to ensure that quality
  instruments are developed?

12
Key Issues Difficult Questions 3

• How can item banks be used most strategically?
• How do you maintain proper job analysis
  documentation when using an item bank?

13
Key Issues Difficult Questions 4

• Should narrative information be collected to
  support self-ratings (i.e., via text boxes)?
• How many narrative responses should be collected?
  
• In what situations?
• What factors should be considered?
• How should the information be used?

14
Key Issues Difficult Questions 5

• How can warnings and verification statements be
  used most effectively?

15
Key Issues Difficult Questions 6

• Should a quality review of supporting materials
  (i.e., resume, narratives, etc) be conducted?
• What should it look like?
• When and how should scores be adjusted?
• By how much? In what increments?

16
Key Issues Difficult Questions 7

• Should factors and/or items within a factor be
  weighted differentially?
• Under what circumstances?
• By how much?

17
Key Issues Difficult Questions 8

• What steps should one take to evaluate the
  effectiveness of the self-report instrument?

18
Key Issues Difficult Questions 9

• When should you use a multiple hurdles assessment
  process?

19
Multiple Hurdle Approach
the multiple hurdle approach can increase
selecting officials ability to predict the job
success of an applicant.  This should increase
the quality of the hire and decrease the costs
associated with a poor performer. - MSPB,
Reforming Federal Hiring Beyond Faster and
Cheaper, September 2006
20
Staff Acquisition Model

Develop Staff Acquisition Strategy
Establish Evaluation Approach
21
Moving Forward Sharing Best Practices
Margaret G. Barton, Ph.D. Personnel Research
Psychologist Center for Talent Services U.S.
Office of Personnel Management email
margaret.barton_at_opm.gov phone [phone] Diann
a Saxman Staff Acquisition Program Manager
Center for Talent Services U.S. Office of
Personnel Management email dianna.saxman_at_opm.gov
phone [phone]   Workforce Solutions by
Government, for Government