Bringing Everyone on Board

1
Bringing Everyone on Board

• Increasing the Opportunity for Diversity in
  Entry-level Professional Appointments in New York
  State
• Jennifer Hoerup Principal Staffing Services
  Representative
• Ken Siegal Chief Personnel Examiner
• New York State Department of Civil Service

2
What is the Professional Careers Test?

• General entry level test for college grads
• No specific degree or experience required
• Generally given every four years
• Used to fill over 50 different titles
• A few hundred appointments from list
• Most appointments from top of list
• Some locations may appoint from further down the
  list

3
What does the PCT Cover?

• Cognitive test - written, multiple choice
• Four subject areas 15 questions each
• Preparing written material
• Understanding and interpreting written material
  related to government and social issues
• Quantitative analysis (tab)
• Verbal analysis (logic)

4
Top Management saidMake it better

• Improve overall results
• Improve diversity in appointments
• Diversity option used in the past with special
  minimum qualifications
• Less restrictive qualifications were deemed more
  appropriate this time
• Make it friendlier to all candidates
• Get the word out and recruit

5
What did we do to make things better?

• Recruitment
• Compelling Concerns
• General decline in exam applications
• Need for greater diversity in the state workforce

6
Recruitment Action Plan

• Centralized recruitment effort
• Adjustments to announcement and application
  timelines
• Info sent to 124 New York State colleges
• Partnership with CUNY to promote exam
• Outreach to college career services
• Information sessions

7
Recruitment Results

• Total application number similar to previous
  holding
• Increase in downstate applications 3350, up
  from 2820
• CUNY students a major component of downstate
  applications
• CUNY appointments

8
What did we do to make things better?

• The Testing Process
• Changes to the test itself
• Test administration issues
• Analysis of results
• Scoring methods
• Unclear what impact each thing had
• Will discuss results at the end

9
Review of Prior Test Results

• Does the test work the same for all groups?
• Principle Components Analysis
• Analysis looked at
• All candidates as a group
• Each ethnic group separately
• Common candidates across holdings
• Generally, sufficiently similar factor structure
  to proceed

10
Analysis Within Subject Areas

• Reviewed each subject areas prior test results
• Compared results by ethnic group
• Individual item statistics
• Subtest statistics
• Cleaned up item selection

11
Sensitivity Reviews

• Big push not done in recent times
• Separate Black and Hispanic panels
• Reviewed all test material
• Items
• Directions and instructions
• Used Mantel-Haenszel statistics to identify
  problem items

12
Sensitivity Reviews (cont)

• Many significant details emerged
• Made changes to directions and instructions
• Added clarity and precision to directions and
  items
• Reviewed before and after test was held
• No changes made to items or scoring based on
  post-test review

13
Action Plan

• Item Selection
• Eliminated items with prior negative results
• Revised items based on sensitivity reviews
• Recast some material into government settings
• Subject areas remained essentially intact

14
Test Guide

• First PCT Test Guide
• Covered all subject areas
• Available on-line (and still is)
• Included special feature a practice test
• Permitted greater candidate practice
• Enhanced candidate preparation
• Sample questions selected to facilitate
  familiarity with actual test material

15
Did Candidates Use the Test Guide?

• YES
• 32,000 hits (about 10K applicants)
• Substantial usage in week before exam

16
Special Candidate Directions

• Candidate Directions are used for all our tests
• A little wordy and some important details are not
  highlighted
• Modified and simplified PCT directions
• Able to get everything on one side of one page
• Emphasized critical info on time/ questions

17
Special Mailing of Admission Notices

• Notices mailed one week early
• Mailing included special letter
• What to bring
• Reminded them of Test Guide
• Reminded them of How To Take a Written
• Provided actual copy of Candidate Directions
• Increased web hits as a result of mailing
• 4000 hits week before exam
• Increased hits on Guide and How To Take (pdf)

18
Test Administration

• Two days of testing Saturday/Sunday
• Simplified directions
• No test administration problems
• Candidate counts similar to prior years
• Recruitment brought these numbers up

19
The Results

• All groups did better than in the past
• All group differences smaller
• Some items flagged by Mantel-Haenszel
• Posttest sensitivity review generated no key
  changes or deletes

20
The Scoring Plan

• A variety of scoring methods were considered
• Band scoring permits consideration of other
  factors
• Reachability of greatest importance
• Passpoint important but of less practical
  significance

21
Banding via SEM

• Recommended SEM-based band sizes
• (SEM standard error of measurement)
• Reliability-related statistic
• Test reliability about .8
• SEM about 4

22
Final Scoring Plan

• Top management reviewed band sizes, candidate
  counts, and final scoring options
• Decision made
• Use 2 X SEM 8 points top band
• Succeeding bands 1 SEM
• Final scores at 5 point intervals
• Improves the initial reachability of top scoring
  candidates
• Manageable subsequent band sizes

23
Results

• Pass point meets 80 rule for all groups
• Appreciable numbers of protected class candidates
  in top band
• Proof of the pudding is in the hiring
• A few appointments have been made
• Gender and ethnic percentages of appointees match
  those in competition

24
Links

• The following links will take you to the
• Archived PCT announcement
• http//www.cs.state.ny.us/announarchive/announceme
  nts/25-145.cfm
• PCT Test Guide
• http//www.cs.state.ny.us/testing/test_guides/PCT
  20Test20Guide.pdf