Phillip W. Boswell MA, B.Ed, RRP

1
Common Tools in Vocational Evaluation for
Marginalized Client Populations

• Phillip W. Boswell MA, B.Ed, RRP
• R. Stephen Russell, MA. Sc, P. Eng

2
Learning Objectives

• Introduction
• Identify the difference between vocational
  assessment and vocational evaluation
• Recognize the three levels of vocational
  assessment
• Ascertain the factors involved in the appraisal
  of clients
• Identify useful voc assessment/evaluation
  resources
• Develop a working knowledge of common tools used
  in vocational evaluations
• Work samples

3
Assessment - Defined

• Systematic procedures to obtain information from
  a variety of sources to draw inferences about
  people
• (Standards for Educational and Psychological
  Testing, 1999)

4
Five Phases of Vocational Rehabilitation

• Same job with the same employer.
• Same employer, modified job (light duties)
• Different employer, similar or closely related
  job
• Different employer, alternative employment in the
  same or related industry
• Development of new occupational skills

5
What can be assessed?

• Functional skills (basic skills functional
  capacity)
• Interest
• Achievement
• Aptitude
• Intelligence
• Learning style

• Work Samples
• Personality
• Assistive technology
• Neuropsychological
• Sensory impairments
• Career Assessments
• Dexterity

Career maturity, competency, employment barriers,
emotional and multiple intelligence, job
readiness, personal factors, personal style,
prior learning and values
6
Factors in Appraisal of the Individual
7
Vocational Evaluation Tools
8
Individual Appraisal/Tools
9
Vocational Assessment - Defined

• A general term for the process of identifying and
  appraising an individuals level of functioning
  in relation to vocational preparation and
  employment decision making

10
Why Vocational Assessment?

• Clients often do not have precise occupational
  goals and aspirations. They frequently have
  insufficient or inaccurate information on the
  labour market and limited self-knowledge of their
  own interests and abilities.

11
Purpose of Vocational Assessment

• Plan a course of action
• Enhance client self-knowledge and vocational
  decision-making abilities
• Predict realistic employment outcomes that result
  in successful client vocational rehabilitation

12
Role of the Rehabilitation Professional

• Collect enough information or data about the
  client to diagnose and make predictive statements
  about his/her potential to obtain a successful
  rehabilitation outcome

13
Levels of Vocational Assessment
Level 1 Screening/Needs Assessment
Level 2 Clinical or Exploratory
Level 3 Comprehensive Vocational Evaluation
14
Level 1 Screening/Needs Assessment

• Initial Process
• Needs Assessment
• Determine what is necessary to develop a plan of
  action (e.g. vocational plan)

15
Level 1 Screening/Needs Assessment

• Includes
• Interview with client
• Collect routine background information
  (demographics)
• Reliance on subjective consumer statements
• Vocational choice/interest
• Self-estimates of competence
• Reported work history
• Functional Assessment (e.g. personal capacity
  questionnaire)
• Limited, if any, standardized testing (e.g.
  interest)

16
Level 2 Clinical or Exploratory

• Intermediate process
• In depth exploration or case study approach to
  the client and
• vocationally related circumstances. Builds on
  level 1 information
• through the use of
• Additional interviews
• Collect/analyze documents (schools records,
  medical records, etc.)
• Career exploration
• Vocational and/or adjustment counselling
• Psychometric/standardized testing
• Transferable skills analysis
• Computerized job matching
• Job Analysis and/or environmental assessment
• Assistive technology considerations

17
Level 3 Vocational Evaluation

• Final Process
• Comprehensive individualized holistic process of
  assessment that utilizes specific instruments,
  procedures, and behavioural observation
• Designed to measure, observe, and document
  interests, values, temperaments, work-related
  behaviours, aptitudes, skills, and physical
  capacities, in order to predict viable employment
  and/or training outcomes
• Used when more in depth information about the
  client is necessary and not available from
  information in level 1 and 2

18
Level 3 Vocational Evaluation

• Real or simulated work used as the focal point of
  the evaluation
• Systematic observation and recording of work
  behaviour performance
• Occurs over time and uses multiple methods and
  techniques to validate finds. Some combination of
  the following methods are used
• Work samples, job samples
• Situational assessments, community based
  assessments
• Standardized testing
• Interview
• Transferable skills, job matching
• Analysis of background information
• Career exploration/job shadow
• Assistive technology considerations
• Prescriptive recommendations

19
Distinction Between VA VE

• VE is holistic considers disability/medical,
  psychological, social, vocational, educational,
  cultural and economic issues
• Ideally conducted by a certified vocational
  evaluator (CVE)
• VE occurs over time (days/week )
• evaluated over time with varying work-related
  demands and environments
• Systematic Behaviour Observation and Recording
• Work performance
• Work behaviour
• Adaptive and/or transferable skills are
  questionable or unknown
• Work is the specific focus of VE
• Work samples
• Situational assessments
• Community-based assessments
• Occupational resources
• Career exploration

20
If we are facing in the right direction, all we
have to do is keep on walking

• Buddhist proverb

21
Vocational evaluation identifies.

• Transferable skills
• Vocational Interests
• Worker Characteristics and Behaviours
• Academic Abilities and Potential

• Learning Styles
• Work Aptitudes and Abilities
• Suitable Employment Alternatives

22
Evaluating Assessments

• When assessing an individual, the first question
  is alwaysShould we use tests at all?
• Testing may not always be appropriate
• Extreme anxiety
• Sensory impairments
• Minimal education
• Limited language abilities

23
Questions to ask

• Is the test reliable (repeatable) and valid (does
  it measure what its suppose to?)
• Does my client meet the reading level required by
  the test?
• Is my client disadvantaged in any way by
  completing this test?
• Will the test provide the information that I am
  looking for?

24
Questions to ask

• Can the test administration procedures be
  modified to accommodate the client?
• Can clients answer the question on the test form
  rather than a separate score sheet?
• Are all the questions relevant and correct for my
  client?

25
Questions to ask

• Are there relevant norms/criteria to compare my
  clients performance?
• Issue in rehabilitation is the lack of
  representation of marginalized individuals in
  sample populations
• Marginalized client interested in competitive
  employment is better served using general
  working population norms rather than disabled
  norms

26
Transferable Skills

• Can be achieved through formal or informal
  training
• Is work related
• Should have a measurable component
• Should have the capacity to be matched to
  other/alternative work options

27
Vocational Interests

• Subjective measure of an individuals preferences
  in work activities
• Can add validity by incorporating variety of
  surveys with similar outcomes
• Does not confirm that skills exist

28
Worker Characteristics and Behaviours

• Vocational Evaluation relies heavily on the
  observational skills of the Counsellor
• The Counsellor takes on different roles
  throughout the evaluation
• Allows the inclusion of subjective information in
  the process

29
Academic Abilities and Potential

• Understanding current skills
• Can evaluate need for upgrading
• To competitively seek employment
• To pursue further training
• Can predict potential
• Timed vs. untimed testing
• Needed to insure proper skill profile is created

30
Learning Styles

• Generally subjective in nature
• Can be helpful in identifying training
  environment
• Can be helpful in identifying suitable work
  environment

31
Work Aptitudes and Abilities

• Computerized assessment
• Work Samples
• Academic Achievement
• Aptitude testing (paper/pencil)
• Work history (transferable skills)
• All are used to create a composite profile

32
Worker Qualification Profile

• General Education Development (GED)
• Reasoning Math Language
• Aptitudes
• General Learning, Verbal Aptitude, Numerical
  Aptitude, Spatial Aptitude, Form Perception,
  Clerical Perception, Motor Co-ordination, Finger
  Dexterity, Manual Dexterity, Eye-Hand-Foot
  Co-ordination, Colour Discrimination

33
General Education Development-GED

• Embraces those aspects of education (formal or
  informal) which contribute to a a workers
  reasoning development and ability to follow
  instructions and, the acquisition of tool
  knowledge such as language and mathematical
  skills. (Revised Handbook for Analyzing Jobs -
  1991)

34
GED FACTORS

• Reasoning Development
• Math Development
• Language Development
• 1 to 6 scale
• 1 Low 6 High

35
Aptitudes

• The capacity to acquire proficiency in and
  activity with a given amount of formal or
  informal training
• Aptitudes may be general, such as learning
  aptitude or special, such as mechanical aptitude
  (Pruitt, 1986)

36
Aptitudes

• General Learning
• Verbal Aptitude
• Numerical Aptitude
• Spatial Aptitude
• Form Perception
• Clerical Perception
• Motor Co-ordination

• Finger Dexterity
• Manual Dexterity
• Eye-Hand-Foot Co-ordination
• Colour Discrimination
• 1-5 scale
• 1High 5Low

37
The Tools of Vocational Evaluation

• Interest Surveys
• Academic Achievement and Aptitude Testing
• Computerized Assessment
• Work Samples
• Observation
• Other Surveys

38
Interest Surveys

• Strong Interest inventory (SII)
• Canadian Work Preference Inventory (CWPI)
• Jackson Vocational Interest Survey (JVIS)
• Career Occupational Preference System Interest
  Inventory (COPS)
• Reading Free Vocational Interest Inventory 2
  (R-FVII2)
• Career Assessment Inventory (CAI)

39
AbilityAchievementAptitude

• Achievement Tests - What a person has already
  done!
• Canadian Academic Achievement Test (CAAT)
• Wide Range Achievement Test (WRAT-4)
• Aptitude Tests What a person has the potential
  to do!
• General Aptitude Test Battery (GATB)
• Differential Aptitude Test (DAT)

40
Work Samples

• Simulated work activity
• Criterion Referenced Test (CRT)
• Method Time Measurement (MTM)
• Standardized tasks
• Directly related to aptitude factors
• Valpar has over 30 work samples for functional
  and job specific assessments

41
Valpar Work Samples

• VCWS03 Numerical Sorting
• VCWS06 Independent Problem Solving
• VCWS07 Multi-Level Sorting

42
Norm-Referenced test scores

• Compares the evaluees performance to the
  performance of members of a particular norm group
• Selected norm group population should be well
  defined, relatively current and appropriate to
  the evaluees goal
• Results are reported in terms of percentile
  rankings

43
Norms

• Frequently referred to as moving targets as an
  individuals relative rank can shift
  significantly depending on the norm group
• Pearl scored at the 87th percentile in
  arithmetic (grade 4 students)
• Pearl scored at the 11th percentile in
  arithmetic (grade 10 students)

44
When norms are not enough

• The world of work expects people to be able to
  perform some job tasks based on criterion of
  success or precision rather than compared only to
  how well others perform
• Expect airline pilots to take off and land an
  airplane with 100 level of success, not just a
  little better than their pilot classmates!

45
Criterion references tests scores

• Criterion-referenced tests have a predetermined
  external standard or criterion
• Work related criterion generally come from
  industry standards
• Number of widgets assembled per hour
• Keyboarding speed 60 wpm 0 errors
• MTM (Valpar)
• Measures a specific knowledge, skill, or trait
  and compares persons performance against the
  criterion
• Focus and importance of on performance rather
  than relative rank of how others performed

46
Norm vs criterion

• The Standards for Educational and Psychological
  Testing provide a basis on which to evaluate
  tests. established by APA, National Council on
  Measurement in Education, American Education
  Research Association
• The samples used for test validation and norming
  must be of adequate size and must be sufficiently
  representative to substantiate validity
  statements, to establish appropriate norms, and
  to support conclusions regarding the use of the
  instrument for the intended purpose

47
A Huge Issue---Norm vs. Criterion

• Most norm referenced tests do not use
  representative norm samples that include
  individuals with disabilities
• This often results in labels of disability and
  the identification of weakness
• Important to note that in some instances it is
  helpful to understand how an individual is
  functioning compared to the average population
  when assessing for competitive employment

48
Norms Criterion

• Competitive compares people to other people
• Relative skills reported
• Content difficulty fixed
• High scores inferred from high performance on
  tasks with lower scores

• Non-competitive compares people directly to jobs
• Actual skills reported
• Content difficulty adjusts to evaluees skills
• Scores measured directly at corresponding content
  difficulty

49
Norms Criterion

• Validity depends on appropriateness of norm group
• Strictly valid only for members of the norm group
• Translation from percentiles by traditional
  methods distorts DOT levels

• Validated by Method Time Measurement (MTM). No
  norm groups
• Accuracy verified by data from over thousands of
  assessments
• Measures DOT levels directly no translation
  required

50
Best Norm Group (when available)

• Norm to the environment you are predicting to
• e.g. employed workers, general working
  population, general
• population, education, training norms
• 62nd percentile Form Spatial GATB (GWP)
• 12th percentile MN Paper Form Board (employed
  drafting technicians)
• Obviously, criterion-referenced tests are better
  indicators of performance, so if you have a
  choice between a norm referenced and criterion
  referenced measure choose criterion-referenced.

51
Computerized Assessment

• Bridges Ability Profile Multi-Aptitude
  Assessment
• Bridges Interest Profiler
• Career Cruising - Career Matchmaker
• Valpar Professional (PRO 3000)
• (Magellan and Aviator)

52
PRO3000

• System Manager
• Computerized Assessment (COMPASS) with 3
  specialized work samples
• DOT Descriptions and database
• Work History for transferable skills analysis
• Pictorial/Audio Interest Survey
• Spatial/Non-verbal
• PET Survey
• Norm Referenced Test Conversion (TECO)

53
Suitable Employment Alternatives

• Counsellor directed selection of vocational
  options
• Objective listing based on testing results
• Can be combined with career exploration

54
The Client Perspective

• Generally 3 to 6 hours
• Includes intake session
• Pre-screening
• Academic achievement testing
• Interest / Aptitude Testing
• Discussion of test results

55
VALPAR International

• Started in 1973 in Tucson, Arizona
• Product Developer with over 6000 customers
  world wide
• Component Work Samples Series (VCWS)
• JOULE Functional Capacity Evaluation
• PRO3000 Professional Vocational Assessment
• MAGELLAN Career Self Assessment/Explorer
• AVIATOR Aptitude Interest Assessment
• SIGI PLUS Values-Based Career Planning

56
Career Systems Canada Limited

• Steve Russell
• P. O. Box 255
• Buckhorn, ON. K0L1J0
• tel [phone]
• fax [phone]
• cell 416.721.5759
• steverussell_at_necicom.net
• www.valparint.com

57
P.W. Boswell Associates

• Phil Boswell
• 9534 Riverbend Road
• Black Creek, BC  V9J 1E8
• tel [phone]
• fax [phone]
• toll  877.790.7074
• info_at_employment-training.org
• www.employment-training.org