Demonstrating the Impact of Career Development Services in Today's Labour Market

1
Demonstrating the Impact of Career Development
Services in Today's Labour Market

• Linking Intervention with Outcome Building
  Cause and Effect Arguments
• Bryan Hiebert
• University of Victoria
• University of Calgary

2
Overview 4 main points

• Expand the scope of accountability
• A framework for demonstrating value
• What are the legitimate outcomes resulting from
  career development services
• Economic benefits of career development services

3
Focus on Accountability

• Traditionally, focus was on outcomes
• Are the outcomes (results, effects, etc.)
  promised, achieved in practice?
• Need to broaden the focus
• Are the outcomes achievable, realistic, and
  relevant?
• What factors contribute to the outcomes?
• What interventions make the outcomes more likely
  to be achieved?
• How do we know that the outcomes are a result of
  the interventions?

4
Clicker Question

• Do you agree that accountability needs to address
  BOTH the outcomes of the services we provide AND
  ALSO the reasons why those outcomes occurred?
• Yes
• No

5
A Challenge by Canadian Policy Makers

• You havent made the case for the impact and
  value of career development services

6
How to address the problem

• We need an approach that is
• Comprehensive enough to include what is needed
• Simple enough for people to use
• The ideal is to incorporate evaluation into
  practice

7
Outcome Focused Evidence-Based Practice
Input?Process?Outcome
Framework developed by the Canadian Research
Working Group on Evidence-Based Practice in
Career Development
8
Outcome Focused Evidence-Based Practice
Input?Process?Outcome
Indicators of client change 1. Learning
outcomes 2. Personal attribute outcomes 3. Impact
outcomes
9
Outcome Focused Evidence-Based Practice
Input?Process?Outcome

• Indicators of client change
• Learning outcomes
• Knowledge and skills linked to intervention
• Personal attribute outcomes
• Changes in attitudes,
• Intrapersonal variables (self-esteem,
  motivation, independence)
• Impact outcomes
• Impact of 1 2 on clients life, e.g.,
  employment status, enrolled in training
• Societal and relational impact
• Economic impact

10
Outcome Focused Evidence-Based Practice
Input?Process?Outcome

• Activities that link to outputs or deliverables
• Generic interventions
• Working alliance, microskills, etc.
• Specific interventions
• Interventions used by service providers
• Skills used by service providers
• Home practice completed by clients
• Programs offered by agency
• To reduce employment barriers
• To enhance client employability
• Involvement by 3rd parties
• Quality of service indicators
• Stakeholder satisfaction

11
Outcome Focused Evidence-Based Practice
Input?Process?Outcome

• Activities that link to outputs or deliverables
• Specific interventions
• Career decision making
• Work-specific skills enhancement
• Work search
• Job maintenance
• Career-related personal development
• Other

12
Outcome Focused Evidence-Based Practice
Input?Process?Outcome

• Resources available
• Staff
• Number of staff, level of training, type of
  training
• Funding
• Budget
• Service guidelines
• Agency mandate
• Facilities
• Infrastructure
• Community resources

13
Outcome Focused Evidence-Based Practice
Quality Improvement of Services
Input?Process?Outcome

• Counsellor
• Skills
• Interventions
• Programs

Resources

• Client change
• Knowledge
• Skill
• Attribute
• impact

14
Clicker Question

• Does this framework for evaluating service
  delivery (Inputs ? Processes ? Outcomes) make
  sense to you?
• Yes
• No

15
Back to the TitleBuilding cause and effect
arguments

• What counts?
• Career-life plan
• Increased self-confidence
• Greater capability to manage career-life plan
• Clients taking action as set out in their
  career-life plan
• Services delivered match client identified needs
• Use of resources
• How closely do service provider and client follow
  program
• How often is career-life plan modified
• Change in employment status
• Socio-economic impact
• How do we document these?

Some of these require long term follow up
16
Evidence-based Practice

• Research trials (traditional way in psychology)
• Professional Practitioner
• Purposeful intervention
• Data ( evidence) on what was done
• Evidence on client change
• Look for patterns in data linking intervention
  with outcome
• Develop scientific attitude toward practice
• Skepticism, curiosity, inquiry
• This is a viable, perhaps even preferable,
  alternative
• Alternative scientific way to demonstrate
  effectiveness

17
Outcome Focused Evidence-Based Practice

• Input?Process?Outcome

Need to link process with outcome

• What will I do?
• What are the expected client changes?
• What client learning will result?
• What sorts of personal attributes will the client
  acquire?
• What will be the impact on their lives?
• How will I tell?

18
Outcome-Focused Evidence-Based Practice

• Input?Process?Outcome

Intervention Process Outcome
What will I do? How is it working?
Professional Practitioner
19
Clicker Question

• In tracking the outcomes of career development
  services, which of the following is it important
  to view as legitimate outcomes
• Knowledge and/or skills acquired in areas related
  to job placement, such as job search skills, job
  maintenance skills, etc.
• Changes in personal attributes, such as
  motivation, self-confidence, anger management,
  stress control, etc.
• Change in employment status, e.g., obtaining
  employment, enrolling in training or upgrading
  programs
• All of the above
• None of the above

20
Integrating evaluation and service deliveryWhat
we do defines who we are

• A Framework for evaluation
• Learning from the evaluation experts
• Gathering evidence of our impact
• Documenting what works and why
• Developing evidence gathering tools

21
Assessment as Decision Making (vs. Judgement)

• Please use a two-step process
• Would you say that your level of mastery of the
  attribute under considerations is
• Then assign the appropriate rating
• 0 really quite poor
• 1 just about OK, but not quite
• 2 OK, but just barely
• 4 really very good
• 3 in between barely OK and really good

unacceptable
acceptable
0
1
2
3
4
0
4
22
Problem with skill self-assessment

• Participants asked to rate their skill (or
  knowledge) before and after a program
• Often, pre-workshop scores are high and
  post-workshop scores are lower
• People find out as a result of the workshop that
  they knew less than they thought or had less
  skill than they thought
• Based on the new awareness, post-scores are lower
• People dont know what they dont know
• How can we get around this problem?

23
Assessing Learning Attribute Outcomes

• Post-Pre Assessment
• We would like you to compare yourself now and
  before the workshop. Knowing what you know now,
  how would you rate yourself before the workshop,
  and how would you rate yourself now?
• Please use a two-step process
• Decide whether the characteristic in question is
  acceptable or unacceptable, then
• assign the appropriate rating

24
Promising Practices

• Several projects have been able (or will be able)
  to provide a solid link between programs
  services and client change
• Provide some examples
• Encourage you to think about how these ideas
  could be applied in your workplace

25
Partnership Workshop Evaluation

• Of the 68 ratings (4 questions times 17 people)
• 18 ratings changed from unacceptable to
  acceptable.
• All now ratings are in the acceptable range.

26
Applied Career Transitions Program(on-line
program for unemployed university grads)

• For Module 1
• All together there were 10 (items) x 29
  (participants) 290 ratings
• Pre 144 Unacceptable Ratings Post 3
  Unacceptable Ratings
• Unacceptable Ratings decreased from 50 to 14
• Pre6 Exceptional Ratings Post130 Exceptional
  Ratings
• Exceptional Ratings increased from 2 to 44 of
  the participants

27
Results Impact outcomes

• Module 1
• 23 out of 29 had found a job
• 10 of the jobs lined up well with career vision
• Module 2
• 4 out of 6 had found a job
• 3 of the jobs lined up well with career vision

28
Attribution for Change

• To what extent would you say that the changes
  depicted above were the result of completing
  Module 1 of the ACT program, and to what extent
  were they a function of other factors in your
  life?

29
Linking Process and Outcome

• We have evidence that the facilitator followed
  the program
• We have evidence about which participants
  followed the program more (vs. less) closely
• We could compare the results of those who follow
  the program compared to those who did not
• We have evidence of how much change took place
• We have a connection
• facilitator process ?client engagement ? outcomes
• Participants attribute change to the program
• A solid link between program and outcome

Learning
Attribute
Impact
30
Multiple Impact

• Skill training for government career
  practitioners
• Input?Process?Outcome

Curriculum development Trainers Staff release
time
Seminars Reading Skill practice
Knowledge test Skill demonstration
Training Phase
Work search skills Employment status Increased
self-confidence Increased job satisfaction
Implementation Phase
Use new knowledge Use skills with clients
31
Employment equivalence

• Consider a client who receives careers
  counselling and
• Decides to return to school so he can
• Find a better job that pays more money with less
  likelihood of unemployment
• Employment status does not change

32
Employment equivalence

• Consider a client receives careers counselling
  and
• Decides to return to school so he can
• Find a better job that pays more money with less
  likelihood of unemployment
• Employment status does not change
• In Canada, men 30 years old are 34 more likely
  to be employed if they have high school
  education(compared to men with no high school
  diploma)
• Employment equivalence for taking training is .34
• Consider also, men described above earn on
  average 6,000 more money per year
• Represents a return on investment
• This is evidence of success

33
Return on Investment for Post Secondary Student
Services

• Post secondary leavers vs. completers
• 50 more likely to have difficulty keeping up
  with the work load
• leavers reported being unsure of what they wanted
  to do,
• 1 reason for leaving school was lack of fit.
• Completers were 50 more likely to report having
  a career plan that was a good match for their
  program
• PSE graduates earn on average 6,500 more, return
  on investment is 3,250 per person per year.

34
Possible Employment Equivalences
35
Question to ponder

• Employment equivalence gt 1
• If the goal is employment, job 1
• A good job with prospects for permanency and
  advancement should contain a bonus
• Consult tables of labour turnover for various
  occupations.
• Turnover for labourer might happen every 6 months
• For other categories it might be, say, 12 months
• People getting low level jobs would get an
  equivalent value of 1 and the latter an
  equivalent level of 2. 
• What do you think of this idea?

36
Clicker Question

• Is employment equivalence a worthwhile construct
  to pursue?
• Yes
• No

37
Professional Identity What we do defines who we
are

• Most practitioners define their job as delivering
  services
• So they do not evaluate the impact of their
  services on clients
• What is career guidance all about?
• Needs to include BOTH process and outcome
• What will I do to facilitate client change?
• How well is it working?

38
To demonstrate value, we need to develop

• Culture of evaluation We need to reach the
  state where
• Identification of outcomes is an integrated part
  of providing services
• Without efficacy data, career services are
  vulnerable
• It is in our best interest to gather evidence
  attesting to the value of the services we provide
• Measuring and reporting processes and outcomes is
  integrated into practice
• Outcome assessment is a prominent part of
  counsellor education
• Reporting processes and outcomes is apolicy (and
  funding) priority
• This needs to be a priority in all sectors

39
Summary

• Counsellors and agencies think it is important to
  evaluate their work with clients
• BUT they dont do it
• They need a framework for integrating evaluation
  and intervention
• What works and why
• They need tools and resources for both short
  term and long term effects
• We need an evaluation culture
• A professional identity that defines our role as
  BOTH delivering services assessing outcomes

40
Dont Ever

• Give Up

41
Discussion

• How would you see these concepts being
  implemented in your workplace?
• How could these concepts be used to enhance the
  quality of services offered?
• How might funder and project monitors incorporate
  these concepts in order to provide greater
  assurance of quality control and accountability?
• Other questions, comments, suggestions?