Valuing Learnings Contribution

1
Session 305 Valuing Learnings Contribution
Dr. Tony ODriscoll Performance
Architecture Analysis and Design Group IBM
Learning
2
What We Offer
3
A Fundamental Disconnect
In organization learning efforts the confusion of
learning and training is fatal. Rectifying this
confusion can start with recognizing the limited
role of training and development professionals.
Senge
What Training Offers
What Learners Want
4
In Search of Value
5
Quantifying the disconnect between learning
investment and HRD evaluation
62B
If America has spent ____________on Training
programs in 2003 And, only ____ of programs were
evaluated at the impact level
8
How was the other 57B Valued?
With over 1 trillion invested in HRD since
Kirkpatricks taxonomy was introduced, Twitchell,
Holton and Trott (2000) ask Is it truly possible
that business and industry would spend this much
money on HRD without verifying the value of what
they purchased?
Source Valuing Human Capital and HRD A
Literature Review. (2006), ASTD State of the
Industry Report (2004)
6
IBM research indicates that innovation and
globalization will drive a resurgent growth
agenda within organizations
Top line revenue growth is back on the CEO agenda
for the next three years

• This growth focus is accompanied by a number of
  organizational, management, talent and
  infrastructure barriers

At the same time, most CEOs also believe that
they must maintain their ongoing emphasis on
cost containment
Growth and differentiation will happen through
great focus and attention on people. 75 of CEOs
interviewed believe that employee education is
critical for future enterprise success.
Source IBM Business Consulting Services, The
Global CEO Study 2004 (n486 CEOs)
7
This research asked CXOs and CLOs how the
learning function should respond to business
needs and be held accountable for adding value
LEARNING FUNCTION RESPONSE
LEARNING FUNCTION PERFORMANCE
CLO
CXO
How should Learning evolve to align with the
changing needs of the business?
What indicators, measures and metrics demonstrate
Learnings value to the business?
CEO, COO, CTO CFO, BU Head
CLO, VP Learning
8
ASTD Strategic Value of Learning Study
How does the learning function provide strategic
value to your organization?
9
Q1 Response Framework
Alignment
Variance
Frequency Counts
Framework Groupings
10
Q1 Strategic Value - Representative Quotes
Business UnitProductivity

Learning is the major investment to build the
capability to drive the business forward.
11
Q1 Strategic Value - Additional Quotes
12
Q1 Strategic Value - Insights and Implications

• CXOs see learnings strategic value as building
  the strategy and capability necessary to address
  the future challenges of the enterprise
• CLOs are focused on moving beyond learning to
  focus on performance and talent issues related to
  the current needs of business units

Insights

• CLOs should develop learning strategies,
  approaches and infrastructures that can
  simultaneously respond to enterprise level
  strategic needs, business unit operational needs
  and individual development needs
• CXOs should articulate where they think learning
  can add strategic value at the enterprise level
  and discuss how this aligns with business unit
  and individual capability needs

Implications
13
Shameless Plug For more detail on this study
read thearticle in the October edition of TD
14
Learning ValueFramework
15
Learning adds value at three discrete needs
levels
Business Outcomes
The Learning Value Framework
Strategic Needs
Sustainability
Operational Needs
Productivity
Developmental Needs
Capability
Organizational Scope
16
Framework coverage yields optimal learning value
Developmental Needs
Operational Needs
Strategic Needs
Leadership
Skills
17
Governance Examples
18
Learning governance progresses through three
discrete phases as it migrates to becoming a
strategic lever for the enterprise
19
Our study also revealed that each value
proposition varies interms of Alignment,
Management and Measurement
Alignment How the Learning Function aligns with
the strategic needs of the business
Management How the learning investment is managed
to deliver optimal value to the business
Articulating Learnings Business Value
Contribution
Measurement How the Learning Function
demonstrates its value contribution to the
business
20
Applying the Framework
21
On the Levels
22
The ubiquity of Kirkpatricks taxonomy in WLP
evaluation is clear
In 1959 and early 1960, Donald Kirkpatrick
published a series of four articles where he
outlined four categories of measures of the
effectiveness of training outcomes
Reduction of costs reduction of turnover,
improved morale
Results
Using learning principles and techniques on the
job
Behavior
Principles, facts, and techniques understood and
absorbed
Learning
Trainees liking of and feelings for a training
program
Reaction
There is much evidence to suggest that
Kirkpatricks taxonomy has dominated the training
evaluation discussion since it was first
published over forty years ago.
23
However, a number of issues have been raised
regarding Kirkpatricks taxonomy

• The Kirkpatrick framework was not derived from
  studies of evaluation practices. It was
  originally offered as a prescriptive evaluation
  framework. Thus it is important to recognize that
  it represents a trainers notion of what
  constitutes effective evaluation, not a business
  managers
• The second issue deals with hierarchy, the
  assumption that each level has more value than
  the previous one. This can lead to a perception
  of level 4 results, as invariably being the best
  measure since it is highest in the hierarchy.
• The third issue deals with causality, the
  assumption that each level is caused by the
  previous one. This assumption is significant as
  it prompts evaluators to begin from the program
  and work backwards through the levels towards the
  business value
• The fourth issue has to do with completeness, the
  assumption that the Kirkpatrick taxonomy
  completely addresses learning valuation.
  Kirkpatricks taxonomic model is conceptually
  flawed as it does not fully identify all the
  constructs underlying the phenomena of interest.
  Furthermore, the results provided by the model
  can lead to incorrect decisions since major
  intervening variables are not included.

Given this analysis it is clear that, while
gaining in popularity, Kirkpatricks approach to
evaluating learning falls short on a number of
counts . ODriscoll
Source Valuing Human Capital and WLP A
Literature Review. (2006).
24
These issues might explain the lack of consistent
use of Kirkpatricks approach in business and
industry
An analysis of ASTDs State of the Industry
Reports from 1999 to 2003 reveals that, on
average, 79 of programs are evaluated at Level
1, 36 at Level 2, 15 at Level 3, while only 8
of programs are evaluated at the results or
impact level
Despite the prominence and simplicity of the
Kirkpatrick taxonomy, the persistent low levels
of training evaluation, particularly at the more
sophisticated levels 3 and 4 raises serious
questions about the current state of training
evaluation in an era when increased
accountability for learning investments has
become the norm.
Source Valuing Human Capital and HRD A
Literature Review. (2006), ASTD State of the
Industry Report (2004)
25
Further research suggests that the primary reason
HRD evaluation is conducted is because it is not
seen as useful by the organization
The top reason training is not evaluated is the
same for all four levels It is not required by
the organization
In short, it could be posited that the ubiquity
of the Kirkpatrick taxonomy within the HRD
profession is a key limiting factor in the
professions ability to demonstrate its value
beyond the transaction/program level
Source Valuing Human Capital and HRD A
Literature Review. (2006), ASTD State of the
Industry Report (2004)
26
Kirkpatrick himself suggests we may have put the
cart before the horse in applying the taxonomy
Take the four levels and work backwards. What
business results are you are looking for? What
behaviors are needed to accomplish those results?
What knowledge, skills and attitudes do people
need in order to behave in that way? And how can
we do it in such a way that they will react to it
favorably? Answering those questions in that
order is at least one less headache. Don
Kirkpatrick (2004)
Learning InvestmentPerspective
Program Justification Perspective
How can we present it in such a way that
students will react favorably? What knowledge,
skills or attitudes do people need to behave in
that way? What behaviors are needed to
accomplish those results? What results are we
looking for?
Did the students Like the program? Did they
learn? Are they applying whatthey learned on
the job? How does this applied learning impact
the business?
Reaction
Learning
Training Out
Behavior
Business In
Results
27
Netting it out
28
Revisiting the purpose of WLP evaluation
Is the purpose to prove that a specific WLP
intervention worked or is it to transform the
whole learning process and significantly raise
the probability of an WLP investment providing a
good return?
Prove WLP Intervention Worked
Optimize WLP Investment
What is the Role of WLP Evaluation?
The purpose of Evaluation is not merely to prove
the value of past investments but, more
importantly to improve the probability of an WLP
investment providing a good return to the
business.
29
WLP Evaluation change on a number of fronts in
order to add strategic value
You get what you measure. It is clear that if the
WLP profession does not shift its evaluation
strategy to a more participatory, proactive and
value perception based approach, WLP budgets will
decrease, WLP practitioner frustration will
increase and executive interest in the potential
for learning to add strategic value to the
enterprise will wane.
To
From
Prove Trainings Value
Improve the Productivity and Sustainability of
the Enterprise
Purpose
Organizational Learning Process
Evaluand
Training Event
Up-Front WLP Investment Optimization
Post-Hoc Program Investment Justification
Emphasis
WLP Function Survival
Enterprise Survival
Desired Outcome
In this transformed WLP approach, the evaluation
process itself becomes the primary WLP process
and the primary role of WLP professionals will to
manage the process such that it yields optimal
WLP investment outcomes for the business.
30
Top Nine List on the Strategic Value of Learning
Lessons Learned

• Learning investment decisions are made based on
  perceived future value on the part of the
  stakeholder
• The value expectation behind the investment
  decision provides guidance on how to align,
  govern and measure learning
• One alignment approach does not fit all value
  expectations
• One governance approach does not fit all value
  expectations
• One measurement approach does not fit all value
  expectations
• Up-front strategic alignment trumps back-end
  justification measurement
• Developmental needs rely more heavily on
  perception data
• Operational needs rely more heavily on business
  outcome/human capital data
• Strategic needs rely on both perception and
  business results data

31
The Dawn of a New Era for Learning Evaluation?
X
RO
I
E
xpectations