Seven Scary Problems:

1
Session 207 Seven Scary Problems Overcoming the
issues preventing learning fromearning a seat at
the table Dr. Tony
ODriscoll Performance Architecture Analysis and
Design Group IBM Sales Learning
2
Leaders appear to have become very interested in
cultivating the capability to learn and innovate
in perpetuity..
A companys success no longer depends on its
ability to raise investment capital, but on the
ability of its people to learn together and
produce new ideas Arie De Geus, 2000
3
..this Organizational Learning focus has been
precipitated by a services driven economy that
demands skilled labor
A Services Driven Economy Since 1950,
employment in the manufacturing sector has fallen
from nearly 40 of total employment to less than
18 currently, while the service sector
employment has risen from less than 14 to more
than 35 percent, essentially flip-flopping from
where it had been in 1950. Merrill Lynch, 1999
4
A number of key market and technology drivers
brought about a keen focus on learning

• Market Drivers
• The shift to service driven knowledge enabled
  economy
• The shortening of product/service life cycles
• The increased complexity and velocity of the work
  environment
• The transitory nature of Knowledge Workers
  (Volunteers)
• The inability of traditional institutions to
  acquire skilled resources
• The short half-life of information age skills
• The changing perception of learning from cost to
  investment
• Technology Drivers
• The arrival of the internet and the information
  age economy
• The increasing importance (and spending) on IT
  within all firms
• The ability of the internet to facilitate
  distribution of learning

5
IBM research indicates that innovation and
globalization will drive a resurgent growth
agenda within organizations
Top line revenue growth is back on the CEO agenda
for the next three years

• This growth focus is accompanied by a number of
  organizational, management, talent and
  infrastructure barriers

At the same time, most CEOs also believe that
they must maintain their ongoing emphasis on
cost containment
Growth and differentiation will happen through
great focus and attention on people. 75 of CEOs
interviewed believe that employee education is
critical for future enterprise success.
Source IBM Business Consulting Services, The
Global CEO Study 2004 (n486 CEOs)
6
The Information Age Knowledge Worker
Now that we are moving from factory work to
anytime, anyplace work, we need an anytime
anyplace educational parallel. Alvin Toffler, 2001
7
In a services driven technology focused economy,
access to talent has become a strategic
imperative.
A half-century ago a man could learn how to drive
a tractor and have a job skill that would remain
useful for 40 years or more. Today a person
learns a software program and has that skill
current for, maybe, 18 months. Merrill Lynch, 2000
requiring that companies leverage learning to
reduce time-to-competency of the firms existing
resources
Technology has revolutionized business, now it
must revolutionize learning. Hambrecht, 2000
8
A Fundamental Disconnect
In organization learning efforts the confusion of
learning and training is fatal. Rectifying this
confusion can start with recognizing the limited
role of training and development professionals.
Senge
What Training Offers
What Learners Want
9
Training Today Seven Scary Problems

• The Autonomous Learner Problem
• The Timing Problem
• The Packaging Problem
• The Performance Problem
• The Routinization Problem
• The Transfer Problem
• The Value Problem

10
The Autonomous Learner Problem
As learning migrates from Just in Case to Just
in Time to Just for Me the learner continue
to take more control of their learning and
information needs Merrill Lynch, 2000
Many organizations report that 85-90 of a
persons job knowledge is learned on the job and
only 10-15 is learned in formal training events.
Raybould, 2000
If almost two-thirds of learning in organizations
happens informally, can we afford to leave it to
chance? Peter Henschel, 1998
11
The Timing Problem
The time between original experience in the field
and the time it again reaches the field is, on
average, well over two years Aldrich 2001
Nortel networks products have a life cycle of
nine months. A typical training initiative takes
a year to develop, pilot, refine and roll out
Harris, 2001
The rapid pace of technological innovation and
economic change almost guarantees that formal
learning will be dated Cross, 2001
12
The Packaging Problem
As the speed of business continues to
accelerate, courses will become highly perishable
goods, knowledge workers dont need old theory,
they need new ideas Aldrich 2001
LearningContext
Work Context
Information
Instruction
75 of e-learners fail to complete courses CUE
2001
People dont deal in subjects they deal in work
Cross 2003
13
The Performance Problem
In complex organization systems, most performance
problems are multi causal. Too often training is
asked to assume the responsibility for problems
with root-causes that have very to do with skill
gaps Rummler
s
Person
Environment
Skills and Knowledge
Data
10.5
35.3
Information
Root Cause Performance Levers
Capacity
Tools and Setting
7.5
29
Instrumentation
Motives
Incentives and Rewards
6.3
11.3
Motivation
Source Gilbert and Dean
14
The Routinization Problem
Weve experienced time-and-again how new
technology arrives and paves cow pathsautomating
the past bad assumptions and all
The vast majority of learning investment will be
spent on tools and content to create e-training
and services to automate the existing training
and development processes Merrill Lynch 2001
15
The Transfer Problem
The impact of common training practices is
shockingly small, no more than 20 of investment
in training pays off in transfer to the job
Brinkerhoff and Gill, 1994
At least 90 of American Industrys spending on
training fails to result in transfer to the job
Ford and Weissbein, 1997
16
The Value Problem
The average percentage-of-revenue invested in
learning for Benchmarking Forum members in 2004
was .44
Linkage reports that e-enablement of key training
and development processes could reduce
operational costs by as much as 50.
X
Average Learning Expenditure as Percentage of
Revenue
Cost reduction achieved viaLearning Function
e-enablement
Business Impact ( of Revenue)

.22
50
X

A flawless implementation of technology and tools
to streamline learning design, development and
delivery processes would yield .22 of revenue to
the business.
17
Synthesizing the Problem Space
Innovation
Problem Overlay
Autonomous Learner Timing Packaging
Performance
Performance
Routinization
e-Training
Transfer
Productivity
Value
Formal
Informal
Learning
This analysis begs the following question How
must learning reinvent itself to add strategic
value in the information age organization?
18
A word from Guido
19
Analyzing the Modes of Learning
What I Read I Forget, What I See I Remember, What
I Do I Understand Proverb
Learning has to do with integrating information
into your own internal framework so you own it
within your own conceptual space. Seely Brown
20
Changing the Focus of Learning.
21
5
22
FutureView On Demand Learning in Action
Start
23
Challenging Conventional Wisdom
24
A Manifesto for Learning in the Digital Enterprise

• Think about Learning to Innovate not just to
  Imitate Change your paradigm from training to
  learning
• Think Business Value over Training Volume
  Clearly define how learning addresses
  business need
• Think Business Needs over Learning NeedsDesign
  your learning strategy at the enterprise level
  and link to the firms most pressing business
  needs
• Think System Flexibility over FunctionalityDevel
  op flexible learning architecture models that can
  quickly adapt to changing business priorities
• Think process execution and employee development
  Use these flexible architectures to enable
  learning while doing rather than requiring
  learning as a precondition to doing

25
A Manifesto for Learning in the Digital Enterprise

• Think Collaboration not CoercionEmulate the
  characteristics of e-mail with your learning
  infrastructure.no-one controls, everyone
  participates
• Think Technology Does not Equal Strategy Put
  technology in its place, it is a delivery
  mechanism not a panacea
• Think of Learners as Clients rather than
  CaptivesMove creative control and access the
  learner
• Think of presenting Information and Instruction
  Know the difference between explicit and tacit
  knowledge and know which instructional strategies
  work best
• Think Just In Time over Just in CaseMake
  intervention selections as if you have to pay
  employees for their time to attend your
  interventions

26
A Manifesto for Learning in the Digital Enterprise

• Think Informal Community and Formal
  CurriculumRecognize that there is more than one
  way to learn and that learning would continue to
  thrive without us
• Think Learning While Doing over Learning Before
  DoingIntegrate information and learning into a
  more authentic context
• Think Customer, Partner, and SupplierRecognize
  that there is more than one learning audience in
  an increasingly connected economic ecosystem

27
My LAST Fundamental Question
A Problem is Always the Simplest the First Time
we See it Chopra
The Problems we Face Today Cannot be Solved with
the Same Level of Thinking that Created Them
Einstein
28
Additional References
Learning Circuits Articles Whats Your Web-Based
Learning Strategy? http//www.learningcircuits.co
m/feb2000/odriscoll.html Adapt or Die The
Strategic Role of Learning in the Digital
Enterprise http//www.learningcircuits.org/2004/ma
y2004/odriscoll-briki.htm My Book Achieving
Desired Business Performance www.ispi.org
29
The ubiquity of Kirkpatricks taxonomy in WLP
evaluation is clear
In 1959 and early 1960, Donald Kirkpatrick
published a series of four articles where he
outlined four categories of measures of the
effectiveness of training outcomes
Reduction of costs reduction of turnover,
improved morale
Results
Using learning principles and techniques on the
job
Behavior
Principles, facts, and techniques understood and
absorbed
Learning
Trainees liking of and feelings for a training
program
Reaction
There is much evidence to suggest that
Kirkpatricks taxonomy has dominated the training
evaluation discussion since it was first
published over forty years ago.
30
However, a number of issues have been raised
regarding Kirkpatricks taxonomy

• The Kirkpatrick framework was not derived from
  studies of evaluation practices. It was
  originally offered as a prescriptive evaluation
  framework. Thus it is important to recognize that
  it represents a trainers notion of what
  constitutes effective evaluation, not a business
  managers
• The second issue deals with hierarchy, the
  assumption that each level has more value than
  the previous one. This can lead to a perception
  of level 4 results, as invariably being the best
  measure since it is highest in the hierarchy.
• The third issue deals with causality, the
  assumption that each level is caused by the
  previous one. This assumption is significant as
  it prompts evaluators to begin from the program
  and work backwards through the levels towards the
  business value
• The fourth issue has to do with completeness, the
  assumption that the Kirkpatrick taxonomy
  completely addresses learning valuation.
  Kirkpatricks taxonomic model is conceptually
  flawed as it does not fully identify all the
  constructs underlying the phenomena of interest.
  Furthermore, the results provided by the model
  can lead to incorrect decisions since major
  intervening variables are not included.

Given this analysis it is clear that, while
gaining in popularity, Kirkpatricks approach to
evaluating learning falls short on a number of
counts . ODriscoll
Source Valuing Human Capital and WLP A
Literature Review. (2006).
31
These issues might explain the lack of consistent
use of Kirkpatricks approach in business and
industry
An analysis of ASTDs State of the Industry
Reports from 1999 to 2003 reveals that, on
average, 79 of programs are evaluated at Level
1, 36 at Level 2, 15 at Level 3, while only 8
of programs are evaluated at the results or
impact level
Despite the prominence and simplicity of the
Kirkpatrick taxonomy, the persistent low levels
of training evaluation, particularly at the more
sophisticated levels 3 and 4 raises serious
questions about the current state of training
evaluation in an era when increased
accountability for learning investments has
become the norm.
Source Valuing Human Capital and HRD A
Literature Review. (2006), ASTD State of the
Industry Report (2004)
32
Further research suggests that the primary reason
HRD evaluation is conducted is because it is not
seen as useful by the organization
The top reason training is not evaluated is the
same for all four levels It is not required by
the organization
In short, it could be posited that the ubiquity
of the Kirkpatrick taxonomy within the HRD
profession is a key limiting factor in the
professions ability to demonstrate its value
beyond the transaction/program level
Source Valuing Human Capital and HRD A
Literature Review. (2006), ASTD State of the
Industry Report (2004)
33
Kirkpatrick himself suggests we may have put the
cart before the horse in applying the taxonomy
Take the four levels and work backwards. What
business results are you are looking for? What
behaviors are needed to accomplish those results?
What knowledge, skills and attitudes do people
need in order to behave in that way? And how can
we do it in such a way that they will react to it
favorably? Answering those questions in that
order is at least one less headache. Don
Kirkpatrick (2004)
Learning InvestmentPerspective
Program Justification Perspective
How can we present it in such a way that
students will react favorably? What knowledge,
skills or attitudes do people need to behave in
that way? What behaviors are needed to
accomplish those results? What results are we
looking for?
Did the students Like the program? Did they
learn? Are they applying whatthey learned on
the job? How does this applied learning impact
the business?
Reaction
Learning
Training Out
Behavior
Business In
Results
34
The ROI approach monetizes the value of learning
investments
Jack Phillips pioneer in the field of HRD ROI.
His specific contributions is widely recognized
as a include developing a process model around
Kirkpatricks taxonomy, bringing standards to
that process, and adding a fifth level that
focused specifically on ROI.
Net Program Benefits
X 100
ROI
Program Costs
ROI couches results and expectations in the
language of business and is often a hurdle a
project must clear in order to secure funding.
Given that learning is increasingly perceived as
an investment by the organization, it seems only
natural that it should be evaluated in the same
way as other large investments, in terms of costs
and benefits
Source Valuing Human Capital and HRD A
Literature Review. (2006).
35
However, a number of issues have been raised
regarding WLP ROI

• An WLP intervention is a process that cannot be
  accurately measured via the current accounting
  mechanisms because it involves dynamic human
  behaviors.
• The traditional ROI formula was often used to
  measure avoided cost or waste, not potential
  added value. Similarly, learning ROI evaluations
  are almost entirely oriented towards past project
  or program performance and shed little light on
  future human capital investment decision-making
  by management.
• The impact of learning on final business results
  often interacts with factors at organizational
  and environmental levels, such as culture and
  market conditions. Current ROI methods overlook
  the complexity of WLP and non-WLP interactions
  and use a simplified approach to isolate the
  effect of WLP
• The benefit of an WLP intervention may take on a
  value other than monetary. Non-monetary returns
  apply to any other returns that have economic
  impact on the business but may not be explicitly
  in dollar values. The purpose of some WLP
  programs is to change participant behavior which
  can rarely be measured in dollars.
• ROI does not provide the kind of information
  needed by business decision-makers.

Given this analysis it is clear that, while
gaining in popularity, the current ROI approach
to evaluating learning falls short on a number of
counts . ODriscoll
36
These issues might explain the lack of use of the
ROI approach
Twitchell, Holton and Trott (2000) in an in-depth
analysis of evaluation practices in technical
training in the United States, found that only
19 of organizations were attempting to use ROI,
and then only in a small percentage of their
programs
In fact, ASTDs 2005 State of the industry
reports that only 2 programs within their
Benchmarking Forum group underwent an ROI
analysis
People are attracted to ROI because they think it
is the answer to all accountability issues. We
have to keep reminding people that ROI is only
one measure that demonstrates the value of
training. Dont overreact to it or overuse it.
Jack Phillips
Source Valuing Human Capital and WLP A
Literature Review. (2006), ASTD State of the
Industry Report (2005)
37
Revisiting the purpose of WLP evaluation
Is the purpose to prove that a specific WLP
intervention worked or is it to transform the
whole learning process and significantly raise
the probability of an WLP investment providing a
good return?
Prove WLP Intervention Worked
Optimize WLP Investment
What is the Role of WLP Evaluation?
The purpose of Evaluation is not merely to prove
the value of past investments but, more
importantly to improve the probability of an WLP
investment providing a good return to the
business.
38
WLP Evaluation change on a number of fronts in
order to add strategic value
You get what you measure. It is clear that if the
WLP profession does not shift its evaluation
strategy to a more participatory, proactive and
value perception based approach, WLP budgets will
decrease, WLP practitioner frustration will
increase and executive interest in the potential
for learning to add strategic value to the
enterprise will wane.
To
From
Prove Trainings Value
Improve the Productivity and Sustainability of
the Enterprise
Purpose
Organizational Learning Process
Evaluand
Training Event
Up-Front WLP Investment Optimization
Post-Hoc Program Investment Justification
Emphasis
WLP Function Survival
Enterprise Survival
Desired Outcome
In this transformed WLP approach, the evaluation
process itself becomes the primary WLP process
and the primary role of WLP professionals will to
manage the process such that it yields optimal
WLP investment outcomes for the business.