How to Get Started with Learning Analytics

1
How to Get Started with Learning Analytics

• Jim Everidge

2
Who Will Benefit?

• Those organizations that have an imperative to
  begin the process of proving the value of
  learning
• Those professionals that are trying to formulate
  a strategy around Measurement and need additional
  data to develop that strategy
• Vendors that have training products that are
  charged with helping their customers connect the
  value of the products to the customers business

3
Agenda
1. Background Approaches
2. Learning Analytics Maturity
3. Developing a Vision
4. Elements of Project Success
5. Project Evolution
6. Case Studies
4
Learning Analytics Approaches

• Opinion-based data
• Relies on collecting data about perceptions of
  impact of learning
• Challenge - Hawthorne Effect
• Harder to defend when the result is a change in
  business process
• Operations-based data
• Focuses on the correlation of learning data and
  business data
• Challenge - Data cleanliness confounds
  correlative ability
• Need to understand impacts on data

5
Business Intelligence

• "The oft-quoted example of what data mining can
  achieve is the case of a large US supermarket
  chain which discovered a strong association for
  many customers between a brand of babies nappies
  (diapers) and a brand of beer.
•  
• The explanation goes that when fathers are sent
  out on an errand to buy diapers, they often
  purchase a six-pack of their favorite beer as a
  reward."

Financial Times of London February 7, 1996
6
Measurement
If you cant measure it, you cant manage it.
Nolan Norton Consultants Founders of the
Balanced Scorecard
Driven by demand for rapid content deployment,
plus growing interest in value-added modules like
training analytics and competency management, the
market for e-learning infrastructure systems from
U.S.-based vendors is expected to grow 12 in
2004 to 529.4 million
Simba Information 1/9/2004
7
History of Business Intelligence

• Originally conceived as Data Warehousing Data
  Mining
• Now called Business Intelligence (coined by
  Howard Dresner of Gartner Group, 1994)
• Dresner defined BI A generation of software
  that allows corporations to accelerate the rate
  at which managers can physically process
  information
• Traditionally expensive and hard to do
• Today available to everyone - Techniques, Best
  Practices, Tools
• Includes data integration, analysis, reporting,
  and data visualization

8
Current Practices

• Better Techniques
• Technology allows for data integration
• Purchase pre-defined configurations
• Best Practices
• Focus on business metrics
• Group and filter data
• Drill up and down
• Create visually expressive charts
• Better Tools
• Microsoft Business Intelligence Platform
• 9 Tools one of which is Microsoft Office
  Professional

9
OLAP

• On Line Analytical Processing
• Allows the user to interact with the data
• Multi-dimensional analysis
• Drill up or down through various dimensions
  characteristics of the data that you are looking
  at
• Contrasted with
• Standard SQL Reports
• One time setup
• Choose parameters
• Static results at a moment in time

10
Why Do Learning Analytics?

• Turn Data into Information
• Measure Learning Effectiveness
• Learning Activity
• Catalog Effectiveness
• Total Cost of Learning
• Manage Compliance
• Understand Business Impact
• Focus Strategic Alignment Initiatives
• Using Business Metrics as your guide
• Allows alignment of learning with strategy

11
Expected Results

• Correlation of business data and learning
  intervention data
• Use correlations to driver operational changes
• Incremental skills in getting the learning team
  to understand operational data
• Incremental skills in getting the operations
  group to understand learning data

12
What NOT to Expect

• Individual prescription based on individual
  results
• Contrast with Performance Management where
  individual performance contributes to the greater
  performance metrics
• Absolute certainty cause effect

13
Correlation of Business Learning Data
Determines
Effectiveness of Learning Experience
Determines
Learning Experiences
14
Learning Analytics Maturity

• Level 1 - Influence Individual Action
• Just starting to collect information about
  Learning Experiences
• Requires tools like LMS or equivalent
• Need to template Business Questions so that the
  right data is collected from the outset
• Level 2 Understanding the Business
• Acquired tools to collect data but early in the
  process
• Trying to understand what Business Metrics have
  value in the organization
• Begin to draft Vision for Learning Analytics
• Level 3 Questioning Effectiveness
• Has collected learning data for 6 months have a
  sense of the Business Metrics that have a
  reliable correlation to Learning Experiences
• Ready to implement Vision for Learning Analytics

15
Documenting the Vision

• Business Opportunity
• Provides context for the initiative(s)
• Includes a Vision Statement
• Benefits Analysis
• Solutions Concept
• Roadmap for initiative(s)
• Analysis -gt Risk, Feasibility, Usability,
  Performance
• Solutions Design
• Proposed Technical Architecture
• Initial Project Scope
• Provides range of features/functions
• Defines out of scope
• Criteria for success

16
Vision Template

• Send an email request for document
• Commitment to provide feedback to first cut at
  Vision document

17
Elements of Project Success
Business Sponsors
Cross Functional
Business Reps
Text
Lots of Data
Learning Analytics Projects
Meta- Data
Skilled Staff
Iterate Projects
Clean Data
WBS
Business Analysis
Text
Computerworld White Paper Shaku Atre, Atre Group,
Inc. 2004
18
Project Team Composition

• Business Executives
• Customers
• External business partners

• Learning
• Finance
• Marketing
• Sales
• IT
• Operations

19
Involving Business Sponsors/Execs

• Understand the value of the project remove
  political barriers
• Focus the initiative to a specific set of
  business questions manage the scope
• Initiate a data-quality campaign within their
  organizations
• Periodic project reviews

20
Iterate Projects

• Develop a Clear Vision
• Go through a Readiness Assessment Exercise
• Operationalize your Learning Analytics
• Integrate Business Analytics

21
Readiness Assessment

• Focus is to define initial cut at Business
  Questions
• Identify Business Owners and Involvement
• Identify Process Outputs and Users
• Identify logical Data Sources and availability
• Identify iterative projects
• Prioritize iterative projects
• Develop SOW for Operationalizing Learning
  Analytics Engine

22
Operationalizing Learning Analytics

• Identify Learning questions
• Identify sources/uses of data
• Validate data integrity
• Install analytics server
• Validate ETL Schema adjust as necessary
• Set up Template Reports that address initial
  questions
• Weekly reviews of reports and opportunities

23
Integrating Business Analytics

• (Assumes Learning Analytics engine is operational
  and OLAP Analysis on learning data is being done)
• Identify Business questions
• Identify sources/uses of different data sets
• Validate data integrity
• Determine ETL schema adjust as necessary
• Validate ETL schema
• Set up Template Reports that address initial
  questions
• Weekly Reviews of reports and opportunities

24
Uses of Information

• Adjust Program Design
• Improve Program Delivery
• Influence Application Impact
• Enhance Reinforcement for Learning
• Improve Management Support for Learning
• Improve Satisfaction with Stakeholders

• Recognize Reward Participants
• Justify or Enhance Budget
• Develop Norms or Standards
• Reduce Costs
• Market Learning Programs

Phillips, Phillips, Hodges Make Training
Evaluation Work ASTD, 2004
25
Case Study Learning Analytics

• Profile
• Packaging Shipping
• 1100 Retail Centers
• 18,000 Employees
• 250,000 Hours Training for new business
  orientation
• Intervention
• 3 different certifications assigned by job role
• 12-14 modules for each certification
• Had to be complete in 8 weeks

• Results
• 1100 Concurrent users of the learning content
• 300 of these were simply running reports
• Allowed for self-service reporting analysis
• Challenges
• Had to teach some level of application
  proficiency to Retail Store managers
• Field support went to regional HR managers (not
  IT Help Desk)

26
Case Study Business Analytics

• Profile
• Telecomm
• 16 Call Centers
• New product rollouts happening quickly
• Customer defections increasing
• Intervention
• Monitored training activity in 6 call centers
• Obtained business data from the same call centers

• Results
• Data indicated that training impacted sales
  6-15
• Four call centers needed to significantly
  increase training
• Challenges
• Use of correlated data is marginal justification
  for significantly altering business operations
• Setting up the right environment with the right
  data set for analysis is challenging

27
Case Study Unintended Results

• Profile
• Retail
• 1254 Store locations
• Product training defined monthly based on
  seasonal merchandise
• High turnover in personnel
• Intervention
• Monitored training completions at the store level
• Utilized store sales results as the business
  operations benchmark

• Results
• Negative correlation between training and store
  results
• Further inquiry revealed inappropriate
  application of training
• Challenges
• Broadly promoting results in advance of
  understanding the data and what is driving it
• Getting complete data sets from large audiences
  without investments in management technology

28
QA - Discussion
Jim Everidge, President Rapid Learning
Deployment, LLC [phone] x
222 JEveridge_at_rapidld.com www.rapidld.com