Hubbard

1
Quantitative Methods Make A Difference
2
Overview

• Quantitative methods (probabilistic analysis,
  operations research, etc.) are widely used in
  other industries, but mostly lacking in IT
  investment analysis
• Over the past 7 years, we have been focusing
  specifically on the application of more advanced
  quantitative methods to IT
• This presentation will review the key findings
  from the application of quantitative methods to
  over 30 projects

3
Quantitative Methods Include

• Computing uncertainties and risks
• Computing the economic value of information
• Measurement methods for many items usually
  considered intangible
• Optimizing solutions when there are huge
  combinations of options for
• Roll-out priorities of systems
• Selection of a portfolio of functions
• Any other problem where different alternatives
  about different aspects of the investment
  generate lots of possibilities

4
Method AIE
Applied Information Economics is the practical
application of scientific and mathematical
methods to quantify the value of IT-enabled
business investments
Economics
Applied Information Economics
Modern Portfolio Theory
5
Some HDR Clients

• Booz-Allen Hamilton
• Accenture w/ the State of North Carolina
• Giga Information Group
• American Express
• The Discovery Channel
• U.S. Federal Government
• Department of Treasury
• Bureau of The Census
• Department of Veterans Affairs
• General Services Administration
• Housing and Urban Development
• The Axa Group 6 major companies
• The Banking Industry Technology Secretariat
• Blue Cross Blue Shield of Illinois

6
What Do the Critics Say?

• Quantifying the risk and comparing its
  risk/return with other investments sets AIE apart
  from other methodologies. It can substantially
  assist in financially justifying a project --
  especially projects that promise significant
  intangible benefits. The Gartner Group
• AIE represents a rigorous, quantitative approach
  to improving IT investment decision making..this
  investment will return multiples by enabling much
  better decision making. Giga recommends that IT
  executives learn more about AIE and begin to
  adopt its tools and methodologies, especially for
  large IT projects. Giga Information Group
• AIE-like methods must become the standard way to
  make (IT) investment decisions. Forrester
  Research, Inc.

7
Five Key Revelations

1. Quantifying risk radically changes IT investment
   priorities
2. Current measurement priorities are upside-down
   when compared to priorities based on economic
   value of information
3. Technology regret is a significant and
   overlooked factor in the the value of IT
   investments
4. The true cost of scope creep is much higher
   than most would think
5. The value of quantitative analysis would make it
   the best investment in most IT portfolios

8
Finding 1 Risk Analysis

• When IT computes risk in the same way that an
  actuary would, many IT investments will look
  completely different
• We define risk a The probability of a quantified
  loss
• Risk analysis of IT investments involves a
  probabilistic analysis of all uncertain variables

9
Real-world Measurements vs. Ideal Values
Ideal Values Point
Real-world Meas.
Normal Distribution
Uniform Distribution
Lognormal Distribution
Hybrid
15
85
Threshold confidence
10
Calibrated Estimates

• Measuring your own uncertainty about a quantity
  is a general skill that can be taught with a
  measurable improvement
• Studies show that most managers are statistically
  overconfident when assessing their own
  uncertainty
• Training can calibrate people so that when they
  provide a 90 confidence interval, it still has a
  90 chance of being right (even though it is
  subjective)

When asked to provide a subjective 90
confidence interval, most managers provide a
range that only has about a 40-50 chance of
being right
Perceived 90 Confidence Interval
Actual 90 Confidence Interval
11
Distribution-based ROI
12
Analyzing the Distribution
ROI 0
Expected ROI
Risk of Negative ROI
Probability of Positive ROI
The cancellation hump
25
50
75
100
125
-25
0
Return on Investment (ROI)
13
Plotting the Risk and Return
Risk
40
A proposed IT investment with a 15 risk and 54
return
30
Probability of less than a risk-free return
20
X
10
Return
10
20
30
40
50
60
14
Example of Risk Effects

• These are real IT investments of 2M-3M
  plotted against a clients investment boundary
• The 27 ROI investment is actually preferred to
  the 83 ROI investment

50
Region of Unacceptable Investments
40
30
Chance of a negative IRR
20
Region of Acceptable Investments
10
0
0
50
100
150
200
Expected IRR over 5 years
15
Risk Increases Required ROIs

• Adjusting for risk causes some previously-acceptab
  le projects to be rejected
• Also, some low return but low risk projects would
  now be acceptable
• More projects with intangible benefits are now
  economically justified
• The net result A completely reshuffled deck of
  IT project approvals

16
Using Risk Analysis to Improve Decisions

• If the Risk is significant (it usually is),
  consider doing the following
• Reduce the size and functionality of the proposed
  system - focus on fewer high-return features
• Wait until specific uncertainties in the
  environment subside - e.g. major mergers,
  reengineering, etc.
• Wait to tackle big projects until proper skills
  are developed and methods are in place
• Consider purchased packages that arent a perfect
  fit but close enough - they may look more
  attractive now
• Invest more on a proper economic analysis of the
  largest IT investments - this should reduce
  uncertainty about critical quantities
• Include deferred benefits in any estimate of
  scope creep costs

17
Finding 2 Measurements

• Information has a value that can be calculated
  with a formula known for 50 years
• Computing the value of additional information on
  uncertain variables causes some surprising
  changes in what gets measures

18
The Economic Value of Information
The Decision Theory Formula

• What it means
• Information reduces uncertainty
• Reduced uncertainty improves decisions
• Improved decisions satisfy business objectives
  (by definition)

19
The IT Measurement Inversion
Least Relevant to Approval Decisions
Receives Most Attention

• Costs
• Initial Development Costs
• Ongoing Maintenance/Training Costs
• Benefits
• A specific benefit (productivity, sales, etc.)
• Utilization (when usage starts and how quickly
  usage grows)
• Chance of Cancellation

Economic Relevance
Typical Attention
Receives Least Attention
Most Relevant to Approval Decisions
See my article The IT Measurement Inversion in
CIO Magazine (its also on my website at
www.hubbardresearch.com under the articles link)
20
Finding 3 Technology Regret

• Most business cases treat IT investments
  implicitly as a now or never choice
• Technology regret is an economic quantity
  associated with committing to a technology after
  which, for whatever reason, becomes regrettable
• The equivalent of buyers remorse
• Technology regret becomes and important
  consideration in any environment where changing
  technology is a constant
• The issue becomes balancing technology regret
  (which tends to defer IT investments) vs. the
  opportunity loss of deferring the benefits of
  making the investment now

21
Changing Technology
32 Annual Growth Rate
350

• Some Areas of Growth
• Processors Memory
• (Moores Law)
• Storage
• Communications (Paynes Law)
• Internet Users
• Sensory devices

300
250
200
Relative Computing Power Per (19801)
150
100
50
0
1995
1980
1985
2000
1990
Year
Competition makes capitalizing on new technology
more critical to survival
22
Changing With Technology
How often should you change with technology?
Avoiding technology regret is often a major
driver in IT decisions.
Upgrade 2
A Critical Technology Measure
Upgrade 1
Time
23
Real Option Theory

1. The option value tells us when an investment,
   even if it looks positive now, should be deferred
   until the opportunity is better
2. In the case of IT, waiting for the possibility of
   better technology around the corner should be
   considered

Invest this cycle, low priority, may be deferred
if resources are strained
Single Period Option Value (Value of Waiting one
period)
Invest in this cycle, high priority

0
Net Value of the Investment
24
The Effects of Tech Regret

• Very long duration IT projects that commit to a
  proprietary solution tend to look much less
  favorable
• Short turnaround projects based on
  non-proprietary standards tend to look better
• Large scale commitments to the fastest improving
  technologies (like data storage, bandwidth) tend
  not to be favorable

25
Finding 4 Scope Creep

• The cost of adding one additional function to an
  software development project is rarely addressed
  properly
• If anything, the only cost of new features
  considered is development cost
• Long term maintenance, increased risk of
  cancellation plus deferred benefits is much more
  significant

26
True Scope Creep Costs

• 24 Initial development costs
• 24 Future maintenance costs (computed over 5
  years)
• 1 Incremental contribution to probability of
  total project cancellation
• 51 Deferred benefits of the other functions
  delayed by the proposed new function

24
51
24
1
27
Finding 5 Value of Quant.

• Organizations have successfully adopted more
  advanced quantitative methods for evaluating IT
  investments
• The cost of analysis routinely comes in below 1
  and has always been under 2 of the investment
  size - including initial training
• This is still less than non-IT investments of
  similar size and risk
• It is also sometimes less time-consuming than the
  previous non-quantitative analysis techniques
  used by the firm (One of the reasons this
  analysis is efficient is we conduct a Value of
  Information Analysis - we only measure what is
  economically justified)
• Using the standard VIA calculation for the value
  of AIE analysis, AIE itself was the best
  investment of all the IT investments we analyzed
  - very conservative measures of payoffs put 20
  to every 1 spent on AIE

28
Overview of RRA Analysis
Classification
Value of Info.

Intangibles Customer Satisfaction Strategic
Alignment Technology Risk Information
Quality etc.


Measurables Errors in Decision X Change to
Strategic Measure M Productivity in Activity
Y Chance of cancellation, etc.
Risk
Organization's investment limit
Acceptable region of investment
Return
29
In Conclusion

• Quantitative methods like AIE cause major IT
  decisions to be very different and better
• Advanced methods can and have been learned and
  adopted by IT organizations
• More quantitative analysis can be the highest
  return investment in your IT portfolio