Estimating Program Benefits

1
Estimating Program Benefits

• NREL Analysis Brown Bag
• October 9, 2003
• Mary Beth Zimmerman
• Office of Energy Efficiency Renewable Energy
• U.S. Department of Energy

2
EEREs been at this a while

• 1993 Managing for Results Initiative.
• Quanity metrics, GPRA benefits
• Always the same goal a consistent set of
  methodologies, assumptions, and metrics
• Annual guidance covering baseline, methodology
  economic data
• AD Little, Inc. reviews of assumptions
• Lots of help from our friends
• 1996 GAO report
• National Academies of Science

3
Why benefits analysis?

• Portfolio management
• Select a mix of RD, deployment that achieves
  desired results
• Be agile in keeping portfolio aligned with
  evolving needs
• Congress wants to buy benefits, not technologies
  or activities
• Public health, national defense, a clean
  environment
• Its required
• 1992 Government Performance Results Act (GPRA)
• 2001 Presidents Management Agenda (PMA)

4
Who wants what?

• GPRA
• Strategic plans, annual performance plans
• Missions, outputs,outcomes, goals, verification
• Accountability reports
• PMA
• Performance Assessment Rating Tool
• RD Investment Criteria
• Decision points, end points, off-ramps, goals
• Performance, outcome information in budget, on
  web
• CFO
• 5-year budget plans
• Budget text

5
GPRA Definitions

• Outcome measure Is it important?
• Output measure Is it well focused?
• Performance goal When are you done?
• Performance indicator Hows it going?
• Program activity Is it tied to the budget
• Program evaluation How do you know?

6
RD Criteria
7
Benefits result from realizing program goals

• Program Performance Goals
• Wind energy _at_ 3cents/kWh
• EERE Strategic Goals
• Efficiency, renewable capacity, production
• Resulting Benefits
• Economic, Environmental, Security/reliability

8
The story is simple . . .
If you give me some ,
Ill do some stuff with it,
making some progress each year
until I produce a product
which will provide the benefits you want
9
The jargon
If you give me some ,
The GPRA logic chain
Budget, Resources, inputs
Ill do some stuff with it,
Key activities
making some progress each year
Targets, milestones
until I produce a product
Program performance
Outputs, program goals, end points
which will provide the benefits you want
Portfolio benefits
Outcomes, impacts, Benefits
10
Basic Performance Management Elements

• Inputs
• Budget
• Expenditures by others

• Activities
• Key
• Project

• Milestones
• Annual
• Critical Path
• Graduation

• Outcomes
• Interim
• Final

• Outputs
• Interim
• Final

PROGRAM PERFORMANCE What we are held accountable
for
BUDGET Available resources
BENEFITS How Programs Address Energy
Goals, Needs
11
Assessing benefits of achieving program goals
Strategic Planning
Performance Planning
Annual Operating Planning

• Inputs
• Budget
• Expenditures by others

• Activities
• Key
• Projects

• Milestones
• Annual
• Decision points

• Outcomes
• Strategic goals
• Benefits

• Outputs
• Program performance goals

Evaluation
12
What Product Does It Support?

• Inputs
• Budget
• Expenditures by others

• Activities
• Key
• Project

• Milestones
• Annual
• Critical Path
• Graduation

• Outcomes
• Interim
• Final

• Outputs
• Interim
• Final

• Planning
• Budget
• APP
• RD Scorecard
• Joule
• PART
• Evaluation
• Account Rept.
• PART

• Planning
• Budget
• AOP
• Evaluation
• Monthly revs

• Planning
• Budget
• APP
• RD Scorecard
• PART
• Joule
• Evaluation
• Monthly revs
• Quarterly revs
• Account Rept.
• PART

• Planning
• Benefits Rept.
• Budget
• Strategic Plan
• RD Scorecard
• PART
• Evaluation
• Benefits Rept.
• PART

• Planning
• Budget
• Benefits Rprt.
• Evaluation
• Benefits Rprt.

13
EERE Logic Model
Source John Mortenson
14
Applied RD timeline
5 to 30 years for appreciable stock turnover
EERE RD 3 to 30 years
Frequent gap between federal RD and
commercialization
15
Start with Program (Output) Goals

• Usually for each major area of research
• E.g., CSP, PV, Solar Thermal
• Exclude exploratory work to establish a pathway
• Two elements
• An end point or date when the effort will be
  completed
• An output, or level of accomplishment when done

16
Trendable performance measures
C/kWh
Conceptual
17
Add External Factors
Outputs What the program produces
Outcomes What it means for the public
External factors
RD output Technologies capable of producing
solar power _at_ 7 cents/kWh Deployment
output Technical assistance supporting the
installation of __ MW of solar power

• Systems purchased
• Energy savings
• Lowe emissions
• Improved reliability

Electricity prices Air quality needs Other DG
techs
18
And a Few More Steps
Resulting benefits
External factors
RD output
Emissions reductions
Technology Performance Cost
Energy cost savings
Market, policy conditions
Net economic benefits
19
EERE Reorganization

• Streamlined (re)organization facilitates
  performance management
• TD Clearly defined programs provide basis for
  measuring program performance.
• BA Integrated planning budget formulation
  provide basis for linking budget decisions to
  corporate results.

EERE
BA
TD
Geo
Solar
Wind
Ind
Wx
DEER
Program Execution
Information Management Systems
Planning Budget Analysis
Hydrogen
Vehicles
Bldgs
Bio
FEMP
20
EEREs Reorganized Approach

• Programs provide output goals as part of their
  multi-year performance plans.
• Market-specific information is developed and
  assessed for importance to bottom-line.
• Market-specific information is basis for
  energy-economic modeling of benefit estimates.
• Where model cannot develop benefit estimates,
  off-line results are adjusted to reflect economic
  interactions.

21
Need to Understand Future with W/out EERE
Programs

• What would have happened without us?
• Baseline technology improvement
• Baseline market evolution
• How will markets respond to EERE technologies
• Individual markets
• Energy economy
• How do deployment efforts change market
  receptivity

22
The Ground Rules

• Benefit estimates based on stated program output
  goals achievable with current funding levels
• Goals are assumed to be successful
• Benefits difference between how energy would be
  used without us with us.
• For the budget, benefits are counted for program
  activities from here on out only
• One baseline for everyone
• Energy prices, demographics, GPD, industrial
  output
• Demands for energy services
• Improvements in competing technologies
• Policies dont change

23
Empirical Foundation

• Not good enough to just assert lock-out, valley
  of death -- some products do succeed without us!
• Need to explain--not explain away--observed
  behavior because this is the key to changing
  markets.
• If we dont account for price effects, market
  interactions, well over- or under-estimate the
  value of our efforts.

24
Technology Characteristics

• Cost
• EERE technology vs. future baseline technology
• Performance
• Energy efficiency
• Energy production efficiencies
• Non-energy performance (e.g., acceleration,
  quality of light)

25
Residential Sector Space Heating Technologies

• Electric resistance heat
• Electric heat pump
• New improved heat pump
• Advanced electric heat pump
• Gas heat pump
• Ground source heat pump
• Oil burner to space heat
• Improved oil burner to space heat
• New oil burner to space heat
• LPG burner to space heat
• Kerosene burner to space heat

• Gas burner to space heat
• Improved gas burner to space heat
• New gas burner to space heat
• Solar heat - electric backup
• Solar heat - oil burner backup
• Solar heat - gas burner backup
• District heat exchanger - space heat
• Fireplaces woodstoves
• Building shell conservation (3 levels)
• New building shell conservation (3 levels)

26
Renewable Electric Generation Technologies

• MSW - mass burning-electricity
• Biomass combine cycle (ATS/IGCC)
• Biomass direct fired electric
• Geothermal electric, liquid
• Geothermal flashed steam electric
• Geothermal binary cycle
• Geothermal central heating plant
• Hydroelectric
• Hydroelectric pumped storage
• Solar central thermal electric

• Central photovoltaic
• Photovoltaic - residential
• Solar central heating plant
• Solar thermal dish sterling
• Ocean thermal electric
• Wind central electric - class 5-7
• Wind central electric - class 4
• Wind central electric - class 3
• Local wind electric generator

27
Market characteristics

• Private sector RD investment rates
• Market fragmentation/integration
• Consumer preferences
• Price responsiveness, non-price attributes
• Differences among customer classes
• Niche markets
• Regional variations
• Price, availability

28
Regional variation
Deregulated States
Source Energy Information Administration, May
2000.
29
Why Focus on Markets?

• Thats where EERE technologies got bought and put
  into use
• These are the realities we are up against
• Competition from improved fossil technologies
• Consumer discount or hurdle rates
• Market frictions
• The more we understand our markets, better we can
  tailor our technologies, deployment effort for
  success
• Is a value to technical potential, addressed
  later

30
RD, Deployment modeled differently
RD modeled as different TC, or earlier start
date for techs already in the model
Deployment modeled as accelerated market
penetration or lower consumer hurdle rate
31
Deployment impacts

• How much do we wish to increase market share?
• What deployment methods best address the market
  barriers observed?
• How many potential consumers must be reached to
  generate the needed additional benefits?
• Are there problems with degredation or pesistance
  of performance?
• Can the market be transformed so that deployment
  efforts are not required to sustain?

32
(No Transcript)
33
Relating Trees to Forest

• Specific markets
• What do suppliers supply, why?
• What do consumers buy, why?
• Nitche markets
• Regulatory, policy factors
• Regional, income, or other market variations

• Energy-economy
• Supply demand curves
• Take-back effect
• Upstream impacts
• Cross-fuel interactions
• New energy pathways

34
Why Use Models

• Keep track of interactions
• Feed-back effects
• Down-stream effects
• Competition, synergies among our technologies
• Get the big picture
• Price reductions
• Balance of growth, fall in natural gas
• Back-out what would have happened without us
• Figure out whats important
• Confusing results sometimes lead to insights
• What if runs can help us understand which
  technologies, policies are having the biggest
  impact

35
Models Summarize Information

• Rely on empirical information about markets to
  specify these conditions
• Can be modified to reflect new types of energy,
  energy services, changes in consumer behavior,
  policy, etc. IFF the necessary information is
  available.

36
EERE Models

• NEMS
• Developed by EIA
• Through 2025
• Rich in market detail, can be limited in
  technology detail
• Regional
• Consumer hurdle rates

• MARKAL
• Developed by Brookhaven
• Through 2050 (our verision)
• Used in many countries IEA EIA developing
  international versions
• Rich in technology detail, can be limited in
  market detail
• Optimization

37
Market interactions
38
Integrated Modeling of Technologies in MARKAL
Ethanol
Hydrogen
Fuel Cell
39
The big picture
40
NAS/NRC Benefits/Costs Framework
41
NAS Benefits frameworkhelps us tell more of our
benefits story
42
Modeling Market Uncertainty

• Scenarios easiest next step
• High/low fuel prices
• Environmental constraints
• Consistent with other DOE offices
• Value of technology on the shelf
• Technical Potential defines the extent of the
  options developed
• Analytical techniques for ascribing value to the
  shelf
• Real options approaches to portfolio management
  use this information about uncertainty to shape
  RD, deployment strategies

43
Market uncertainties
EIA coal price projections
44
Where we are Options

• Taking uncertainty into account changes RD
  choices
• Positions firm for greater range of future
  markets
• More likely to invest in higher risk, longer term
  RD
• Better manage stage-gating of RD, more likely to
  cancel poorer performing projects earlier
• A product doesnt have to enter the market to
  provide value
• Lots of approaches to uncertainty
• Some dont seem quite ready for prime time
• Some would be tedious, data-intensive to apply to
  our portfolio
• We can stage-gate our development of the options
  column
• Scenarios give good first picture of how
  portfolio meets possible future needs.
• Risk assessment tools give a more realistic
  expectation of research results.

45
Where we are Security

• There are standard approaches to evaluating
  security that we can use
• Probability, severity, duration, recovery
• Important to identify which of these we
  address--and which we do not.
• Valuable to be in-sync with these general
  approaches.
• Getting a handle on energy infrastructure will be
  key to assessing security, reliability benefits.
• Agent-based approaches may be necessary to
  assessing security reliability

46
What were working on now

• Streamline benefits analysis within new
  organization
• Use Logic Chain to improve transparency
• Extend timeframe looking forward
• Add retrospective component
• Develop metrics for all NAS rows
• Economic, environment, security
• Develop 2-3 scenarios for option column
• Select and adopt a risk assessment approach
• Removing EERE impacts from baseline
• Improved reflection of market features
• Regional variation
• Infrastructure
• Consumer preferences