Handling Conflict of Interest in Utility Evaluation Studies

1
Handling Conflict of Interestin Utility
Evaluation Studies
Presentation/Discussion TopicCALMAC Meeting
October 19, 2005 Douglas Mahone, Heschong Mahone
Group, Inc.with support from Matt Brost,
RLWRichard Ridge, Ridge Assoc.Kathleen
Gaffney, KEMA
2
Common Points of Agreement

• Evaluation must be credible, disinterested,
  independent and professional, and this cant be
  done when there are conflicts of interest.
• Important that the evaluation studies for
  utility administrators be done right
• For evaluators, this is central to their
  credibility and professional standing

3
Issues for CALMAC Discussion

• Review CPUC Decision 05-01-055
• Types of evaluation studies
• Firewall between evaluators implementers
• Rules for utility evaluation studies
• Review proposal conflict of interest scoring
• Explicitly address in contractor selection
• More fair, transparent, fact-based than simple
  firewall
• Decide course of action
• Adopt, reject, amend, leave alone, etc.

Interim Opinion on the Administrative Structure
for Energy Efficiency Threshold Issues, January
27, 2005
4
CPUCs Impact Studies Type

• For PY 2006 and beyond
• Energy Division manages contracts
• EMV studies used to
• Measure verify savings
• Generate data for savings cost effectiveness
• ME achievements for performance basis
• Evaluate whether goals are met
• Firewall These evaluators cannot do program
  delivery work

5
Utilities Program Market Studies Type

• CPUC terminology Program design evaluation and
  market assessment
• Purpose Provide information feedback
• to administrators and implementers
• to improve program performance
• Excluded from the firewall much less relevant
  for these types of studies
• But ED makes final selection of contractors,
  consulting with ad hoc committee

6
Why Our Concern?

• Despite Decision, firewall may be applied anyway
• Twice, PGE has started to apply it incorrectly,
  and
• Twice has corrected itself
• Safe way to go
• People with CA market experience would be
  unavailable to utilities for market studies
• Need to formalize an alternative approach, and
  apply it consistently

7
Ambiguous Conflict Scenarios (just a sample)

• University scenario
• Mechanical engineering professor consults on a
  A/C efficiency program (implementation)
• Statistics professor consults on sample design
  for impact study of same (EMV)
• Firewall says this is a conflict (but its
  probably not)
• Vendor scenario
• Entity performs EMV studies
• Same entity sells electric metering equipment or
  billing software to utilities (not
  implementation)
• Firewall says this is not a conflict (but maybe
  it is)
• CEC PIER contractor scenario is it
  implementation?

8
Types of Conflicts

• Direct Financial Conflict direct stake in
  evaluation outcome
• Indirect Financial Conflict future business
  advantage based on evaluation outcome
• Personal, professional, political bias - bias or
  interest in evaluation outcome

9
Dealing with Conflicts

• Everybody has bias, and some degree of conflict
  of interest
• Firewall only addresses
• Direct Financial Conflict
• One variety of Indirect Financial Conflict (hope
  of future implementation work)
• Other types of conflicts ignored
• American Evaluation Association Guiding
  Principles for Evaluators - addresses conflicts
• Conflicts must be disclosed and managed

10
Recommendations

• Full Disclosure require entities to fully
  disclose all potential conflicts
• Should even extend to proposal review committees
• Model on CPUC RFP (see attachment)
• Automatically disqualify if direct financial
  conflict
• Include in scoring of proposals
• highly experienced with insignificant conflict
  could outscore inexperienced with no conflict

11
Open, Transparent Scoring

• Establish Conflict Guidelines - see AEA Guiding
  Principles to start
• Case-by-Case Review part of normal contractor
  selection process
• Dispute Resolution Open, consistent, timely
  process to address conflict bias

12
CPUC Disclosure Language

• Rationale Conflicts may
• Prevent impartial, technically sound, objective
  assistance and advice
• Result in a biased work product
• Result in an unfair competitive advantage
• Required disclosure of
• Current prior contracts/financial
  relationships( involved, duration, nature of
  service, past 3 yrs)
• Any current business (including active proposals)
• Signed statement on disclosure

13
CPUC Scoring (example)

• 20 - Experience with technical content
• 20 - Work plan
• 10 - Resources available
• 10 - Ability to execute
• 10 - Experience with utility projects
• 10 - Familiarity with public purpose progs
• 15 - Potential for conflicts of interest
• 5 - Education and credentials

14
Third EMV Type Research Analysis Studies

• Grey areas
• Technical Potential Studies
• DEER Database
• Net-to-Gross Ratio Assessments
• Best Practices
• Unclear how firewall rule applies to theseor
  how much of an issue it would be
• But will be managed by ED
• Recommend conflict scoring, not firewall

15
so what should CALMAC do?

• Unofficial body, but respected evaluation
  advisory group
• Only sitting body with insight and knowledge to
  address this issue
• Recommendation will help utilities contract for
  their evaluation services
• Statewide consistent method will be better than
  many different approaches