GAOs Design Matrix A Tool for Policy Analysis and Program Evaluation

1
GAOs Design MatrixA Tool for Policy Analysis
and Program Evaluation

• Jay BermanAnalyst, GAO
• Dawn LockeAnalyst, GAO
• October 19, 2005

2
Bardachs 8-Fold Path to Problem Solving

• Define the Problem
• Assemble Some Evidence
• Construct the Alternatives
• Select the Criteria
• Project the Outcomes
• Confront the Trade-Offs
• Decide
• Tell Your Story

3
GAOs 5-Fold Path to Policy Analysis

• Identify Objectives
• Identify Information and Sources
• Determine Scope and Methodology
• Understand the Limitations
• Determine What Can be Said

4
Design Matrix The Form
5
Design Matrix The Benefits

• Encourages the use of systematic data collection
  and analysis
• Encourages individual ownership, team
  participation, and incorporates the teams best
  ideas
• Communicates the scope and methodology to top
  management and client(s)

6
Issue Statement Whats happening and who
cares?

• Express in a narrative format the issue or
  problem the assignment will research
• Nature of the issue
• Potential problem or concern and its magnitude
• Political environment and key players
• Possible report users
• Contribution to GAOs strategic plan

7
Researchable Questions

• What do they want to know?
• Identify objectives in the commitment letter
• Ensure each major evaluation question is
  specific, objective, neutral, measurable, and
  doable
• Each major evaluation question should be
  addressed in a separate row

8
Information Required and Source

• What information does the team need to address
  the evaluation question? Where will they get it?
• Identify information
• Collect documents that establish the criteria
  against which programs/projects will be evaluated
• Follow up on known significant findings and open
  recommendations from previous reports

9
Scope and Methodology

• How will the team answer each evaluation
  question?
• Describe strategies for collecting the required
  information
• Describe the planned scope of each strategy
• Describe the analytical techniques to be used

10
Limitations

• What are the engagement design's limitations and
  how will it affect the product?
• Cite limitations
• Questionable data quality and/or reliability
• Inability to access certain types of data
• Security classification restrictions
• Inability to generalize or extrapolate findings
• Discuss how these limitations will affect the
  product

11
What the analysis will allow you to say

• What are the expected results of the work?
• Describe what GAO can likely say
• Ensure that the proposed answer addresses the
  objective(s)

12
Case Study

• National Wildlife Refuges
• Opportunities to Improve the Management and
  Oversight of Oil and Gas Activities on Federal
  Lands

13
Case Study

• The U.S. contains of a 95-million acre National
  Wildlife Refuge System
• The Government owns the surface lands in the
  system
• Sometimes private parties own the subsurface
  mineral rights
• How GAO comes in