Managing for Results CommunityBased Abstinence Education Grantees Conference

1
Managing for Results Community-Based Abstinence
Education Grantees Conference
Vijaya ChannahSorah, Ph.D. ACF
Performance Management Operations
2
Managing for Results

• Uses logic modeling to relate
• gt tracking program implementation
• gt measuring outputs and outcomes
• gt budgeting
• gt planning research/evaluation priorities
• The purpose is to improve program outcomes and
  service to program recipients.

3
The World a few Years ago

• Focus on incremental budgeting
• traditional management
• Focus on reporting processes and outputs

4
Traditional Budgeting

• Focus on inputs, such as increased funding,
  rather than outcomes
• Changes in inputs at the margin (for example, X
  more dollars than last year will fund x more
  grants)

5
The World of Managing for Results

• Strategic Planning
• Performance Budgeting
• Performance Measurement / Reporting
• PART
• Presidents Management Agenda (PMA)
• Perform. Accountability Rept.
• Research / Program Evaluation
• Logic Modeling to link
• all these (how the puzzle fits) !

6
A New Way of Budgeting

• Budget/performance information is integrated
  for the first time in 2006 into the
  Congressional budget request
• Budget justifications include discussions of
  goals, measures, and performance
• Budget submissions include PART recommendations
  for program and management improvement and how
  those findings influenced budget decisions
• Example 500,000 was included in LIHEAP request
  for feasibility study for national evaluation
  (PART recom.)

7
What Does this Mean forProgram Funding?

• Bottom Line Program effectiveness may affect
  budget decisions
• Reduced funds, or in some cases no funding, might
  be requested for poor performers
• High performing programs might have a better
  opportunity to garner additional funds
• Example Refugee program demonstrated
  significant improvements in refugee employment
  which reinforced ORR request for 68M

8
PERFORMANCE INFORMATION

• Key outcome performance measures for programs
• Output measures
• Key efficiency measures
• Program Assessment Rating Tool results
• Research and Evaluation

9
How Performance Information is Integrated into
Budget Request

• Performance information (including PART results)
  is now included in
• - the Overview in the Budget Request and
• - each programs section of Budget Request
• ALL the ACF detailed Performance Information
  (tables, data issues, etc.) included in the
  performance budget as well.

10
Challenges in Performance Management

• Defining efficiency for human services programs
  (e.g., abstinence education)
• Defining outcomes (e.g., family violence
  prevention)
• Timely, reliable data

11
Program Assessment Rating Tool (PART)

• 25 questions about
• Program Purpose Design (20)
• Strategic Planning (10)
• Program Management (20)
• Program Results (50)
• And approximately 5 supplemental questions
  tailored to the type of program, e.g., grant,
  direct service delivery, research, etc.
• Ratings can be Inadequate, Results not
  Demonstrated, Adequate, Moderately Effective, or
  Effective

12
Program Assessment Rating Tool

• PART experience 22 of our ACF programs have
  participated in PART
• FY 2004 3 programs
• Head Start, Foster Care, Refugee Resettlement
  (Social Services and Targeted Assistance Formula
  Grants)
• Scores ranged from 37 to 68 (RND to Adequate)
• FY 2005 5 programs
• CSBG, LIHEAP, Child Support, ADD, RHY
• Scores ranged from 35 to 93 Foster Care was
  reassessed, increased from 33 to 54
• FY 2006 6 programs
• AFI, CAPTA, CBCAP, Child Care, Indep Living, VCR
• Scores ranged from 39 to 81 (Ratings RND to
  Moderately Effective)

13
Program Assessment Rating Tool

• FY 2007 8 programs
• - ORR/TAMS - effective
• - ORR/Human Trafficking, Adoption
  Assistance, TANF - moderately effective
• - Adoption Incentives Adoption
  Opportunities adequate
• - Mentoring Children of Prisoners SSBG
  results not demonstrated
• Abstinence Education might be PARTed this year

14
Research and Program Evaluation

• Outcome measurement can show
• Trends in outcomes
• Comparisons between actual and desired outcomes
• Rigorous evaluation can show
• Outcomes relative to what they would have been in
  the absence of the program
• The programs causal contribution to the observed
  outcome

15

16
Overview of Logic Models

• A logic model tracks how we get from our
  challenges to our solutions and desired long term
  outcomes

CHALLENGE
INPUTS
ACTIVITIES
OUTPUTS
DEMOGRAPHICS
LONG TERM OUTCOMES (GOALS)
OUTCOMES
EXT. FACTORS
CONSTRAINTS
ASSUMPTIONS
17
Logic Model Defined

• What is a Logic Model?
• A graphic representation of a program.
• It shows what the program is designed to
  accomplish, including the services it delivers,
  expected results of those services, and the
  linkages between services and program goals.

18
Logic Model Formal Definition

• A logic model is a picture of how your
  organization does or plans to do its work it
  links outcomes (both short and long term) with
  program activities/processes and the theoretical
  assumptions/principles of a program

19
Logic Models

• can go both ways
• X -------------------------------------------
  --?Y
• (e.g., how can we make a raft float?)
• X lt- ----------------------------------------
  ----Y
• (e.g., why did the raft sink?)

20
Logic Model Uses

• Use for program
• Design
• Budgeting
• Implementation
• Evaluation
• Communication
• Marketing
• So a logic model can be used to help plan a
  program, to help manage a program, and to guide
  the evaluation of a program.

21
Logic Model Construction Process

• Model Development
• Think about what it is that you are really trying
  to do
• Establish the scope and context
• Determine challenge(s), outcomes, inputs,
  activities, outputs, and measurement components
• Create model draft
• Express relationships among/between key
  components
• Determine evaluation needs/points using dotted
  line arrows (solid arrows show known
  relationships)

22
Logic Model Development

• 1. Identify CHALLENGE/Social ill What do we want
  to improve in the population?
• Out of wedlock births
• Relationships before marriage
• Diseases (STDs)
• Challenges are often expressed as statements of
  fact, based on empirical data/statistics.
• There may be multiple challenges addressed by a
  single program.

23
Logic Model Development

• 2. Identify LONG-TERM OUTCOMES The ultimate end
  goals of your program, how your service
  population will look after your interventions
  have taken place
• Decrease out of wedlock births
• Increase proportion of abstinent youths
• Decrease preventable disease (STDs)
• Long-term outcomes/goals (as well as all other
  outcomes) are usually expressed as changes you
  will use words such as improved, increased,
  decreased, etc.
• Ultimate long-term outcomes/goals are sometimes
  pie in the sky or utopian.

24
Logic Model Development

• The remainder of the logic model elaborates how
  we get from the CHALLENGE to the LONG TERM
  OUTCOMES.

CHALLENGE
INPUTS
ACTIVITIES
OUTPUTS
DEMOGRAPHICS
LONG TERM OUTCOMES (GOALS)
OUTCOMES
EXT. FACTORS
CONSTRAINTS
ASSUMPTIONS
25
Logic Model Development

• 3. Identify Inputs/Resources (personnel, funds,
  laws/regulations, creative ideas, etc.)
• 4. Identify Activities
• Train the trainer
• Delivering abstinence education in churches,
  community centers, schools

26
Logic Model Development

• 5. Identify Outputs, such as number of trainers
  trained, number of training courses developed and
  administered.
• 6. Identify/develop Key Outcomes and Measures
• Decrease the rate of births to unmarried teenage
  girls ages 15-19 (35.4 in 2002) Target 35 in
  2003
• Decrease the proportion of youth ages 15-19 who
  have engaged in sexual intercourse (46.7 in
  2003) Target 45.5 for 2004
• and set Targets

27
Logic Model Development

• 7. Determine where program evaluation/research
  needs to take place (and depict arrows
  accordingly)
• - - - - - - gt plausible causal
  relationship (or desired effects/results)
• ----------? known causal
  relationship (based on scientific
  research/program evaluation)

28
Logic Model Development

• Complementary Tools
• Strategic plans
• Flowcharts
• Process diagrams
• Related logic models
• Performance budgets containing global Agency
  performance measure information

29
Linking ! How does all this relate?

• Results Act (requires performance measurement
  reporting in the Performance Budget)
• PMA (requires integrating this performance with
  budget requests and with financial reporting)
• PART (very detailed analysis of Agency
  management, measurement, reporting)
• Logic Modeling (gives a clear picture of what
  outcome measures needed, program evaluation
  needed, etc.)

30
In Conclusion

• ACF is increasingly focusing on results
• We have lots of information! how do we assemble
  it to help us manage?
• Use Logic Modeling to track and improve
  performance at Federal, state, and grantee levels

31
Questions / Discussion

• Questions on any aspect of Managing by Results?
• Is it clear how all aspects relate to one another
  and how logic modeling can be used as a powerful
  tool?