Evaluating Ongoing Programs:

1
Evaluating Ongoing Programs

• A Chronological Perspective to Include
  Performance Measurement
• Summarized from Berk Rossis Thinking About
  Program Evaluation, Sage, 1990 Martin
  Kettners Measuring the Performance of Human
  Service Programs, Sage, 1996

2
Stages of Assessment

• Stage 1 Determining Whether the Program is
  Reaching the Appropriate Beneficiaries.
• Stage 2 Making Sure the Program is Being
  Properly Delivered
• Stage 3 Ensuring Funds Are Being Used
  Appropriately
• Stage 4 Ensuring Effectiveness can be Estimated
• Stage 5 Determining Whether the Program
  Works
• Stage 6 Determining Program Worth

3
Stage One Program Impact

• Program Impact Research is designed to identify
  who is actually served by a program to determine
  the number being served that meet program service
  criteria and those that are being served that do
  not meet service criteria.

4
Stage Two Program Integrity

• Program Integrity Research analyzes the
  essentials of program delivery such as
• personnel qualifications skill assessment
• consistency of program services with program
  mission
• targeting marketing of services
• service coordination

5
Stage Three Fiscal Accountability

• Accountant Perspective
• Out of Pocket Expenses
• Historical Costs
• Depreciation
• Current Anticipated Revenues
• Product Inventory
• Income Outgo of Funds

6
Stage Three Fiscal Accountability

• Economist Perspective Opportunity Costs
• Opportunity Costs May be Considered as What was
  Given Up to Direct Resources in a Particular
  Direction
• Opportunity Costs May Also Be Construed as the
  Next Best Use of Resources

7
Stage Four Evaluability

• Criteria for Evaluability
• Clarifying Goals
• Specifying Program Goals
• Determining Possible Outcomes in the Absence of
  the Program

8
Stage Five Program Effectiveness

• Comparisons Across Subjects
• Comparison Across Settings
• Comparison Across Time
• Comparison Across Criterion
• Pooled Comparisons

9
Research Designs for Estimating Effectiveness

• Random Assignment Comparing Mean Outcomes of
  Control Experimental Ss
• Interrupted Time Series (Pre- and
  Post-Assessment Model)
• Cross Sectional Designs Comparisons of Different
  Types of Units (e.g. comparing smaller larger
  cities) with Comparisons Occurring at Only One
  Point In Time

10
Research Designs for Estimating Effectiveness

• Regression Time Series Assignment of Ss by
  variables (Criterion Based). Objectives
• To provide estimates of values of the dependent
  variable (outcome variable) from values of the
  independent variable (assignment Variable)
• To obtain measures of the error involved in using
  the regression line as a basis of estimation
  (I.E. Standard Error of Estimate)
• To obtain a measure of the degree of association
  or correlation between the two variable

11
Research Designs for Estimating Effectiveness

• Pooled Cross Sectional Time Series
• Randomized Experiments
• Regression Designs
• May Be Compared
• Across Units (Cross Sectional)
• Across Time (Time Series)

12
Stage Six Cost Effectiveness

• Ongoing Versus New Programs
• Ongoing Programs Have Historical Data to work
  With.
• New programs lack such historical data from which
  to determine cost effectiveness

13
Performance Measurement
14
Defining Performance Measurement

• The regular collection of and reporting of
  information about the efficiency, quality, and
  effectiveness of human service programs. (Urban
  Institute, 1980)

15
Perspectives of Performance Measurement

• Efficiency Perspective
• Quality Perspective
• Effectiveness Perspective

16
Systems Model Essentials

• Inputs Includes anything used by a system to
  achieve its purpose
• Process Involves the treatment or delivery
  process in which inputs are consumed to produce
  outputs
• Outputs That which is produced
• Feedback System information reintroduced into
  the process to improve quality, efficiency
  effectiveness

17
Efficiency Perspective

• Productivity ratio of outputs to inputs
• Efficiency maximizing outputs to inputs
• Efficiency can not reflect whether program goals
  are being met
• Inefficiency is how many programs are regarded by
  the public - often in the absence of a full
  understanding of the goals, mission, clientele,
  resources, and services of the agency

18
Quality Perspective

• Typically involves benchmarking against standards
  and criteria of excellence (as in TQM, or Total
  Quality Management)
• TQM now defines productivity as the ratio of
  outputs that meet a specified quality standard

19
Effectiveness Perspective

• Focuses on outcomes such as the results, impacts
  and accomplishments of programs
• Effectiveness is the highest form or performance
  accountability
• Focuses upon which intervention works in which
  settings
• Effectiveness accountability is primarily
  concerned with ratios of outcomes to inputs.

20
Reasons for Adopting Performance Measurement

• Performance measurement has the potential to
  improve the management of human service programs
• Performance measurement has the potential to
  affect the allocation of resources to human
  service programs
• Performance measurement may be a forced choice
  for many, if not, most human service programs

21
Key Questions in Performance Measurement

• Who are the clients?
• What are their demographic characteristics?
• What are their social or presenting problems?
• What services are they receiving?
• In what amounts?
• What is the level of service quality?
• What results are being achieved?
• At what costs?

22
Performance Measurement as a Management Tool

• Performance Measurement promotes client centered
  approaches to service delivery
• Provides a shared language for comparing human
  service programs for quality, efficiency,
  effectiveness
• Allows administrators to continuously monitor
  programs to identify areas for improvement
• Provides direct feedback to personnel, allowing
  them to improve their service provision

23
Performance Measurement Programs
24
Government Performance Results Act (1993)

• Effective 1998, all federal agencies must begin
  reporting effectiveness data for their services
  products
• This requirement will be passed on to agency
  contractors subcontractors
• Increasingly Federal block - grant programs also
  have this requirement

25
National Performance Review

• Refers to governmental efforts at instituting
  program effectiveness, efficiency, and quality,
  to implement the 1992 report on government
  practices entitled Reinventing Government (Osborn
  Gaebler, 1992)

26
Total Quality Management Movement

• National Movement to Improve Quality
• Focuses upon
• consumer satisfaction
• outputs as measured against a quality standard

27
Managed Care

• Emanates from health care
• Promotes efficiency to assist health care
  industry shift from cost-based to capitated
  reimbursement

28
(SEA) Service Efforts and Accomplishments
Reporting

• Standard introduced by the Governmental
  Accounting Standards Board (GASB)
• SEA is GASBs term for performance measurement

29
SEA Reporting Model

• Built upon an expanded system model including
• inputs
• outputs
• quality outputs,
• outcomes BUT
• Excludes Process

30
SEAs Lack Of Emphasis Upon Process

• Absence of the Process component reflects SEAs
  primary emphasis upon performance performance
  cost considerations

31
SEA Reporting Elements

• Service Efforts
• Service Accomplishments
• Measures or Ratios Relating Service Efforts to
  Service Accomplishments

32
Service Efforts

• Service Efforts are inputs utilized in a human
  service program, which are measured by the GASB
  in terms of
• Total Program Costs
• Total Full Time Equivalent Staff (FTE)
• Total Number of Employee Hours

33
Service Accomplishments

• Outputs
• Total Volume of Total Service Provided
• Proportion of Total Service Volume Meeting
  Quality Standard
• Outcomes
• Measures of results, accomplishments, impacts

34
Service Accomplishment Ratios

• Efficiency (output measures) cost per unit of
  service
• cost per FTE
• cost per service completion
• service completions per FTE
• Effectiveness (outcome measures)
• cost per outcome
• outcome per FTE

35
Output Performance Measures

• Intermediate
• episode or contact unit of service
• material unit of service
• Final Service completions

36
Outcome Performance Measures

• Intermediate
• numeric counts,
• standardized measures
• level of functioning scales
• client satisfaction
• Ultimate
• numeric counts
• standardized measures
• level of functioning scales