Overview of the Individual Self-Sufficiency Planning (ISSP) Project Evaluation

1
Overview of the Individual Self-Sufficiency
Planning (ISSP) Project Evaluation
2
Type of Evaluation
3
Outcome (or impact) Evaluation

• Outcomes of interest to leaders of Californias
  State Partnership Initiative (SPI) project have
  been (1) increased employment (number working
  hours) (2) increased earnings (3) job stability
  and attachment to the workforce (4) leaving the
  SSA benefit roles (5) reducing reliance on such
  benefits (6) reducing use of selected,
  publicly-funded mental health services and (7)
  contributing findings to public policy.

4
Process Evaluation

• Q1. Have there been any unintended or
  unanticipated outcomes experienced, positive or
  negative?
• Q2. What barriers have been encountered in
  delivery of the intervention strategies or
  achieving intended outcomes and, if so, how were
  those barriers addressed?
• Q3. How have State and local structures and
  procedures changed as a result of the
  demonstrations?

5
Process Evaluation

• Q4. What State and Federal policy changes would
  have made systems change efforts more effective
  and/or permanent?
• Q5. What was the role of the project in expanding
  access to health care coverage for individuals
  with disabilities?
• Q6. How did the activities of the project assist
  in the design and implementation of benefits
  planning and assistance activities?

6
Process Evaluation

• Q7. How was the project involved in efforts to
  expand access to generic employment services on
  the part of individuals with disabilities? Was
  the project involved in attempts to improve the
  ability of One-Stop Centers to meet the needs of
  individuals with disabilities?

7
Process Evaluation

• Q8. Did the project design and implement new and
  innovative service strategies that led to
  increased support for individuals attempting to
  reduce dependence on Federal benefits and obtain
  or expand employment opportunities?
• Q9. How did the project collaborate with other
  reform initiatives underway in the State at the
  same time as the SPI project, such as the WIG,
  MIG, Medicaid Buy-In, BPAO, Ticket, etc.?

8
Research Design and Data Collection
9
Quasi-experimental Design

• Some SPI projects (e.g., New York) tested the
  impact of work-related information, and used
  experimental designs with random assignment to
  treatment (e.g., basic plus more advanced
  information and assistance) and control groups
  (e.g., basic information only).

10
California A Matched Comparison Group Design
11
Target Group

• Those signing up for enhanced services (i.e.,
  enrolling in the project) had to have
• (1) a severe psychiatric disability
• (2) be receiving SSI, SSDI, or both and
• (3) be receiving vocational rehabilitation and/or
  employment services.

12
Service Settings

• 17 of about 29 DMH/DOR Cooperative Projects
  expressed interest and submitted proposals.
• Two sites (Vocational Rehabilitation Services, in
  San Mateo County and Kern County Vocational
  Services in Bakersfield) were selected.
• One criterion was engagement with One-Stop Career
  Service Centers, because of interest at both
  State and federal levels.

13
Enhanced Services

• Over and above existing DMH/DOR Cooperative
  program services (assessment, planning, pre-job
  search, job search, job coaching, etc.),
  enhanced services involve 1 Benefits
  Coordinator and 1 Service Coordinator for 50
  individuals (502 ratio).
• San Mateo has had 100 slots, with 2 BCs and 2
  SCs. Kerns program is much smaller, and has had
  50 slots with 1 BC and 1 SC.

14
Matched Comparison Groups
15
Three Matched Comparison Groups

• All selected from the DOR database.
• 1- DOR service recipients in Sacramento and
  Fresno counties.
• 2- DOR service recipients across all of the other
  DMH/DOR Cooperative Project sites.
• 3- DOR service recipients in selected counties
  without a DMH/DOR Cooperative Project, where the
  recipient received services through a Generalist
  VR Counselor.

16
Stratified Systematic Sampling

• Random start used across two strata (1) Major
  Disability (Psychosis Mood Disorder Other
  Psychiatric) and (2) SSA Benefit Status (SSI
  SSDI Both).
• If information had been available, a match on
  work history would have been useful
• SSA Benefit Status was chosen as a proxy for work
  history

17
Measuring Outcomes

• VCU and Mathematica Policy Research (MPR),
  oversee the 17 SPI projects for the SSA.
• Designed a data collection system and are
  carrying out a multi-state evaluation.
• ISSP project staff use about 30 of their time
  collecting initial demographic and quarterly
  update information and transmitting it to VCU.

18
Measuring Outcomes

• Much of the outcome information through this
  system is self-reported.
• DOR obtains quarterly UI covered earnings data
  through EDD.
• MPR working with SSA to obtain benefit and
  related information from SSA files.
• DMH collaborated with DOR to retrieve selected
  Medi-Cal information on mental health services,
  without jeopardizing privacy and confidentiality.

19
Measuring Services or Interventions

• Some use has been made of DOR status codes,
  including types of case closures (e.g., 26s,
  28s).
• Most project-related service intervention data
  comes from site reports to VCU and MPR, as part
  of their data collection system.

20
Measuring Services or Interventions

• Individuals participate in the project for
  varying amounts of time. Some individuals will
  leave the project sites may graduate others
  and some participants will become unreachable.
• Initially, it was thought that the ISSP project
  would touch about 500 individuals over the five
  years. This target cumulative enrollment was
  subsequently reduced to 250, which has been
  achieved.

21
Site Visits and Project Meetings

• Decided to make five annual visits to the two
  demonstration sites
• Every other year visit the two chief comparison
  sites.
• Subsequent visits asked about changes over the
  preceding 12 months, in the kinds of people
  entering the project, services and supports,
  immediate outcomes, significant community
  developments and the like.

22
ISSP Statewide Coordinating Committee

• Initial meetings were monthly, then decided on
  day-and-one-half quarterly meetings.
• First half-day devoted to site reports and data
  elements (Data Elements Workgroup).
• Second half-day devoted to the work of a Waiver
  Workgroup.
• Morning of second day devoted to summary of
  preceding days work and big picture matters.
  This group is known as the ISSP Statewide
  Coordinating Committee.

23
ISSP Statewide Coordinating Committee

• Expressed a desire for additional information
• relationships between project personnel and local
  SSA office staff
• the role and importance of natural supports
• ramifications of having someone specializing in
  benefit planning/assistance for the rest of the
  team providing voc rehab and employment services
• job descriptions
• the paradox of service systems helping some
  individuals get benefits, while helping others
  reduce reliance on public benefits and so forth.

24
Lessons Learned, Things to Think About
25
National Data Collection

• Can California influence the national data
  collection scheme, if there is one, to make it
  fit better with Bridges and generate the kind of
  information Bridges project leaders envision?

26
National Data Collection

• In ISSP project, data collection forms were
  announced, but changes not possible.
• Sites ended up developing their own basic
  information system to guide their work and to
  report quarterly.
• Some aspects of the national evaluation data
  collection scheme seem excessive.
• National evaluation uses a much different
  comparison group than the California evaluation.

27
National Evaluation

• Should the California Evaluator serve as a
  facilitator for the National Evaluation, or
  should there be parallel evaluations (one
  California one across the set of States
  participating in the Youth Initiative)?

28
Gathering Information

• What kind of support will be needed at DOR,
  CDE, and SSA to get, store, and analyze
  information of value?

29
Resource and Information Needs

• It will make sense to follow an interactive
  process in reaching agreement on what exactly the
  California Evaluation team should do.
• Every aspect of the evaluation has resource,
  cost, and other considerations (e.g., whether
  data is shared on paper, via pen or pencil, or
  electronically).

30
Outreach and Intake

• Very important for several reasons
• choice of comparison group
• advertising potential waivers may encourage more
  at-risk youth and their families and more SSA
  benefit recipients to be referred and to apply to
  the project and
• it may prove difficult to stay on message.