Linking administrative data sets for self-evaluation: Preliminary results from the Annie E. Casey Foundation Family to Family Initiative in California

1
Linking administrative data sets for
self-evaluation Preliminary results from the
Annie E. Casey Foundation Family to Family
Initiative in California

• Anne K. Abramson-Madden William C. Dawson
• Center for Social Services Research
• University of California Berkeley

2
Linking administrative data sets for
self-evaluation

• Mandatory outcome reporting with Adoption and
  Safe Families Act of 1997 (ASFA)
• Statewide reviews of selected indicators as part
  of Child and Family Services Reviews
• In California, the California Child Welfare and
  System Improvement Accountability Act (AB636)
  requires quarterly county reports

3
The Family to Family Initiatives Four Core
Strategies

• Recruitment, Training and Support of Foster and
  Kinship Families
• Building community partnerships
• Team Decision Making
• Self-evaluation

4
Team Decision Making (TDM)

• Meetings held to make placement decisions.
• Meetings are led by trained facilitators who are
  not the case-carrying social workers.
• Decision is reached by consensus with a safety
  plan in place. If consensus cannot be reached,
  agency is ultimately responsible for the
  decision.
• Family decides who makes up the team and may
  reject members. May also have community members
  childs caregivers on the team.
• Meetings generally last one to two hours.

5
CA F2F Implementation

• 24 of 58 California counties
• Approximately 88 of the 85,286 children in child
  welfare supervised foster care live in a Family
  to Family county

6
CA F2F Implementation (Contd)
Cluster Number of Counties Implementation Year(s)
I 7 2000-2003
II 4 2001-2003
III 5 2004-2005
IV 7 2004-2005
LA 1 (3 clusters by SPA) Various
7
Self-Evaluation in California

• Self-Evaluation is part of the Annie E. Casey
  Foundations Family to Family Initiative, also
  funded in CA by
• Stuart Foundation
• Walter S. Johnson Foundation
• California Department of Social Services

8
Family to Family Self-Evaluation in California

• Integration of data with practice
• Web reports using state administrative data
  provide information about child welfare outcomes
• TDM database
• Self-evaluation and quarterly reports
• Linkage to state administrative data has
  potential to examine implementation progress and
  child welfare outcomes

9
CA CWS/CMS

• Child Welfare Services Case Management System
  statewide computer system
• SB 370 (Chapter 1294, Statutes of 1989)
• Case management information for Child Welfare
  Services workers

10
CA CWS/CMS (Contd)

• Common database for Californias 58 counties
• Allows state and county administrators to monitor
  progress
• Consolidates information for state and federal
  reporting requirements

11
Administrative Data SourceUCB_FC at CSSR

• Longitudinal file containing foster care
  placement histories from 1998 to present
• Constructed from California's version of the
  federal Statewide Automated Child Welfare
  Information System (SACWIS)

12
Data Access and Analysis

• California Department of Social Services (CDSS)
  and the Stuart Foundation provide support for the
  California Performance Indicators Project (web
  reports)
• Analysts at CSSR produce a range of measures for
  use by California counties and the public
• http//cssr.berkeley.edu/cwscmsreports/

13
CWS/CMS reports
14
Self-Evaluation using TDM CA

• Customized Microsoft Access database
• Counties collect TDM meeting and child
  information
• Create reports regarding attendance, meeting
  participants, involved children, etc
• Counties produce quarterly report for
  self-evaluation

15
TDM CA Export Form
16
TDM CA to UCB_FC linkage

• Linkage allows counties to keep TDM data
  collection to a minimum, only focusing on items
  unique to TDM process while still capturing
  needed information about outcomes

17
Characteristics of Sample Counties
County 2004 Child Population (0-18) July 1, 2004 Child Welfare Caseload (0-18)
County 1 100,000-250,000 lt1,000
County 2 100,000-250,000 gt1,000
County 3 100,000-250,000 lt1,000
County 4 lt100,000 lt1,000
County 5 gt250,000 gt1,000
18
Preliminary Findings

• Five California Family to Family counties
• TDM database information on all children for
  whom placement recommendations were discussed in
  a TDM meeting
• UCB_FC contains information on all child
  welfare-supervised out-of-home placements
• TDM meetings and child welfare events (placement
  moves) restricted to Quarter 1, 2005
• (January 1, 2005-March 31, 2005)

19
Preliminary Findings (Contd)

• Data only as good as we get from counties- there
  may be errors (especially with respect to reason
  for involvement and recommendations)

20
Implementation Analysis

1. Start with a qualified event (entry, placement
   move, or exit).
2. What was the closest preceding event another
   child welfare event or a TDM meeting?
3. If a meeting, was it a related meeting? Count
   number of associated meetings.
4. Count remaining meetings without associated child
   welfare events.

21
Recommendation Analysis

1. Group children by reason for involvement and
   recommendation type.
2. Was there a related move during timeframe?
3. What was the actual move during the timeframe?
4. If both 2 and 3 match the recommendation, then
   the recommendation is achieved.
5. If recommendation achieved, then we look to the
   time to achievement.

22
Implementation Methods
TDM Data Start
TDM Data Cutoff (Current Quarter)
Qualified Moves
Time
TDM Entry Meeting
Entry to Care
1
TDM Entry Meeting
Entry to Care
2
3
1
TDM Plc Move Meeting
Placement Moves 6 Associated Meetings 3 Events
with Associated Meeting 50 Meetings without
Events 1
?
23
Implementation Summary Numbers
Section 1 Entered Placement Placement Move Exit from Placement
1) County CW Events 656 1110 479
2) Associated TDM Meet-Child Events 138 137 6
3) CW Events with Assoc. TDM Meet-Child Event 21.04 12.34 1.25
4) TDM Meet-Child Event with No Assoc. CW Event 124 171 27
24
Meeting Count Example County 4
Primary Reason For Meeting Number of Meetings Number of Unique Children Max Number of Meetings Per Child Mean Number of Meetings Per Child
Imminent risk of placement 12 15 2 1.07
Emergency placement 13 17 1 1.00
Placement move 25 30 2 1.07
Exit from placement 4 7 1 1.00
ALL 54 67 2 1.07
25
Event Count Example County 5
Primary Reason For Meeting Number of Meetings Number of Meeting/Child Events Max Number of Children Per Meeting Mean Number of Children Per Meeting
Emergency placement 42 74 6 1.76
Placement move 97 111 3 1.14
Exit from placement 2 2 1 1.00
ALL 141 187 6 1.33
26
Placement Move Meeting Attendance
Attendee Type Number of Meetings Number of Attendees Mean Attendees Per Meeting Number of Meetings Attended By At Least One Percent of Meetings Attended By At Least One
Facilitators 268 289 1.08 268 100.0
Supervisors 268 176 0.66 169 63.1
FR/PP Workers 268 170 0.63 157 58.6
Children 268 148 0.55 129 48.1
FFA Social Workers 268 149 0.56 94 35.1
Birth Parents 268 105 0.39 88 32.8
Relatives 268 176 0.66 81 30.2
27
Placement Move Meeting Attendance
Attendee Type Number of Meetings Number of Attendees Mean Attendees Per Meeting Number of Meetings Attended By At Least One Percentage of Meetings Attended By At Least One
Mental Health Staff 268 89 0.33 74 27.6
Other DSS Staff 268 76 0.28 70 26.1
FFA Foster Parents 268 84 0.31 66 24.6
Other Relative Caregivers 268 88 0.33 63 23.5
Other Service Providers 268 76 0.28 48 17.9
Interested Individuals 268 54 0.20 36 13.4
County Foster Parents 268 49 0.18 35 13.1
Family Maintenance Workers 268 39 0.15 32 11.9
28
Placement Move Meeting Attendance (contd)
Attendee Label Number of Meetings Number of Attendees Mean Attendees Per Meeting Number of Meetings Attended by at Least One Percent of Meetings Attended by at Least One
Adoptions Workers 268 31 0.12 28 10.4
CASA Advocates 268 31 0.12 27 10.1
Community Representatives 268 30 0.11 27 10.1
Other Social Workers 268 25 0.09 21 7.8
Other 268 128 0.48  
ALL 268 2013 7.51  
29
Recommendations Analysis

• Analysis restricted to Placement Move as the
  Childs Reason for Involvement
• Five counties 301 recommendations
• Possible recommendations include
• Change to less restrictive placement
• Maintain in present placement
• Change to same level placement
• Change to higher level placement

30
Placement Move Recommendation Change to Less
Restrictive Placement
31
Placement Move Recommendation Change to Less
Restrictive Placement
32
Placement Move Recommendation Maintain in
Present Placement
33
Placement Move Recommendation Maintain in
Present Placement
34
Limitations

• Data
• TDM data entry errors
• Missing data
• Analysis
• Logic errors
• Paper to Practice errors

35
Implications for Research

• Linking small database to Californias full child
  welfare system has huge potential
• Longitudinal nature of database has wealth of
  information about childrens lives and child
  welfare histories
• Ability to evaluate practice quarterly

36
Implications for Policy

• TDM reports can influence county boards and state
  policy makers, leading to change in child welfare
  services allocations
• Integrating practice and evaluation may serve as
  a model for future initiatives

37
Implications for Practice

• Access to data provides a feedback loop
• Agency workers (TDMS facilitators, analysts, and
  management) can easily see data regarding the
  TDMs to inform practice
• TDM reports provide a nice way to communicate
  with community partners, county boards, and other
  agencies involved

38
Next Steps

• Continue refining methods for linkage and
  expanding analysis
• Analyze entry and exit meetings
• Consult with counties regarding linkage results
• Work with counties to improve data accuracy

39
For more information

• Anne K. Abramson-Madden
• William C. Dawson
• TDMSupport_at_lists.berkeley.edu
• Child Welfare Services (CWS/CMS) Reports
• http//cssr.berkeley.edu/cwscmsreports/
• TDM CA Support Page
• http//cssr.berkeley.edu/tdm/