Nuts and Bolts on Collecting Family Outcomes Information

1
Nuts and Bolts on Collecting Family Outcomes
Information

• Kathy Hebbeler
• ECO at SRI International
• Prepared for the NECTAC National Meeting on
  Measuring Child and Family Outcomes, Albuquerque,
  NM
• April 2006

2
So state has decided its wants information from
families Now what?

• Many decisions related to collection of family
  data
• When to survey?
• How to get surveys to families?
• How to get surveys back?
• What about families who cant read?

3
New document from ECO

• Guidance to States for Documenting Family
  Outcomes for Early Intervention and Early
  Childhood Special Education
• Discussion of decisions, options and
  recommendations

4
3 Sections

• Measurement Decisions
• Implementation Decisions
• Instrument Options
• No instruments are included

5
General ECO Recommendations

• Involve stakeholders in deciding what outcomes to
  measure
• Collect data in addition to what is required by
  OSEP
• Conduct pilot studies to explore validity and
  usefulness of data being collected and share what
  you are learning

6
Examples of Measurement Decisions

• Who will provide the data?
• Should the items be closed or open-ended?
• What will be the format of the closed-ended
  items?
• How long will the instrument be?

7
ECO Recommendation Survey length

• No more than 20 minutes to complete, 10 to 15 is
  ideal

8
Implementation Decisions

• Document is focused on measuring family outcomes
  but the issues related to how to implement a data
  collection relate to any kind of family
  information
• Satisfaction
• Family involvement
• How much services helped, etc.

9
How will the assessment of family outcomes fit in
with other needs for information?
Existing State Surveys NCSEAM Survey ECO FO Survey
What services/ supports were provided? ? ?
Were families satisfied? ? ?
Did services help families meet each of the outcomes? ? ?
Were the outcomes achieved? ?
10
How will the assessment of family outcomes fit in
with other needs for information?

• ECO Recommendation
• Assess outcomes separately from satisfaction
• Minimize response burden
• Create a clear separation
• If must be on same survey, make a clear
  distinction between questions about service and
  questions about outcomes

11
Sample or all program participants?

• Sampling is cost effective
• Have to present by LEA or service units
• Need to know at how state wants to look at data
• Need to make sure there will be enough families
  in each cell

12
Sample or all program participants?

• ECO Recommendation
• Consult a sampling statistician
• Be prepared to discuss how state wants to look at
  the data and how much precision you want

13
When and how often should data be collected?

• Depends on how the state plans to use the data
• Minimum once at end
• But will miss lots of families who leave without
  announcing they are leaving
• Will need to be sending out surveys year round

14
When and how often should data be collected?

• Options
• Send in designated months (May and November)
• One designated time (May)
• May be the easiest to manage
• For tracking change over time, need to send at
  entry and then some interval(s) afterwards

15
When and how often should data be collected?

• ECO Recommendation
• Consider both what state wants to learn and what
  is feasible given the resources

16
Should surveys be anonymous?

• Anonymous no identifying information
• Anonymous ? confidential
• Almost never a reason to have a name on a survey
• ID number have advantages

17
Should surveys be anonymous?

• Advantages of IDs
• Can link data to other data
• Can ask questions about subsets of families
• Can link to the child outcome data
• Can track which kinds of families returned the
  survey

18
Should surveys be anonymous?

• If cant link, could add questions to survey to
  get at factors of interest BUT survey gets longer
• Focus group families had no problem with ids on
  surveys as long as surveys went to the state
  agency

19
Should surveys be anonymous?

• ECO Recommendation
• States need to consider whether there is a need
  to link survey data to other data

20
How should surveys be distributed to families?

• Survey distributed with
• Cover letter
• Survey
• If applicable, stamped addressed envelope for
  returning

21
How should surveys be distributed to families?

• Options
• Handed to the family by someone in the program
• If state doesnt have a data base, the program
  has to distribute
• Mailed from the program or the state

22
How should surveys be distributed to families?

• States can prepare the packets and send to locals
  for distribution
• Have to have surveys in appropriate language
• State would need to have this information to send
  the survey

23
How should surveys be distributed to families?

• ECO Recommendation
• Have a local provider hand the survey to the
  family, or
• Mail it from the state with a cover letter from
  the local program

24
How do families return the survey?

• ECO Recommendation
• Mail the survey back to the state agency

25
What can be done to maximize return rate?

• Target 70-80
• Problem of non-respondents
• Strategies
• Actively promote the importance of the survey and
  survey return to families
• Make sure families notice the survey.

26
What can be done to maximize return rate?

• Strategies (Contd)
• Remind families to complete the survey
• Postcards, second mailings
• Build in incentives to return the survey
• Can be chosen for a gift

27
What can be done to maximize return rate?

• Strategies (Contd)
• Build a system that allows tracking of surveys
  while they are being collected and provide
  programs feedback
• Implement a system of supports for families to
  complete survey
• Volunteer readers (parent groups?)
• Never use a program person

28
What can be done to maximize return rate?

• ECO Recommendation
• Implement a variety of strategies

29
Instrument Options

• Use existing measures
• Scale for each outcome
• All outcomes on the same scale
• Develop own measures

30
Other Decisions Not Addressed in This Document

• Analytic decisions
• How to report the data?
• What other factors or variables to look at?
• Reporting decisions
• Who gets what information in what form?