Are your C4 data reflective of the families you serve?

1

• Are your C4 data reflective of the families you
  serve?

Joy Markowitz, Director Jean Dauphinee, TA
Specialist
Measuring Child and Family Outcomes Conference,
August 28, 2008
2
Purpose of Collecting C4 data

• To figure out how to serve our families better
  specifically related to Indicator C4
• Percent of families participating in Part C who
  report that early intervention services have
  helped the family
• A Know their rights
• B. Effectively communicate their children's
• needs and
• C. Help their children develop and learn.

3
Why do we care who responds to our surveys?

• We dont want to invest time and money in
• improvement activities if data do not reflect the
  
• opinions and/or experiences of the
• families/children we serve.
• This is an issue if you collect data from all
• families (census) or a sample of families
• because not all families who receive a survey
• respond.

4
Population

• Population may be defined by the state and could
  be (for example)
• all Part C families
• all Part C families exiting in a given year, or
• all Part C families in program more than 1 year.

5
Other terms used

• Population
• Target group
• Target population

• Respondents
• Response Pool
• Respondent Pool
• Respondent Group

6
Nonresponse Bias

• Representativeness is a word we have been
• using. Its not a statistical term.
• Correct term nonresponse bias.
• Meaning Are the families who did not
• respond different from the families that did?

7
What is a Sample?

• A sample is a subset of the population that
• you define. However, this sample must
• be derived using an approved sampling
• plan.
• OSEP collaborates with DAC to review and
• approve submitted sampling plans.

8
Questions about Response Rates

• What is high enough?
• How do you determine if your respondents
• are reflective of the population you
• defined?
• Whether using census or sampling, your
• return or response rate is unlikely to be as
• high as you want it to be therefore, you must
  
• address nonresponse bias.

9
Example 1
Families served in 2006-2007 (n100)

• Survey respondents
• Black 24
• White 44
• AI/AN 2
• Total 70

• Families served
• Black 30
• White 50
• AI/AN 20
• Total 100

Do these data reflect the population?
10
Example 2
Families exiting in 2006-2007 (n100)

• Survey respondents
• English 65
• Other language 5
• Total 70

• Families served
• English 75
• Other language 25
• Total 100

Do these data reflect the population?
11
Example 3
Families served in programs for at least one year
(n100)

• Families served by agencies
• Agency 1 100
• Agency 2 100
• Agency 3 100
• Agency 4 100
• Total 400

• Survey respondents
• Agency 1 49
• Agency 2 63
• Agency 3 66
• Agency 4 21
• Total 199

Do these data reflect the population?
12
State Data Collection Methods/Strategies

• Analysis strategies
• Frequencies and
• percentages
• Weighting
• Sampling among
• respondents

• Methods
• Census (n41 states)
• Sample (n 15 states)
• oversampling populations that are known to be
  hard to reach.

13
Summary of C4 Findings(Based on analysis of 54
state 2006-2007 APRs)

• 67 (n 36 states) reported nonresponse bias, 26
  (n14 states) did not.
• Among the 67 who reported nonresponse bias, most
  reported by one variable (e.g. race/ethnicity,
  region, gender, or childs age)

14
Summary of C4 Findings(Based on analysis of 54
state 2006-2007 APRs) cont.

• Most states measured nonresponse bias by
• Race/ethnicity or
• Race/ethnicity and other factors such as
• length of time in program,
• region of the state,
• childs age, or
• gender.

15
From SPP/APR Instruction Sheet

• States are allowed to use sampling when so
  indicated on the Part C Indicator Measurement
  Table. When sampling is used, a description of
  the sampling methodology outlining how the design
  will yield valid and reliable estimates must be
  submitted to OSEP. The description must describe
  the
• (a) sampling procedures followed (e.g.,
  random/stratified, forms validation) and
• (b) the similarity or differences of the sample
  to the population of children with disabilities
  in the early intervention program (e.g., how all
  aspects of the population such as disability
  category, race, age, gender, etc. will be
  represented).

16
From SPP/APR Instruction Sheet (cont.)

• The description must also include how the Lead
  Agency addresses any problems with
• (1) response rates
• (2) missing data and
• (3) selection bias.
• Samples from EIS programs must be representative
  of each of the EIS programs sampled, considering
  such variables as eligibility definition
  (diagnosed condition or developmental delay),
  age, race, and gender.

17
From SPP/APR Instruction Sheet (cont.)

• In reporting on the performance of small EIS
  programs, the Lead Agency shall not report to the
  public or the Secretary any information on
  performance that would result in the disclosure
  of personally identifiable information about
  individual children or where the available data
  is insufficient to yield statistically reliable
  information, i.e., numbers are too small.
• Source Part C State Performance Plan (SPP) and
  Annual Performance Report (APR) Instruction Sheet
  dated 10/19/2007.

18
Examples from States

• How have states tackled these issues?
• State representatives
• Wendy Whipple, NV
• Sue Campbell and Rosanne Griff-Cabelli, DE
• Alice Ridgeway, CT

19
Questions?

• What questions are you getting from OSEP that you
  need help to answer?
• What guidance do you need from OSEP?
• What are realistic expectations of states for
  reporting nonresponse bias?