Understanding and Using the Results from the NCSEAM Family Survey

1
Understanding and Using the Results from the
NCSEAM Family Survey

• Batya Elbaum, Ph.D.
• NCSEAM
• Measuring Child and Family Outcomes
• NECTAC National TA Meeting
• Baltimore, MD, August 2007

2
Part C Indicator 4

• Percent of families participating in Part C
  who report that early intervention services have
  helped the family a) know their rights, b)
  effectively communicate their childrens needs,
  and c) help their children develop and learn.

3
Sample results

• Mean 594, S.D. 147, S.E. 10
• Percent of families who report that early
  intervention services have helped the family
• a) know their rights 59
• b) effectively communicate their childrens needs
  54
• c) help their children develop and learn 65

4
What do the numbers mean?

• Interpreting the measures and percentages

5
Measuring the 3 subindicators

• To measure something means to locate it on a
  continuum.
• Do we need three different measurement rulers for
  the three specific OSEP outcomes, or can we use a
  single measurement ruler?

6
Measuring the 3 subindicators

• NCSEAM analyses of responses from thousands of
  families confirmed that the three outcomes
  selected by OSEP as family outcome indicators are
  all part of a single hierarchy of positive family
  outcomes.

7
Measuring the 3 subindicators

• Families appear to achieve these outcomes in a
  very consistent order.
• Families who report that EI helped them know
  their rights almost invariably report that EI
  also helped them help their child develop and
  learn.
• Families who report that EI helped them
  effectively communicate their childrens needs
  almost invariably report that EI also helped them
  know their rights and help their child develop
  and learn.

8
Locating families on the pathway

• The data tell us that there is a continuum of
  Impact of EI Services on Families, such that
  the impact can range from very low to very high.
• ?----------------------------------------
  -------------?
• Very Low
  Very High
• Different families experience different levels of
  positive impact as a result of their
  participation in early intervention, and so
  different families will be located at different
  points along this continuum.

9
Locating families on the pathway

• When we put numbers on the continuum, we create a
  measurement ruler measuring the impact of EI
  for each family
• Fam-1 Fam-2
  Fam-3
• ? ?
  ?
• ?----------------------------------------?
  
• 400 500 600 700
  

10
Item locations on the ruler
11
Location of key items related to Indicator C4
Indicator 4b Effectively communicate their
childrens needs. IFS Item Communicate more
effectively with the people who work with my
child and family. 556
556
Indicator 4a Know their rights. IFS Item Know
about my child's and family's rights concerning
Early Intervention services. 539
539
Indicator 4c Help their children develop and
learn. IFS items Understand my child's special
needs. 516 Do things with and for my child
that are good for my child's development. 498
516
12
Measurement of family outcomes at different levels

• Just as we can locate individual families on the
  ruler, we can take the average measure for all
  families in the state and locate the state on the
  ruler.
• Similarly, we can take the average of all
  families in a particular program and locate that
  program on the ruler.

13
What is the meaning of the percentage?

• The percentage reported to OSEP is the percent of
  families with measures that are at, or above, a
  certain point on the measurement ruler.

14
Percent of measures above the adopted standards
standards
15
Sample descriptive interpretation

• Approximately 90-95 of families agreed, with
  approximately two-thirds of families expressing
  strong or very strong agreement, that early
  intervention helped them
• Understand their childs special needs.
• Do things with and for their child that are good
  for their childs development.
• Feel that they can handle the challenges of
  parenting a child with special needs.

16
Sample descriptive interpretation

• Approximately 80-90 of families agreed, with
  approximately 50-60 expressing strong or very
  strong agreement, that early intervention helped
  them
• Know about their childs and familys rights
  concerning early intervention services.
• Know where to go for support to meet their
  childs needs.
• Be more effective in managing their childs
  behavior.

17
Sample descriptive interpretation

• Approximately 70-75 of families agreed, with
  about 45 expressing strong or very strong
  agreement, that early intervention helped them
• Communicate effectively with the people who work
  with their child and family.
• Know about services in the community.

18
Sample descriptive interpretation

• Approximately 65-70 of families agreed, with
  about 40 expressing strong or very strong
  agreement, that early intervention helped them
• Participate in typical activities for children
  and families in the community.

19
The meaning of agreement

• The raw percentages of families that agree (in
  any category of agreement) with the key items
  related to the indicators is higher than the
  percentages reported on the indicator. Why?
• Because the standard that most states adopted for
  the indicators is a higher standard than a simple
  agree.

20
Evaluating the percentages on the indicator

• Against what standard is the percentage being
  evaluated?
• What percent of families do we want to be at or
  above the measure that represents our idea of
  where families should be on the continuum of
  positive family outcomes?

21
The APR Report on C4
22
What to cover in relation to your data

• Sample
• Method
• Results
• Interpretation

23
Sample

• Describe the sampling plan
• Describe the extent to which the obtained sample
  is representative of the population
• State your survey response rate

24
Method Instrument

• Describe the instrument
• Provide a copy of the instrument
• Report reliability estimates for the instrument
  (based on NCSEAM pilot, .90 or above using 22
  items)

25
Method Procedure

• Describe how the survey was distributed/administer
  ed (face-to-face, mail survey, online)
• State whether the survey was available in
  languages other than English
• Describe efforts to ensure the participation of
  low-literacy families

26
Results Data analysis

• Describe how the data were analyzed
• Describe how the percent on the subindicators was
  calculated

27
Results Reliability

• Report the reliability of the data
• Report a confidence interval for the percent on
  each subindicator (this confidence interval will
  depend on the size of your sample)

28
Results Validity

• Face validity Does the instrument appear to
  address what we are supposed to be measuring?
• Convergent validity Do the results accord with
  results obtained from other sources (e.g.,
  monitoring data) that address the same thing?

29
Interpretation

• Report progress (or slippage) in relation to
  target
• Consider the results in relation to
• Effectiveness of improvement activities
• Extent of implementation of improvement
  activities

30
Improving Performance on the Indicator
31
Short-term goals

• Increase response rates
• Ensure equal access to participation
• Build a broad base of understanding of the
  indicator and how it is measured

32
Longer-term goals

• Improve performance on the indicator
• Document implementation of improvement activities
• Examine the relation between implementation and
  improvement so as to better understand what
  works
• Improve performance on the indicator for all
  subgroups

33
Further information

• www.monitoringcenter.lsuhsc.edu
• elbaum_at_miami.edu