Why Collect Data on Child Outcomes

1
Why Collect Data on Child Outcomes?
Kathy Hebbeler Early Childhood Outcomes
Center SRI International
Prepared for Child and Family Measurement Project
Stakeholders Work Group September 2006
2
Why do we want good data on child and family
outcomes?

• Improve the lives of young children and their
  families
• But
• This is achieved in different ways by different
  players

3
Public Policy Context

• Age of accountability
• Accountability increasingly means looking at
  results not just process
• Applies across all private and public human
  service and education programs

4
The Need for Outcome Data at the Federal Level

• Accountability increasingly means looking at
  results not just process
• Office of Special Education Programs (OSEP) under
  increasing pressure to produce outcomes data for
  children and families participating in early
  intervention programs

5
Federal Need for Outcomes Data

• OSEP required states to submit outcomes data in
  their Annual Performance Report (APR) in 2004
• Reporting requirements revised in 2005 and 2006
• Entry data submitted in 2007
• Progress (entry to exit) data will be submitted
  in 2008

6
Need for Data at the Federal Level

• The push for child outcome data is not new only
  the reporting requirement is
• OSEP is under pressure from Office of Management
  and Budget (OMB)
• There could be serious fiscal implications for
  Part C if data on child outcomes are not
  forthcoming
• Read more at http//www.whitehouse.gov/omb/expectm
  ore/summary.10000190.2005.html

7
Why might Washington want good outcome data?

• To assure comprehensive, quality services for
  Washington children and families
• To improve existing programs
• To justify funding for programs for young
  children and their families

8
Why might Washington want good outcome data?

• Also, because the federal government requires it
  and gives the state money to enhance systems
  including collecting the data
• This purpose cannot be ignored but it doesnt
  have to drive the system

9
First Decision for Work Group

• Why does Washington want data on child outcomes?
• Need to be clear on why the state is doing this
• Purpose drives many other decisions
• What are some options?
• (Remember the parameters)

10
Many Players Want Outcome Data
Federal administrators and policy-makers
State policy-maker and administrators
Local administrator and policy-makers
Program Directors
Distance from child
Teachers, Providers, Early Interventionists, etc.
Parents/Families
11
But the different players need data..

• For different purposes
• At different levels of detail

12
Different Roles, Same Need for Data

• Federal government
• Accountability (is the money being spent well?)
  ? More money?
• Planning for support investments
• Oversight, monitoring

13
Different Roles, Same Need for Data

• State government
• Demonstrate programs are high quality
• Accountability (is the money being spent well?) ?
  More money?
• Planning for support investments
• Oversight, monitoring

14
Different Roles, Same Need for Data

• Upper management, local areas
• Accountability (is the money being spent well?) ?
  More money
• Oversight, monitoring
• Program development
• Staff development
• Supervision

15
Different Roles, Same Need for Data

• Program Directors
• Program development
• Staff development
• Supervision
• Program development identification of weak areas
• Identification of children needing extra help

16
Different Roles, Same Need for Data

• Teachers, Caregivers, Early Interventionists,
  etc.
• Curriculum planning identification of weak areas
• Effectiveness of intervention strategy
• Identification of children needing extra help
• Communicate child progress to parents

17
Different Roles, Same Need for Data

• Parents/Families
• Track their childs progress
• Understand their childs development and what
  will happen next
• Be informed about areas of concern
• Examine overall effectiveness of the program

18

• All of these uses are legitimate and important
• What uses do you want to see in your state?

19
Designing an accountability system

• Developing an outcomes system requires many
  interrelated decisions
• Choosing one course will close the gate to other
  courses
• There is no perfect or easy course
• Not designing a system is not an option

20
Using Date for Program improvement

• Outcomes data alone are of very limited
  usefulness for program improvement
• Need to be part of an evaluation system that also
  looks at implementation issues (e.g., program
  quality, services received, procedural issues,
  satisfaction)

21
Simplistic Logic Model
Good outcomes for children and families
High quality services for children 0-5 and their
families
Appropriate State Policies and programs
Appropriate Local Policies and Programs
Appropriate Federal policies and programs
Supports
22
Questions

• Whose needs does Washington want to address with
  its child outcomes system?
• What purposes will the system address?

23
(No Transcript)
24

• What Outcomes Should Be Measured?

25

• Decision Should the child outcomes system be
  structured around the 3 OSEP outcomes?

26
3 Child Outcomes

• Children have positive social-emotional skills
  (including social relationships)
• Children acquire and use knowledge and skills
  (including early language/ communication)
• Children use appropriate behaviors to meet their
  needs

27
Options

• Add more outcomes
• Children will be healthy
• Add sub-outcomes under the outcomes
• Use Early Learning Guidelines as organizing
  framework
• Adopt the OSEP outcomes
• (Remember the parameters)

28
How do these differ from IFSP outcomes?

• OSEP outcomes are global
• Meant to apply to all children
• IFSP outcomes are specific outcomes that apply to
  an individual child and family
• Both kinds of outcomes are important

29
(No Transcript)
30

• Measuring the Outcomes

31
What is Assessment?

• Assessment is a generic term that refers to the
  process of gathering information for
  decision-making.
• (McLean, 2004)

32
What is Assessment?

• Early childhood assessment is flexible,
  collaborative decision-making process in which
  teams of parents and professionals repeatedly
  revise their judgments and reach consensus ...
• Bagnato and Neisworth, 1991
• Quoted in DEC Recommended Practices, 2005

33
DEC Recommended Practices for Assessment

• Involves multiple sources
• Examples families, professional team members,
  service providers, caregivers
• Involves multiple measures
• Examples observations, criterion- or
  curriculum-based instruments, interviews,
  curriculum-compatible norm-referenced scales,
  informed clinical opinion, work samples

34
Assessment Instruments

• Assessment tools can inform us about childrens
  functioning in each of the 3 outcome areas
• Challenge There is no assessment tool that
  assesses the 3 outcomes directly

35
Outcomes are Functional

• Functional refers to things that are meaningful
  to the child in the context of everyday living
• Refers to an integrated series of behaviors or
  skills that allow the child to achieve the
  outcomes
• They are not
• a single behavior, nor are they
• the sum of a series of discrete behaviors

36
Outcomes are Functional

• Integrate behavior across domains not trying to
  separate child development into discrete areas
  (communication, gross motor, etc.)
• Emphasize how the child is able to carry out
  meaningful behaviors in a meaningful context

37
Assessment Tool Lens

• Each assessment tool carries its own organizing
  framework
• Many are organized around domains
• But what is covered in the domains isnt always
  the same, even if the names are the same

38
Something to Think About

• Good outcomes data requires good assessment
  procedures
• How good are current assessment practices?
• Providers may need to learn new or different ways
  to do assessment

39
Options Other States Have Chosen Assessment
Tools

• One instrument statewide
• Menu of approved instruments
• Any instrument programs use

40
Problems Posed by Multiple Assessment Tool

• Different assessment tools produce different
  kinds of scores
• Need a way to combine data across tools
• Need to aggregate data on 100s of kids

41
Why is There a Need for the Child Outcomes
Summary Form?

• No assessment instrument assesses the 3 outcomes
  directly
• Different programs will be using different
  assessment instruments and outcomes data will
  need to be aggregated across programs

42
Features of the Child Outcomes Summary Form

• Not an assessment tool
• Uses information from assessment tools and
  observations to get a global sense of how the
  child is doing at one point in time

43
Features of the Child Outcomes Summary Form

• 7-point rating scale
• Rating is based on childs functioning
• what child does across settings and situations
• compared to age expectations

44
Essential Knowledge for Completing the COSF

• Between them, team members must
• Know about the childs functioning across
  settings and situations
• Understand age-expected child development
• Understand the content of the 3 child outcomes
• Know how to use the rating scale

45
The Form

• Cover page 3 outcome pages
• On each outcome page
• 2 questions per outcome
• Space to document the basis for the
  rating

46
Child Outcomes Summary Form
47
The 2 COSF Questions

• To what extent does the child show
  age-appropriate functioning, across a variety of
  settings and situations, on this outcome?
  (Rating 1-7)
• Has the child shown any new skills or behaviors
  related to this outcome since the last outcomes
  summary? (yes-no)

48
Summary Rating (1-7)

• Reduces rich information from assessment and
  observation into a rating to allow a summary of
  progress across children
• Does not provide information for planning for the
  individual child. Information at the rich,
  detailed level will be more helpful for
  intervention planning purposes.

49

• What Role Will Parents and Other Family Members
  Play?

50
Learning about the Childs Functioning

• Input from parents or other family members about
  the childs functioning is critical
• Family members see the child in situations that
  professionals do not
• Need to ask family members about what the child
  does at home
• Need a way to learn about what family members
  know about the child

51
Learning about the Childs Functioning

• Options
• Assessment that includes information from parents
• Interview process that collects information
• Team discussion where parent contributes
• All of the above
• Need a way to learn about what family members
  know about the child

52
Including Parents in the
Deciding on a Rating

• No expectation that all parents will be able to
  determine if what they are seeing is age
  appropriate
• No consensus around the country on whether
  parents should be included in deciding on the
  summary rating. Even parent groups dont agree.

53
State Options

• Include parents in the rating discussion
• Explain the process and give parents the option
• Let program or providers decide on a case by case
  basis
• Dont include parents in the rating discussion
• (Remember the parameters)

54
Explaining the Rating to Parents

• If parents are included in deciding on a rating,
  professionals will need to be able to explain
  this process to parents
• Even if parents are not included in deciding on a
  rating, professionals will need
  to be able to explain why the rating is being
  done and what it means
• The ECO Center is developing materials to help
  with this discussion