Measuring Progress: Improved Outcomes for Children and Families

1
Measuring Progress Improved Outcomes for
Children and Families

• Kathy Hebbeler
• The Early Childhood Outcomes Center
• SRI International

Virginia EI Conference Creating Connections
Strengthening Partnerships between Families and
Providers Roanoke, VA, March 2007
2
Objectives

• Review history and rationale related to the
  selection of child indicators
• Describe what we have learned about implementing
  measurement systems
• Describe why child indicators are important for
  improved services

3
Goal of the ECO Center

• Promote the development and implementation of
  child and family outcome measures for infants,
  toddlers and preschoolers with disabilities that
  can be used in local, state, and national
  accountability systems

4
Being clear

• For VA, the word indicator refers to the 3 OSEP
  outcomes
• to distinguish them from IFSP outcomes
• ECO Centers materials use the word outcomes to
  refer to both
• I will try to use indicator for the OSEP
  outcomes unless I mean all kinds of outcomes

5

• What are Outcomes?

6
The Concept of Outcomes

• An outcome is the result of some action or set of
  actions
• Often expressed in the form of a statement
• Examples
• All passengers will wait 10 minutes or less in
  the airport security line.
• All children will be fully immunized by 2.

7
The Concept of Outcomes

• Two parts
• The expected result
• The action(s) that produces the result
• May not be stated but always present
• Outcomes are the effect in cause and effect

8
Outcomes Closer to Home

• EI programs provide high quality services and
  supports
• EI services are family centered
• Families help their child develop and learn
• Children take appropriate action to meet their
  needs

9
Different Kinds/Levels of Outcomes

• Program or systems outcomes
• EI programs provide high quality services and
  supports
• EI services are family centered
• EI services are coordinated
• What are the actions/activities that produce
  these results?

10
Different Kinds/Levels of Outcomes

• Child or family outcomes
• Families know their rights and advocate
  effectively for their children
• Children have positive social relationships
• What are the actions/activities that produce
  these results?

11
Logic Model for Child and Family Outcomes
Adequate funding
Good outcomes for children and families
High quality services and supports for children
0-5 and their families
Good State policies and programs
Good Local policies and programs
Good Federal policies and programs
Strong Leadership

• Supports
• Preservice training
• Inservice training

12
Two Critical Outcome/Indicator Questions for
Building High Quality Programs

• What is the intended result?
• What is the action/set of action that will
  produce the intended result?

13

• How Looking at Child Indicators
• for EI Became Important

14
Focus on Accountability

• Followed an era of focusing on and measuring
  actions/activities
• Call for interest in the ultimate result child
  and family outcomes/indicators
• Cut across both the public and private sectors
• Funders demanded data to determine whether a
  program was doing what it was supposed to do

15
Critical Events in EI Accountability

• 1992 Osborne and Graebler, Reinventing
  Government
• 1993 GPRA (Government Performance and Results
  Act) passed
• Intervening years.
• 2002 PART finds there are no data on outcomes
  for Part C
• 2003 OSEP begins to ask states for EI child
  outcome data (and funds the ECO Center!)
• 2005, 2006 OSEP releases revisions to the
  reporting requirements

16
PART Review for Part C and Part B Preschool

• Results Not Demonstrated
• Part C
• While the program has met its goal relating to
  the number of children served, it has not
  collected information on how well the program is
  doing to improve the educational and
  developmental outcomes of infants and toddlers
  served.
• Part B Preschool
• The Department has no performance information
  on preschool children with disabilities served by
  this program.
• Read more at Expectmore.gov

17
Intervening Years

• Special Education
• National study found poor outcomes for special
  education adolescents Results
• Push to include students with disabilities in
  statewide assessment systems
• Early Childhood
• Debate about whether child outcomes should be
  measured at all
• Discussion of all the problems in trying to
  measure outcomes for young children with
  disabilities

18
Measuring Child Indicators for EI

• The PART findings put an end to the debate about
  whether or not to do it
• Unfortunately, almost no progress had been made
  in the intervening years as to HOW to do it

19
Meanwhile Re-thinking Assessment in Early
Childhood

• Major changes in last 15 years in how assessment
  of young children is viewed
• Old position Do not test little kids
• New position Ongoing assessment is part of a
  high quality early childhood program

20
What changed

• New and different tools became available
• Curriculum-based assessments were developed
• General EC Tools for 3-5 came first 0-3 tools
  are coming now
• Interesting sidebar Curriculum-based
  assessments for programs serving children 0-5
  with disabilities have been around for years

21
What changed

• The purpose of assessment was redefined
• Not about sorting, labeling, using to deny
  access (the unready)
• Now about Getting a rich picture of what
  children can do and cant do and using that
  information to help them acquire new skills
• progress monitoring

22
What changed

• Assessment had always been seen as a process with
  multiple purposes
• Distinctions were made between good and bad uses
  of assessment with young children
• Good uses are now promoted
• For more information NAEYC web site (Position
  statement on Curriculum, Assessment and
  Evaluation)

23
Interesting Irony

• Even though the disability community had
  developed many curriculum-based assessment
  tools, currently many programs do not
  practice ongoing assessment
• Assess for eligibility only
• The push for ongoing assessment to monitor how a
  child is doing and plan for instruction/interventi
  on is coming from the general education community

24
Where We Are Now

• Push for measuring outcomes for accountability
  (top down but not just federal)
• Push for ongoing assessment of progress
  (monitoring outcomes) as best practice in EC
  programs
• Done well this could be a powerful blending of
  forces to improve outcomes for children

25

• Why These 3 Indicators?

26
OSEP Reporting Requirements Child Indicators

• Positive social emotional skills (including
  positive social relationships)
• Acquisition and use of knowledge and skills
  (including early language/ communication and
  early literacy)
• Use of appropriate behaviors to meet their needs

27
Origin of the Child Indicators

• ECO Center stakeholder meetings followed by
  public comment period (2004)
• First, collected themes and ideas
• Then, drafted and re-drafted indicator wording
• Made recommendation to OSEP (2005)

28
Themes from the Stakeholders Child Indicators

• Consistent with IDEA and legislative intent
• Reflect what EI and ECSE are trying to do
• One set for birth to 5
• Reflect what is known about development and
  learning

29
Themes from the Stakeholders

• Be compatible with best practice (esp.
  transdisciplinary service models, functional
  behaviors)
• Do not base them on domains
• Have potential to influence practice in a
  positive way
• Incorporate universal design
• Be readily understood

30
Make Outcomes/Indicators Functional

• Functional refers to things that are meaningful
  to the child in the context of everyday living
• Refers to an integrated series of behaviors or
  skills that allow the child to achieve the
  outcomes
• They are not
• a single behavior, nor are they
• the sum of a series of discrete behaviors

31
Functional

• What does a child typically do?
• Actual performance across settings and situations
• How child uses his/her skills to accomplish tasks
• Not the childs capacity to function under
  unusual or ideal circumstances

32
Recommendations

• Decision One set of indicators birth to 5
• Decision Functional outcomes
• Decision Global, not specific

33
The need for an introductory statement

• Family and child outcomes are linked
• There are overarching goals for children and
  family that cut across the outcomes
• Helpful to frame the outcomes with these
  overarching goals

34
Concepts in the introduction Goal for children

• The ultimate goal is for young children to be
  active and successful participants now and in the
  future in a variety of settings in their homes,
  in their child care, preschool or school
  programs, and in the community

http//www.fpg.unc.edu/eco/pdfs/eco_outcomes_4-13
-05.pdf
35
Concepts in the introduction Goal for children

• Active and successful participants
• Now and in the future
• In variety of settings

36
Concepts in the introduction Goal for families

• Enable families to provide appropriate care for
  their child
• Have resources they need to participate in
  community activities

37
Concepts in the introductionEffective programs

• Support families in their quest for a
  satisfactory quality of life
• Provide needed supports and services in a timely
  fashion

Remember Outcomes are the result of actions
38
Concepts in the preface Outcomes and
accountability

• Outcomes determined by variety of factors
• Not all families and children will achieve all
  outcomes
• BUT, system should still strive to achieve them

39
ECO Family Outcomes

• Understand their childs strengths, abilities,
  and special needs
• Know their rights and advocate effectively for
  their children
• Help their children develop and learn
• Have support systems
• Access desired services, programs, activities in
  their community

40

• The OSEP Reporting Categories

41
OSEP Reporting Categories

• Percentage of children who
• a. Did not improve functioning
• b. Improved functioning, but not sufficient to
  move nearer to functioning comparable to
  same-aged peers
• c. Improved functioning to a level nearer to
  same-aged peers but did not reach it
• d. Improved functioning to reach a level
  comparable to same-aged peers
• e. Maintained functioning at a level comparable
  to same-aged peers

3 outcomes x 5 measures 15 numbers
42
Functioning
13
43
Entry
14
44
Entry
Exit
15
45
Entry
Exit
16
46
CISF Numbers to OSEP Reporting Categories

• Each possible combination of numbers (and answer
  to the b question at entry and exit generates a
  line, i.e., a developmental trajectory)
• Each trajectory corresponds to an OSEP category
• Taking measurements more frequently than entry
  and exit generates more powerful information

47
Functioning
13
48
OSEP Reporting Categories Child Outcomes

• Percentage of children who
• Did not improve functioning
• Improved functioning, but not sufficient to move
  nearer to functioning comparable to same-aged
  peers
• Improved functioning to a level nearer to
  same-aged peers but did not reach it
• Improved functioning to reach a level comparable
  to same-aged peers
• Maintained functioning at a level comparable to
  same-aged peers

3 outcomes x 5 measures 15 numbers
49
Comparing to Same Aged Peers

• Deficit model? Not individualized?
• Goal of EI Active and successful participation
  now and in the future
• Kindergarten, school readiness, having friends,
  community participation
• Setting high expectations
• Moving from 1 to 2 moving from
• 4 to 5
• How much progress is enough?

50

• COSF/CISF Implementation Issues

51
Overall

• Child indicators have been very well received
• Many states have embraced outcomes measurement at
  some level

52
(No Transcript)
53
Assessing functioning

• Process helps teams think about the child
  functionally across a variety of settings and
  situations
• Requires good assessment practices (and suggests
  need for improving current practice)
• Assessment only at entry. What is the role of
  assessment in EI?
• Moving from domains to functioning

54
Assessing functioning

• Providers need a way to have a conversation with
  families about child functioning across settings
  and situations
• Providers need training in explaining assessment
  results to parents
• soft pedal

55
Age Expectations

• Requires providers understand typical child
  development and how children of different ages
  are expected to function in all 3 indicators
• Reveals need for training in this area

56
Working with Families

• Changing the conversation between providers and
  families?
• Changing the nature of IFSPs?
• Do we write IFSPs to these indicators?
• Would that be such a bad thing?

57
Opportunity?

• Measurement of indicator data can encourage more
  frequent collection of assessment data
• Progress monitoring
• Improve practice
• And provide outcome indicators

58
Validity and Reliability

• Validity is not a characteristic of a tool
• Validity is a characteristic of a set of scores
  produced by a tool
• CISF is judgment-based
• Substantial research to support that with defined
  criteria, providers and parents can reach
  reliable judgments

59
Validity and Reliability

• ECO working on procedures to help states assess
  validity of their data
• Good training and ongoing supervision will be
  critical
• Quality assurance will be critical

60
Some Concerns

• Being implemented very quickly
• Not all implementers are well trained
• Few states have procedures in place for assuring
  the process is high quality and reliable
• States doing the minimum will not have very
  useful data

61
How Will These Data Be Used?

• Hopefully not at all -- UNTIL checks are in place
  to insure they are of high quality
• Meaning needs to be attached to the findings by
  those closest to the program
• Craft your own message

62
Next steps for ECO

• More support and training materials
• More guidance on supervision and quality
  assurance
• More guidance on using data for program
  improvement

63

• A Vision
• (What Does This Mean for You)

64
Logic Model for Child and Family Outcomes
Adequate funding
Good outcomes for children and families
High quality services and supports for children
0-5 and their families
Good State policies and programs
Good Local policies and programs
Good Federal policies and programs
Strong Leadership

• Supports
• Preservice training
• Inservice training

65
Building a high quality system

• What if every piece of the system functioned as
  it was supposed to?
• What information is needed to know if this is
  happening?
• What information is needed to know how to
  identify and address weaknesses in the system?

66
Your contribution

• Where do you fit?
• What is your contribution to good outcomes? What
  actions do you take that produce good child and
  family outcomes?
• What information would you need to know if your
  actions were producing good outcomes?

67

• The ideal outcome system gives persons at each
  level of the system the information to know if
  they are producing good outcomes -and if not, to
  change their practices/policies/procedures so
  eventually they will.

68
Remember the goal for children

• Active and successful participation now and in
  the future

69
And an overarching goal for families

• To enable families to provide care for their
  child and have the resources they need to
  participate in their own desired family and
  community activities

70
The Take Home Message

• Measuring outcomes is not primarily about data
• It is about doing good things for children and
  families
• And using data as a tool to help programs and
  providers to know whether what they are doing is
  making a difference

71
Finding out more

• Additional information about OSEP requirement and
  state activities
• New resources will be coming including training
  materials, resources on assessment and typical
  child development, and materials for parents
• www.the-eco-center.org