Part C and Preschool Child Outcome Indicators

1

Progress toward Measuring Goals in Early
Intervention Whats New from What Counts
Kathy Hebbeler ECO at SRI International
Hawaii January, 2008
2
Objectives

• Review why data are being collected
• Describe national trends
• Identify and address challenges to good data
• Discuss some preliminary data from Hawaii

3

• Why are we doing this?

4

• Keeping our eye on the prize
• High quality services for children and families
  that will lead to good outcomes.

5
High Quality Data on Outcomes

• Data are a piece of a system that helps to
  achieve overarching goals for children and
  families
• Data yield
• Findings that can be interpreted as having a
  particular meaning that should lead to specific
  actions to improve the system.

6
System for Producing Good Child and Family
Outcomes
Adequate funding
Good outcomes for children and families
High quality services and supports for children
0-5 and their families
Good Federal policies and programs
Good State policies and programs
Good Local policies and programs
Strong Leadership

• Profl Development
• Preservice
• Inservice

7
The Vision Using Data as a Tool for Program
Improvement

• Hawaii will have quality data available on an
  ongoing basis about multiple components of the
  system
• Goals for children and families
• Services provided
• Personnel (types, qualifications, etc.)
• Etc.

8
Driving Force for Data on Child Goals Comes from
the Federal Level

• Government Performance and Results Act (GPRA)
• Program Assessment Rating Tool (PART)
• Individuals with Disabilities Education Act (IDEA)

9
Government Performance and Results Act (GPRA)
passed in 1993

• Requires goals and indicators be established for
  IDEA
• Indicators and data collection further along for
  school age population than for EC
• Previously, for early childhood data had been
  collected on
• Number of children served (Part C)
• Settings (both Part C and 619)

10
OSEP PART evaluation results (2002)

• 130 programs examined in 2002 50 programs had
  no performance data
• Programs looking at inputs, not results
• Part C and Section 619
• No long-term child outcome goals or data
• Need to develop a strategy to collect annual
  performance data in a timely manner

11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
Federal Funding for Early Intervention
Total U.S. Hawaii
2004 444,362,700 2,177,738
2005 440,808,096 2,160,317
2006 436,399,920 2,138,714
15
Individuals with Disabilities Education Act
SEC. 616. ltltNOTE 20 USC 1416.gtgt MONITORING,
TECHNICAL ASSISTANCE, AND ENFORCEMENT. (a)
Federal and State Monitoring.-.. .. (2)
Focused monitoring.--The primary focus of Federal
and State monitoring activities described in
paragraph (1) shall be on-- (A) improving
educational results and functional outcomes for
all children with disabilities
16

• Where are we now
• Federal reporting requirements

17
OSEP Reporting Requirements the Goals

• Positive social emotional skills (including
  positive social relationships)
• Acquisition and use of knowledge and skills
  (including early language/ communication and
  early literacy)
• Use of appropriate behaviors to meet their needs

18
OSEP Reporting Categories

• Percentage of children who
• a. Did not improve functioning
• b. Improved functioning, but not sufficient to
  move nearer to functioning comparable to
  same-aged peers
• c. Improved functioning to a level nearer to
  same-aged peers but did not reach it
• d. Improved functioning to reach a level
  comparable to same-aged peers
• e. Maintained functioning at a level comparable
  to same-aged peers

3 outcomes x 5 measures 15 numbers
19
Reporting Schedule

• Reported February 2007
• Entry information Age expected? Yes, No
• One time requirement
• Reported for children entering between July 1,
  2005, and June 30, 2006

• Due February 2008
• Data in reporting categories at exit for all
  children who have been in the program for at
  least 6 months
• Must be reported for the year beginning July 1,
  2006
• Repeat with next years data in 2009, etc.

20
Also

• States are required to
• Make public data reported to OSEP
• Analyze state data by program (i.e., compute a
  through e for each program)
• Make public the data by program

21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
Age in Months
25
Point of clarification

• Why are we comparing children with delays and
  disabilities to typically developing children?

26
Point of clarification

• Process is NOT about comparing groups of children
  it IS about asking how close children are to
  being able to do what is expected at their age
• Early learning guidelines
• Kindergarten and access to the general curriculum

27
Source National Early Intervention Longitudinal
Study
28

• Where are we now
• State decisions and activities

29
WHY?
Purpose
To meet provider/teacher, local and/or state need
for outcome information and to respond to
federal reporting requirements
To respond to federal reporting requirements
30
State approaches

• Most states have embraced outcomes measurement
  and are collecting outcomes data for their own
  purposes.
• Many states are building bigger systems than
  needed to produce the federal data.
• Go to www.the-eco-center.org for more information
  about what other states are doing

31
How are states collecting child outcomes/goal
data?

• Possible state approaches to collection of child
  data
• Child Outcomes Summary Form (COSF) Early
  Intervention Child Goals Summary Form in HI
• Publishers online assessment system
• Single assessment statewide
• Other approaches

32
State approaches to measurement for Part C child
outcomes

• 40 states using the ECO Child Outcomes Summary
  Form (COSF)
• 8 states using 1 assessment tool statewide
• 3 states using on-line assessment systems with
  the capacity to report OSEP data reports
• 5 states using other unique approaches

33
(No Transcript)
34
Variations across states in CGSF implementation

• Some states started early (HI) some did not
  start until mid to late 2007
• Some states completing at IFSP others at a
  separate meeting
• Some states including parents in the discussion
  some are not

35
Where states are now

• First data on 5 categories due to OSEP February 3
• Many states do not have data on many children yet
  
• Many states focusing on improving the process of
  collecting the data

36

• What do we know so far
• Positive impacts of the goals rating process

37
Positive impacts reported by states

• Increases focus on functional outcomes on IFSPs
• Easier to write functional outcomes on IFSP
• Facilitates communication with parents

38
Benefits of discussing the 3 goals

• Requires us to talk think in terms of
  functional behaviors, not test items
• Incorporates the parents as active and
  knowledgeable participants
• Looks at all settings and situations
• Bridges the gap between assessment tools and real
  life.

From presentation by Sandi Harrington, Norfolk,
VA Infant Development Program, at the OSEP EC
Meeting, December 2007
39
Benefits

• Is more meaningful to families
• Prepares the family for setting IFSP outcomes
  thinking about the skills they want their child
  to have to function in their daily family life
• Guides us towards discipline-free contextualized
  goals.

From presentation by Sandi Harrington, Norfolk,
VA Infant Development Program, at the OSEP EC
Meeting, December 2007
40
Benefits of INCLUDING families

• Determining child progress requires we use the
  familys expertise and knowledge of their child
  across setting and situations
• Our discussion becomes more inclusive with the
  family as an equal source of information for
  assessment purposes.

From presentation by Sandi Harrington, Norfolk
Infant Development Program, at the OSEP EC
Meeting, December 2007
41
Benefits of INCLUDING families

• One of the biggest shifts in practice, for many
  systems, was the move to compare their children
  in Part C to their same age peers.
• Looking to children in the frame of same age
  peers allows us to have authentic, honest
  discussions with families about their childs
  strengths and needs.

From presentation by Sandi Harrington, Norfolk
Infant Development Program, at the OSEP EC
Meeting, December 2007
42
Benefits of INCLUDING families

• We need to be comfortable with reporting
  strengths AND areas of delay, while being family
  friendly.

From presentation by Sandi Harrington, Norfolk
Infant Development Program, at the OSEP EC
Meeting, December 2007
43

• What do we know so far
• Challenges to getting good information

44
Need for good data

• Encompasses all three levels federal, state,
  local
• Depends on how well local programs are
  implementing procedures

45
What we are learning nationally

• The process of training for child outcomes data
  collection has uncovered other areas of
  significant need related to professional
  development.

46
Essential Knowledge for Completing the Child
Goals Summary Form

• Between them, team members must
• Know about the childs functioning across
  settings and situations
• Understand age-expected child development
• Understand the content of the three child
  outcomes
• Know how to use the rating scale
• Understand age expectations for child functioning
  within the childs culture

47
Important point

• It is not necessary that all team members be
  knowledgeable in all 5 areas
• Especially, no expectation that parents
  understand the rating scale or typical child
  development
• But the professionals have to!

48
Providers need to know more about

• Assessment
• How to gather assessment data to reflect
  functioning across settings and situations,
  especially how to gather child functioning
  information from families
• Understanding the results of the assessment
• Sharing assessment results sensitively and
  honestly with families

49
Providers need to know more about

• Functional outcomes
• What are they?
• How do they differ from outcomes organized around
  domains?
• What do they mean for how professionals from
  different disciplines operate as a team?
• Typical child development
• What are the functional expectations for children
  at different ages with regard to each of the 3
  goal statements?

50
Is this process too subjective to produce good
data?

• Best practices in assessment requires looking at
  multiples sources of information
• Assessment as a tool vs. assessment as a process
• Research on judgment-based assessment indicates
  it is as good or better than traditional
  assessment

51
What is informed opinion?

• Clinical judgment (informed opinion)
  knowledgeable perceptions of caregivers and
  professionals about the elusive and subtle
  capabilities of children in different settings

52
Clinical judgment provides good data when.

• Operational definition of child characteristics
  to be judged
• Structured format for quantifying characteristics
• Information from multiple setting and individuals
• Training in methods that structure and quantify
  characteristics
• Decision making based on consensus
• From Bagnato, Smith-Jones, Matesa
  McKeating-Esterle, 2006

53
(No Transcript)
54
Ratings clarification

• Highest category (Completely, 7) Child
  functions in an age appropriate manner across
  settings and situations
• Next highest (6) Child functions in an age
  appropriate manner but there is a significant
  concern about some aspect of the childs
  functioning

55
Ratings clarification

• Somewhat (5) Child shows a MIX of age
  appropriate and not age appropriate behaviors
  across settings and situations
• Between emerging and somewhat (4) Child shows
  some age appropriate behavior but rarely

56
Ratings clarification

• Emerging (3) No age appropriate behavior yet.
  Shows immediate foundational skills in some to
  all settings and situations
• Between not yet and emerging (2) No age
  appropriate behavior yet. Rarely uses immediate
  foundational skills (but does show some).

57
Ratings clarification

• Not yet (1) No age appropriate behavior yet.
  No immediate foundational skills yet.

58
Should the rating be whatever parents want?

• No, the rating is a team consensus
• Need to think about what and how parents are
  being involved in this process
• Have the assessment results been thoroughly
  explained?
• Like so much in EI, the rating requires a
  partnership

59
ECO Discussion Prompts Child has positive social
relationships (see ECO Tools)
Thinking about relating to adults, relating to
other children, and (for those older than 18
months) following rules related to groups or
interacting with others. ? How does the child
relate to his/her parent(s)? ? How does the
child relate to other relatives or extended
family and close family friends (e.g.,
grandparents, aunts, extended kin, etc.)? Do
these interactions with people differ depending
on the setting the child is in with these people?
? How does the child interact with familiar
caregivers (e.g., child care providers,
babysitters)? ? How does the child relate to
strangers? At first? After a while? In different
settings and using different approaches? ? How
does the child interact with/respond to people in
community settings (e.g., park, library, church,
grocery store, with neighbors on walks, at the
bus stop, in restaurants, at playgroups or
outings, etc.)? .
60
Obtaining good data

• Threats to good data
• Local providers do not understand the procedures
• Local providers do not follow the procedures
• And others..
• Process requires good training procedures
• Initial
• Ongoing

61
Many steps to ensuring quality data
Before Information sharing Good data collection/Training Good data system
During Ongoing supervision Feedback Refresher training
After Monitoring Validity analyses
62

• Initial Data from Hawaii

63

• These data are very, very preliminary.

64
(No Transcript)
65
EIS Average Ratings at Initial IFSP
Social-Emotional Skills Acquire and Use Knowledge and Skills Appropriate Action
EIS (N1774) 5.9 5.3 5.1
66
(No Transcript)
67
HS Average Ratings at Initial IFSP
Social-Emotional Skills Acquire and Use Appropriate Action
HS (N1311) 6.4 6.4 6.4
68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
(No Transcript)
72
(No Transcript)
73
(No Transcript)
74
(No Transcript)
75
(No Transcript)
76
(No Transcript)
77
Appropriate Action Appropriate Action
Review Rating Review Rating
Initial_3B 1 2 3 4 5 6 7 total
1 1 4 2         7
2 1 1 5 6 9 3 1 26
3   2 15 14 27 19 6 83
4   4 4 21 39 28 12 108
5   1 12 14 71 86 48 232
6   1   3 21 48 63 136
7       2 18 23 56 99
Review Total 2 13 38 60 185 207 186 691
78
Another State OSEP Categories for Goal 1 by
Exit Status
OSEP Categ ALL Exited
a 1
b 5
c 39
d 44
e 12
N 893
79
Another State OSEP Categories for Outcome 1 by
Exit Status
OSEP Categ Exited Before 3 Exited at 3 ALL Exited
a 1 1 1
b 2 6 5
c 19 45 39
d 57 39 44
e 22 9 12
N 231 662 893
80
Another State OSEP Categories for Outcome 1 by
Exit Status
OSEP Categ Completed IFSP Exited Before 3 Exited at 3 ALL Exited
a 0 1 1 1
b 1 2 6 5
c 12 19 45 39
d 65 57 39 44
e 22 22 9 12
N 156 231 662 893
81
Questions to ask

• Do the data make sense?
• Am I surprised? Do I believe the data? Believe
  some of the data? All of the data?
• If the data are reasonable (or when they become
  reasonable), what might they tell us?

82
Validity

• Validity refers to the use of the information
• Does evidence and theory support the
  interpretation of the data for the proposed use?
• Or
• Are you justified in reaching the conclusion you
  are reaching based on the data?
• Standards for Educational and Psychological
  Testing (1999) by American Educational Research
  Association, American Psychological Association,
  National Council on Measurement in Education

83
How will/might these data be used?

• Federal level
• Overall funding decisions (accountability)
• Resource allocation (e.g., what kind of TA to
  fund?)
• Decisions about effectiveness of program in
  individual states
• State level
• Program effectiveness??
• Program improvement??
• Local level
• Program improvement??

84

• What has ECO learned after 4 years?

85
ECO Message Strong commitment

• States are committed to building good systems to
  collect data on how children are progressing
• Variations in how data are being collected
• Variations in how states plan to use the
  information
• Common thread Widespread recogniton of the
  importance of the data

86
ECO Message Need to build state capacity

• Implement oversight procedures around data
  quality
• Examine data for validity
• Analyze and interpret data for program
  improvement
• Develop messages for policy-makers, public,
  media, families from the data

87
ECO Message Need to build provider capacity

• Assessment
• Functional outcomes
• Typical child development

88
ECO Message Need for better early childhood
assessment tools

• Designed around the 3 functional outcomes for all
  children
• Designed to capture child functioning in a
  variety of setting and situations
• Designed to be used in accountability and program
  evaluation
• Current tools are antiquated
• Need to incorporate latest research, recommended
  practices, psychometrics

89
ECO Message Need for more resources to ensure
quality data

• National resources to support and coordinate
  across states
• Training needs
• Analysis and use of data
• Support for states to continue to develop and
  validate their systems
• Investment in research to examine how outcomes
  data collection being carried out (impact on
  quality, local practice, etc.)
• Investment in new assessment tools

90

• Change can be difficult.

91

• For more information.

92
(No Transcript)
93

• http//hawaii.gov/health/family-child-health/eis/w
  hatcounts.html

94
(No Transcript)
95

• www.the-eco-center.org