Outcomes and Evidence Statements

1
Outcomes and Evidence Statements
Kathy Hebbeler SRI International
Prepared for the Michigan Results Group Lansing,
MI July 2005
2
Objectives for this presentation

• Review the ECO outcomes
• Introduce some concepts related to evidence
  statements to assist Michigan in formulating its
  evidence statements
• Share ECOs recommendations for evidence
  statements
• Describe what OSEP will be requiring
• Introduce the measurement approach ECO is
  developing

3
Early Childhood Outcomes (ECO) Center Mission

• Promote the development and implementation of
  child and family outcome measures for infants,
  toddlers and preschoolers with disabilities that
  can be used in national and state accountability
  systems

4
ECO Center

• 5-year project funded by OSEP in October 2003
• Collaboration among
• SRI International
• Frank Porter Graham Child Development Institute
  (U of North Carolina, Chapel Hill)
• Juniper Gardens Childrens Project (U of Kansas)
• National Association of State Directors of
  Special Education
• University of Connecticut

5
Public Policy Context

• Age of accountability
• Accountability increasingly means looking at
  results not just process
• Applies across all private and public human
  service and education programs

6
Presidents Commission on Excellence in Special
Education (2002)

• Major Recommendation 1 Focus on results not
  process
• IDEA will only fulfill its intended purpose if
  it raises expectations for students and becomes
  result-orientednot driven by process,
  litigation, regulation and confrontation. In
  short, the system must be judged by the
  opportunities it provides and the outcomes
  achieved for each child.

7
Federal push for outcome data Intro to Acronyms

• GPRA Government Performance and Results Act
• PART Program Assessment Rating Tool
• OMB Office of Management and Budget
• IDEA Individuals with Disabilities Education
  Act

8
GPRA (Government Performance and Results Act)

• IDEA goals and indicators established
• Indicators and data collection further along for
  school age population than for EC
• For early childhood, data have been collected on
  
• Number of children served (Part C)
• Settings (both Part C and 619)
• Note Part C Programs for 0-3 year olds
  619 Programs for 3-5

9
PART (Program Assessment Rating Tool)

• Tool used to review federal programs
• Four critical Assessment Areas including
• results and accountability
• Programs given ratings from ineffective to
  effective
• Purpose to enhance budget analysis

10
PART Evaluation Findings and Recommendations

• Part C and 619 Findings No long-term child
  outcome goals or data
• The PART required the Department to
• 1. Establish long-term outcome-oriented
  objectives
• 2. Develop a strategy to collect performance
  data
• Office of Special Education Programs (OSEP) asked
  states to provide outcome data in their Annual
  Performance Reports (APRs)

11
Reality Check (aka Challenges)

• Michigan will be required to submit data to OSEP
• OSEP has recently released the indicators on
  which it will be requiring data and the timeline
  is impossible
• ECO is simultaneously working on many of the same
  issues the states are. We have a measurement
  approach in development but it is far from
  complete.
• We have no choice but to live with the
  uncertainties and move forward as best we can

12
Steps in Development of an Outcomes System

• Identify overall purpose for the system
  (including priorities)
• Identify outcome areas for children and families
• Develop outcome statements
• Formulate the evidence statements
• Identify measurement approaches (e.g., select
  instruments)
• Steps 6 to 11 after that

13
Need to acknowledge the range of emotional
responses to outcomes systems

• Hate it, have to do it so I will
• Hate it, have to do it but I wont do a very good
  job of it because it is a waste of my time
• Neutral, just another fad that will pass
• Not sure, might be good but not likely and
  probably will be useless to harmful
• Worried that this will take time away from
  children and families
• Hopeful that this will live up to the potential
• Excited about the prospect of having data on
  child outcomes

14
Identify outcomes areas

• What are the important outcomes areas?
• Child
• Family
• Everyone wants to know How are children doing?
  but need to decide with regard to what?

15
Definitions

• Outcomea statement of a measurable condition(s)
  desired for the population of children or their
  families
• Children take appropriate action to meet their
  needs
• Families know how to advocate for the services
  they need

16
Identify outcome areas

• Outcome statements are the systems overall
  vision for children and for families
• Not the same as outcomes on an IFSP or IEP which
  are very specific outcomes for an individual
  child or family
• Outcomes in an accountability system are global
  statements of what we are trying to do for
  children and families

17
Identify outcome areas

• How should we think about child outcomes?
• for more than three decades, researchers and
  service providers have struggled with both the
  identification of significant child outcomes and
  their valid and reliable measurement.
• National Research Council and Institute of
  Medicine, 2000

18
Identify outcome areas

• How should we think about child outcomes?
• Domains (language, cognitive, adaptive, etc.)
• Functional skills (feeding, dressing, mobility,
  etc.)
• Underlying functional capacities
  (Self-regulation, engagement, knowledge
  acquisition skills, etc.)
• Pre-academic skills (Early reading, math,
  science, etc.)

19
Pitfall Alert!

• What we want for children and families (desired
  outcomes) reflects our values
• What we want for young children may not be the
  same as what can be easily measured
• When what is valued and what can be measured
  easily or well do not line up, EITHER
• Outcomes will be determined by what can be
  measured easily or well, OR
• Some of the outcomes will be more easily measured
  or more adequately measured than others

20
ECO Family Outcomes

• Families know their rights and advocate
  effectively for their children.
• 2. Families understand their children's abilities
  and special needs
• 3. Families help their children develop and
  learn.
• 4. Families have social supports.
• 5. Families are able to gain access to desired
  services, programs, and activities in their
  community.

21
Need for an overarching goal for children

• The ultimate goal is for young children to be
  active and successful participants now and in the
  future in a variety of settings in their homes,
  in their child care, preschool or school
  programs, and in the community

22
ECO Child Outcomes

1. Children have positive social relationships
2. Children acquire and use knowledge and skills
3. Children take appropriate action to meet their
   needs

23
Issues from stakeholder discussions

• Functional outcomes
• Best practice kind of outcomes recommended for
  IFSPs and IEPs
• Consistent with transdisciplinary service
  delivery
• Not reflected well in a pure domains framework
• Not captured well in current assessment tools

24
Elaboration of the ECO Outcomes
To be active and successful participants now and
in the future in a variety of settings
To be active and successful participants now and
in the future in a variety of settings
25
Alternative Ways of Thinking about Child Outcomes
Example of how movement can be sub-divided
26
What is happening with these outcome areas and
statements?

• ECOs work is independent of OSEP
• OSEP has been involved in ECOs work from the
  beginning
• ECO submitted these outcomes to OSEP as its
  recommendations
• ECO submitted recommended indicators based on
  these outcomes to OSEP for the APR/SPP
  instructions that were available for public
  comment

27

• Evidence Statements
• (or what do you want to know about those outcomes)

28
The Importance of Evidence Statements

• What constitutes evidence of good outcomes?
• If you were asked to testify before your state
  legislature to demonstrate the effectiveness of
  EI, what kind of evidence do you want to have?
  (Purpose accountability)
• What kind of evidence about outcomes is needed by
  states and local programs to improve services for
  children and families? (Purpose program
  improvement)

29
Definition of Evidence Statement

• Evidence Statement - a statement that
  incorporates a statistic and provides evidence as
  to whether not an outcome has been achieved
• of children who have positive social
  relationships
• of children who show progress toward acquiring
  knowledge and skills
• Note States will be asked to submit a specified
  kind of evidence to OSEP. Each state will need
  to be able to produce this kind of evidence but
  might want other kinds of evidence as well.

30
Features of a Good Evidence Statement

• Credible Based on valid data
• Meaningful The evidence can be interpreted
• Powerful The evidence is convincing to those
  who will be receiving it (purpose
  accountability)
• Useful The evidence is helpful to those who
  will use it (purpose program improvement)

31
Measurement Evidence
Statements

• Evidence statements are the ultimate product in
  an outcomes-based accountability system
• The nature of the evidence statement that can be
  produced will depend on what is measured and how
  often

32
Possible Categories of Evidence Statements for
Outcome Data

• Status (achievement at one point in time)
• Example of children who achieved this..
• Progress (change relative to earlier status)
• Example of children who made progress in..

33
Two Ways to Think about Progress with Young
Children

• Continuous Progress acquisition of new skills
  and behaviors (almost all children show this)
• Or
• Age-anchored Progress change in developmental
  trajectory (i.e., narrowing the gap between a
  child with a delay and typical development less
  delayed after intervention)

34
Examples of Two Kinds of Progress

• Continuous increase in vocabulary
• Age-anchored change from being 1.5 standard
  deviations below norm to .5 standard deviation

35
(No Transcript)
36

• Advantages and Disadvantages of Different
  Evidence Statements

37
Status (where children are)

• Reports the percentage of children who achieved
  or could do X
• Would likely involve some kind of comparison to
  expectations for children of a given age
• Examples
• 53 of early intervention graduates were rated as
  typical and proficient with regard to having
  positive social relationships

38
Status Evidence Statement

• Pro
• Requires one measurement point
• Could be strong evidence
• Nearly all K-12 evidence statements are of this
  form (grad. rates, proficient reading)
• Con
• What is the benchmark that makes sense for all
  children with disabilities?
• How does one interpret the data (is this finding
  good news or bad news?)

39
Continuous Progress (change compared to earlier
status)

• Reports of children who improved
• Progress on a curriculum-based assessment
• Examples
• of preschool children who made progress toward
  having positive social relationships
• of preschool children who showed an increase of
  3 of more objectives related to positive social
  relationships scale after one year

40
Continuous Progress

• Pro
• Straightforward
• Could quantify extent of child growth
• Incorporates all levels of gain, even the very
  small gains made by children with severe
  impairments
• Con
• Requires at least 2 points of data
• Nearly all children will improve over time
• Weak evidence. What constitutes good news?

41
Age-Anchored Progress (closing the gap toward
typical development)

• Captures progress relative to same age peers
• Examples
• 23 of children moved from low performing to age
  appropriate in positive social relationships by
  kindergarten entry
• With regard to acquiring and using knowledge and
  skills, 75 of children made progress sufficient
  to maintain their functioning at an age
  appropriate level

42
Age-Anchored Progress (closing the gap toward
typical development)

• Pro
• Consistent with the intent of intervening
• Could produce strong evidence
• Con
• Requires at least 2 time points of data
• Closing the gap is not an expectation for all
  children with disabilities.
• How is the evidence to be interpreted?
• If some children dont close the gap, is this
  interpreted as lack of success for the program?

43
Deciding on desired evidence

• Not an either-or
• Different levels will want different kinds of
  evidence
• Many different kinds of evidence can be produced
  from the same set of data
• If the desired evidence statements are identified
  as part of the planning

44

• What OSEP will be requiring from states for Part
  C and Preschool Special Education

45
Reporting Requirement Part C

• Percent of infants and toddlers with IFSPs who
  demonstrate improved
• Positive social-emotional skills (including
  social relationships)
• Acquisition and use of knowledge and skills
  (including early language/communication)
• Use of appropriate behaviors to meet their needs

46
The details

• of infants and toddlers who reach or maintain
  functioning at a level comparable to same-aged
  peers
• of infants and toddlers who improve functioning
• of infants and toddlers who did not improve
  functioning.
• If children meet the criteria for a, report them
  in a. Do not include children reported in a in b
  or c.

47
(No Transcript)
48
3 OSEP Evidence Statements

• OSEP a
• Maintained functioning comparable to age peers
• Achieved functioning comparable to age peers
• OSEP b
• Moved nearer functioning comparable to age peers
• Made progress no change in trajectory
• OSEP c
• Did not make progress

49
Two kinds of outcomes questions for each child
with a disability

1. How is this child doing relative to the
   individualized outcomes that were established
   through the IFSP or IEP process?
2. How is this child doing relative to same-aged
   peers?

50
Part C Indicators Family Outcomes

• Percent of families participating in Part C who
  report that early intervention services have
  helped the family
• Know their rights
• Effectively communicate their childrens needs
  and
• Help their children develop and learn.

51
Measurement guidelines

• State selected data source, sampling permitted
• of families who report that early intervention
  services have helped their family in each of the
  three areas

52
SPP/APRMeasurement implications

• Requires data collection from family
  members/caregivers
• Data at one time period
• Asks families to recognize that early
  intervention services contributed to the change
  or status listed.
• Does not require family data to be linked with
  child data

53
Key Questions

• How can Michigan provide the indicators being
  requested by OSEP?
• What kinds of evidence statements does Michigan
  want to be able to produce for its own use?
• What do the answers mean for measurement?

54

• Producing Data on the Outcomes

55
Rolling up assessment data to the outcomes

• Given
• Different children will have different kinds of
  assessment data.
• Also
• The same child may have several different kinds
  of assessment data.
• Challenge How to roll up multiple pieces of
  information (e.g., data from different assessment
  tools) to a single score?

56
Example of how movement can be sub-divided
57
Different assessments on different children
Assessment 1
Roll up Process
Assessment 2
Single Score
Assessment 3
Assessment 4
58
Different pieces of information on the same child
Assessment 1
Roll up Process
Assessment 2
Single Score
Parent Report
Informed Clinical Opinion
59
Rolling up from the different assessments

• ECO is working on a scale that will provide a way
  to give a child a numerical rating on each
  outcome
• It is NOT an assessment tool.
• It assumes teachers and providers are already
  using an assessment tool to collect information
  about how children are doing.
• The scale is a way to take what those closest to
  the child know and put a number on it.

60
What does the outcomes scale look like?

• 5 pt and a 7 pt version
• The precision question.
• The high point (5 or 7) indicates outcome
  achieved at an age-expected level
• The lowest point (1) indicates the farthest
  distance from age-expectations

61
What does the outcomes scale look like?

• Challenge is in how to anchor and provide the
  descriptors for the middle points on the scale.
• Progress is moving up the scale between time 1
  and time 2.

62
Measuring the Family Outcomes

• ECO working on a tool
• ECO working on a document with guidance for
  states that reviews key decisions and includes
  several options

63
Staying in touch

• Web site the-eco-center.org
• Follow developments related to work of the Center
• Obtain ECO resource documents
• Obtain other related resources
• Email staff_at_the-eco-center.org
• ECO sends to existing listservs
• Can join the ECO mailing list

64
Vision

• To build a system where all the pieces will fit
  together smoothly
• Meaningful outcomes data collected regularly that
  can used in programs by programs AND reported to
  the state (and feds)

65
(No Transcript)
66
Local data
State needs
67
(No Transcript)
68

• When Michigan can realize this vision, children
  and families will reap the benefits.