Child and Family Outcomes Measurement Project

1
Child and Family Outcomes Measurement Project

• General Supervision Enhancement Grant

2
Kick-Off Agenda

• Welcome and Introductions
• Robin Arnold-Williams, Secretary of DSHS
• Measuring Outcomes
• Dr. Kathleen Hebbeler, SRI International,
  Director of ECO Center
• Westat
• Joy Markowitz, Westat, Project Director
• Infant Toddler Early Intervention Program
  (ITEIP)
• Sandy Loerch Morris, ITEIP Director
• Questions and Comments
• Jean Dauphinee, Westat, Project Coordinator
• Closing Remarks
• Sandy Loerch Morris, ITEIP Director

3
Participants

• On the conference call today we have 94
  conference sites representing
• Parents and Parent Support Organizations
• State Interagency Coordinating Council
• Office of the Governor
• Early Learning Council
• Washington State Agencies
• Tribal Governments/Programs
• ITEIP Local Lead Agencies
• Community and County Agencies, Organizations, and
  Providers
• School and Education Service Districts, and
  Universities
• Washington Migrant Council
• Health and Insurance Plans

4
Measuring Outcomes for Children with Disabilities
and Their Families
Kathy Hebbeler Early Childhood Outcomes
Center SRI International
Prepared for the WA State Child and Family
Outcomes Measurement Project ITEIP/Westat
Kick-Off August 21, 2006
5
Purpose of this presentation

• Explain the need for outcome data at the federal
  level and present the federal reporting
  requirement
• Describe decisions related to building a
  state-level child outcomes measurement system
  including some information about what other
  states are doing

6

• Why is the Federal Government Requiring Data on
  Child Outcomes?

7
The Need for Outcome Data at the Federal Level

• Age of accountability
• Accountability increasingly means looking at
  results not just process
• Office of Special Education Programs (OSEP) under
  increasing pressure to produce outcomes data for
  children and families participating in early
  intervention programs

8
Federal push for outcome data Intro to Acronyms

• GPRA Government Performance and Results Act
• PART Program Assessment Rating Tool
• OMB Office of Management and Budget
• IDEA Individuals with Disabilities Education
  Act (Now IDEIA-Individuals with Disabilities
  Education Improvement Act)

9
GPRA (Government Performance and Results Act of
1993)

• IDEA goals and indicators established
• Indicators and data collection further along for
  school age population than for EC
• For early intervention, data have been collected
  on
• Number of children served
• Settings

10
PART (Program Assessment Rating Tool) (2002)

• Tool used to review federal programs
• Four critical Assessment Areas including
• results and accountability
• Programs given ratings from ineffective to
  effective
• Purpose to enhance budget analysis
• Findings for Part C Results not demonstrated

11
Need for Data at the Federal Level

• The push for child outcome data is not new only
  the reporting requirement is
• OSEP is under pressure from Office of Management
  and Budget (OMB)
• There could be serious fiscal implications for
  Part C if data on child outcomes are not
  forthcoming
• Read more at http//www.whitehouse.gov/omb/expectm
  ore/summary.10000190.2005.html

12
OSEP Activities

• Required states to submit outcomes data in their
  Annual Performance Report (APR) in 2004
• Funded the Early Childhood Outcomes Center to
  make recommendations and to assist states in 2004
• Awarded competitive grants to states to work on
  development of outcomes measurement system

13
ECOs Work in 2004

• Convened stakeholders to identify child and
  family outcome areas and develop outcomes
  statements
• Received input from state Part C Coordinators,
  researchers, families, administrators, and the
  general public

14
Result of Process ECO made recommendations to
OSEP

• Child and family outcomes (Feb 2005, revised
  April 2005)
• Type of information that should be collected
  about these outcomes (May 2005)
• Read more at
• www.the-eco-center-org

15
OSEP Reporting Requirements Child Outcomes

• Positive social emotional skills (including
  positive social relationships)
• Acquisition and use of knowledge and skills
  (including early language/communication and
  early literacy)
• Use of appropriate behaviors to meet their needs

16
OSEP Reporting Categories for Child Outcomes

• Percentage of children who
• Did not improve functioning
• Improved functioning, but not sufficient to move
  nearer to functioning comparable to same-aged
  peers
• Improved functioning to a level nearer to
  same-aged peers but did not reach it
• Improved functioning to reach a level comparable
  to same-aged peers
• Maintained functioning at a level comparable to
  same-aged peers

3 outcomes x 5 measures 15 numbers
17
Schedule for OSEP Reporting for Child Outcomes

• Data at exit (categories a to e) for all children
  who have been in program for 6 months must be
  reported for the year beginning July 1, 2006 due
  February 2008, 2009, etc.
• One time data collection Entry information (age
  expected? yes, no) must be reported for
  children who entered between July 1, 2005 and
  June 30, 2006 Baseline data are due February
  2007

18
Family Outcomes

• ECO also was charged with making recommendations
  on family outcomes
• Used the same stakeholder process to identify 5
  family outcomes

19
What is a Family Outcome?

• A benefit experienced by families as a result of
  services and supports received
• An outcome is not the receipt of services or
  satisfaction with services
• An outcome is what happens as a result of
  services provided to families

20
ECO Family Outcomes
ECO

• Understand their childs strengths, abilities,
  and special needs
• Know their rights and advocate effectively for
  their children
• Help their children develop and learn
• Have support systems
• Access desired services, programs, activities in
  their community

21
ECO Family Outcomes and OSEP APR Requirements for
Part C
ECO
OSEP

• Understand their childs strengths, abilities,
  and special needs
• Know their rights and advocate effectively for
  their children
• Help their children develop and learn
• Have support systems
• Access desired services, programs, activities in
  their community

• Percent of families participating in Part C who
  report that EI services have helped the family
• Know their rights
• Effectively communicate their childrens needs
• Help their children develop and learn

22
Family Issues

• Information a state might want to know about
  families
• Family outcomes
• Family involvement
• Family satisfaction
• OSEP items
• All can be measured with a survey

23
Measuring Family Issues

• Methods states may use to collect family
  information
• Family outcomes (ECO Survey)
• Family involvement (NCSEAM survey)
• Family satisfaction (Many states have their own
  surveys)
• OSEP items (ECO survey NCSEAM survey add items
  to state surveys)

www.the-eco-center.org (ECO tools) National
Center for Special Education Monitoring http//www
.monitoringcenter.lsuhsc.edu/parent_Family_Involve
ment.htm
24

• Decisions and
• Possible Approaches
• for Collecting
• Child Outcomes Data

25
Designing an accountability system

• Developing an outcomes system requires many
  interrelated decisions
• Choosing one course will close the gate to other
  courses
• There is no perfect or easy course
• Not designing a system is not an option

26
Multiple sources of information?
Other early childhood initiatives?
Policymakers want data?
Burden on locals?
Resources?
Authentic assessment?
Context Values Drive Decisions
Local control?
Interagency issues?
Standardized assessment?
Stakeholder input?
Early learning guidelines?
Minimize change?
27
Accountability can go bad

• Accountability done poorly can
• Lead to bad practices
• Do harm to children
• Etc., etc.
• Be wary, be thoughtful, be involved
• Not designing a system is not an option

28
Steps in Development of an Outcomes System

• Identify overall purpose for the process and
  system (including priorities)
• Identify outcome areas for children and families
• Develop outcome statements
• Formulate the evidence statements
• Identify measurement approaches (e.g., select
  instruments)

29
Steps (Note the Washington State system is in
place at this point)

• Collect data (submit to state)
• Analyze local and state data
• Interpret/report/share the findings
• Make changes to policies and programs based on
  what was learned
• Refine the data collection approach based on what
  was learned

30
1. Determine the Purpose of the System

• Accountability system can serve multiple purposes
• Different uses for an outcomes information at
  different levels
• Federal and state government (accountability,
  planning for support programs)
• Therapists, teachers, and providers (altering an
  intervention, identifying children who need extra
  help)

31
Why might Washington want good outcome data?

• To assure comprehensive, quality services for
  Washington children and families
• To improve existing programs
• To justify funding for programs for young
  children and their families

32
Why might Washington want good outcome data?

• Because the federal government requires it and
  gives the state money to enhance systems
  including collecting the data
• Be clear on why the state is doing this
• Purpose drives the other decisions

33
Purpose Program Improvement

• Our discussion today only focuses on outcomes
  measurement BUT
• Outcomes data alone are of very limited
  usefulness for program improvement
• Need to be part of an evaluation system that also
  looks at implementation issues (e.g., program
  quality, services received, procedural issues,
  satisfaction)

34
Simplistic Logic Model
Good outcomes for children and families
High quality services for children 0-5 and their
families
Appropriate State Policies and programs
Appropriate Local Policies and Programs
Appropriate Federal policies and programs
Supports
35
Purpose determines which children are to be
included in the outcomes system

• State approaches
• Part C system Part B system
• Some blending of C and B
• Same assessment
• Data sharing
• Data linking
• Early Childhood System that
• Includes C and B
• Based on analyses of the State Performance
  Plans, December 2005

36
Collaboration between C and B

• 25 states reported in Part C SPP collaborating
  with Part B on outcomes
• 21 states reported in Part B SPP collaborating
  with Part C on outcomes

37

• Goal is to design an accountability process and
  system that will produce the right information
  on outcomes for each stakeholders needs
• Right valid, reliable, meaningful, useful,
  appropriate level of detail

38
Questions for Washington

• Why does Washington want outcomes data?
• Who are the audiences?
• What will they want to do with the information?

39
Questions for Washington

• Is the state comfortable with the 3 OSEP outcomes
  areas?
• Does the state want more outcomes? Sub-outcomes?
• Does the state need to relate these outcomes to
  other programs outcomes (e.g., early learning
  activities and guidelines)?

40
2. Identify outcomes areas

• What are the important outcomes areas?
• Child
• Family
• Everyone wants to know How are children doing?
  but need to decide with regard to what?

41
3. Develop outcomes statements

• Based on stakeholder input, ECO developed a set
  of child and family outcome statement
• These were revised numerous times as stakeholders
  groups reviewed them and provided more input

42
ECO consensus building process identified an
overarching goal for children

• The ultimate goal of early intervention is for
  young children to be active and successful
  participants now and in the future in a variety
  of settings in their homes, in their child
  care, preschool or school programs, and in the
  community
• Read more at http//www.fpg.unc.edu/eco/pdfs/eco_
  outcomes_4-13-05.pdf

43
ECO Child Outcomes

• Children have positive social relationships
• Children acquire and use knowledge and skills
• Children take appropriate action to meet their
  needs

44
Outcomes are Functional

• Functional refers to things that are meaningful
  to the child in the context of everyday living.
• Refer to an integrated series of behavior or
  skills they allow the child to achieve the
  outcomes.
• They are not
• a single behavior, nor are they
• The sum of a series of discrete behaviors

45
Outcomes are Functional (continued)

• Integrate behaviors across domains not trying
  to separate child development into discrete areas
  (communication, gross motor, etc.)
• Emphasize how the child is able to carry out
  meaningful behaviors in a meaningful context

46
4. Formulate the evidence statements

• What constitutes evidence of good outcomes?
• Definition Evidence Statement a statement
  that incorporates a statistic and provides
  evidence as to whether or not an outcome has been
  achieved
• OSEP reporting categories are evidence statements
• Note Washington will need to be able to produce
  this kind of evidence, but also can design a
  system that produces outcome data organized in
  other ways as well.

47
Evidence statements

• What evidence will be needed is closely linked to
  the purpose of the process and system.
• Demonstrating effectiveness?
• Improving programs?
• Identifying strengths and weaknesses in the
  process/system?
• What local programs need to be strengthened?
• What sub-groups are being served less well?

48
5. Identify measurement approaches

• Need assessment instruments to collect data on
  outcomes
• State approaches
• Use one instrument across state
• Develop an approved list of instruments
• Let programs use whatever they are using

49
Measurement approaches

• Having a good process/tool for getting
  information about children is critical
• Foundation on which the accountability system
  rises or falls

50
Pitfall Alert!

• What we want for children and families (desired
  outcomes) reflects our values
• What we want for young children may not be the
  same as what can be easily measured
• When what is valued and what can be measured
  easily or well do not line up, EITHER
• Outcomes will be determined by what can be
  measured easily or well, OR
• Some of the outcomes will be more easily measured
  or more adequately measured than others

51
How does the state get data on outcomes?

• Who provides?
• What assessments are used?
• How often is data collected?
• When is data collected? (When is it reported)
• Summarizing across multiple sources?
• Summarizing across different assessments?

52
Decision How best to capture a childs
functioning?

• Data needs to reflect a childs functioning in
  each broad outcome area
• Functional outcomes summarize each childs
  current functioning across settings and
  situations
• Best practice for assessing young children
  recommends the use of multiple measures
• Will single sources ( assessment tool) produce
  valid data on functional outcomes?

53
Decision What will be the role of families in
the outcomes system?

• Impossible to understand how a child is
  functioning across a variety of everyday settings
  and situations without family input
• Options
• Incorporate into the assessment tool
• Collect through a parent-completed tool
• Incorporate into a summary rating
• Some combination
• Decision How will information from families be
  included in the outcomes data?

54
Capturing Child FunctioningApproaches to
identifying assessment tools

• 3 approaches being used by states
• One assessment tool selected by state
• List of assessment tools developed by state
  programs pick
• Programs use the evaluation and assessment tools
  they have been using

55
Commonly Reported Assessment Instruments (Part C)

• 20 different assessment tools identified 3
  states using state developed tool
• Of 28 states who listed specific assessment
  instruments
• HELP - 15 states
• BDI/BDI2 - 13 states
• AEPS - 11 states
• Creative Curriculum - 6 states
• ELAP- 6 states
• Not reported 30 states
• Not yet determined - 23 states

56
Capturing Child Functioning Combining
Information from Multiple Sources (Part C)

• Using or considering using the ECO Child Outcomes
  Summary Form -37 states
• Developing own summary tools 3 states
• Dont need a summary tool 6 states
• Approach unknown 10 states

Data as of April 2006
57
Child Outcomes Summary Form
Go to www.the-eco-center.org and click on ECO
Tools to see the entire form
58
Features of the Summary Form

• Not an assessment tool
• 7- point scale that uses information from
  assessments and observations to get a global
  sense of how the child is doing on each outcome
• Rating is based on childs functioning compared
  to other children the same age distance from
  typical
• Based on childs functioning
• what child does across settings and situations,
• not what a child can do under ideal circumstances.

59
Role of the Early Learning Guidelines (ELGs)?

• May change or add to the outcomes questions
• Are children meeting the ELGs?
• May mean mapping the ELGs to the 3 OSEP outcomes
• Aligning with ELGs
• Part C 8 states Part B 18 states

60

• and then the system has to be built

61
Build the infrastructure, etc.

• Much is involved in this step
• Right now it is extremely important to build the
  capacity of teachers, caregivers, providers, etc.
  to use assessment tools and possibly a summary
  device
• Other pieces will be coming including building a
  mechanism for entering and ultimately, reporting
  the data

62
What does this mean for Washington?

• Data on how young children with disabilities will
  be collected throughout the country, but
  Washington and other states are still facing many
  decisions
• What assessment tools?
• How to summarize across tools?
• How often?
• Where does the data get recorded?
• These data will be analyzed and reported
• Who enters the data?
• Who analyzes the data?
• How does it get reported to the state?
• How often?

63
Vision for outcomes-based accountability

• To build a process and system where all the
  pieces will fit together smoothly
• Meaningful outcomes data collected regularly that
  can used to meet the state-identified purposes
  and provide data for the federal government

64
Conclusions

• Many (tough) decisions to be made
• Short federally driven timelines
• There is no perfect course
• Keep your eye on the prize High quality programs
  and good outcomes for children and families

65
Staying in touch with ECO

• Web site the-eco-center.org
• Follow developments related to work of the Center
• Obtain ECO resource documents
• Obtain other related resources
• Email staff_at_the-eco-center.org
• ECO sends to existing list serves
• Can join the ECO mailing list

66
What is Westat?

• Joy Markowitz, Project Director
• Employee-owned company
• Research, statistical design, data collection,
  program evaluation, technical assistance
• Staff of 1,700 research, technical, and
  administrative
• Based in Rockville, Maryland
• Federal, state and local government agency
  clients and others since 1961

67
Westat Expertise Experience

• Many projects and studies with the U.S.
  Department of Educations (ED) Office of Special
  Education Programs (OSEP) and other ED offices.
• More than 15 years working with OSEP, Part C lead
  agencies, and state education agencies (Part B)
  on issues related to data collection, analysis
  and report preparation.
• Many projects and grants with state education
  agencies and Part C lead agencies.
• Involvement with the Early Childhood Outcomes
  (ECO) Center from its inception.

68
Westat ITEIP GSEG Partnership

• Long-term professional relationships between
  Westat and ITEIP staff
• ITEIP interest and need for GSEG funds
• Westat interest in staying involved in early
  childhood outcomes arena
• Westat interest in ITEIP Data Management System

69
General Supervision Enhancement Grant (GSEG)

• OSEP grant competition to support the work of
  states
• 2004 2005 early childhood and family outcomes
  grant priority areas
• 12-month grants, with option for no-cost time
  extension
• State applicants or other applicants on behalf of
  states

70
GSEG Grant Writing

• Collaborative endeavor between Westat and ITEIP
• Grant writing coordination - August September
  2005
• Grant submitted by Westat on behalf of DSHS/ITEIP
  on October 3, 2005
• OSEP announced grants in March 2006, much later
  than expected.

71
Washington GSEG ? Child and Family Outcomes
Measurement Project

• Marsha Brauen, Corporate Officer-in-Charge
  based in Maryland
• Joy Markowitz, Project Director based in
  Maryland
• Jean Dauphinee, Project Coordinator based in
  Washington
• Sandy Loerch Morris, ITEIP Program Director
  Washington State Lead Agency
• Additional ITEIP staff provide support to the
  project
• Other subcontractors For example SRI
  International/ECO Center Logicalis ITEIP Data
  Management Contractor

72
Objectives of Project

• To bring stakeholders together to examine
  research, national policy and reporting
  requirements related to Part C child and family
  outcomes in order to make recommendations for the
  state
• To explore and enhance the states database
  capabilities and revisions needed for collection
  of Part C child and family outcome data
• To contribute to national policy and practice

73
Phases of the Project

• Phase 1 Obtain stakeholder input and generate
  draft recommendations September-November 2006
• Phase 2 Send recommendations to SICC and
  broader group of stakeholders December 2006
• Phase 3 Pilot collection of child and family
  outcome data January-April 2007
• Phase 4 Design and begin update of ITEIP Data
  Management System May-August 2007

Share Washingtons experience with others around
the country
74
Infant Toddler Early Intervention Program (ITEIP)

• Sandy Loerch Morris, Program Director
• ITEIPs role in the Child and Family Outcomes
  Measurement Project
• Provide policy guidance, general supervision of
  state progress, or assurance of implementation of
  measures and data collection at state and local
  level
• Ensure stakeholder advisory committee is selected
• Generate and provide required reports to state
  and federal authorities

75
Stakeholder Meeting Information

• Three meetings scheduled for
• Monday, September 18, 9-4 (lunch provided)
• Thursday, October 19, 9-4 (lunch provided)
• Monday, November 20, 9-4 (lunch provided)
• Location
• SeaTac Area

76
Stakeholder Meeting Information Continued

• Stakeholders will be selected to represent a
  diverse workgroup
• We encourage parents and providers from all
  geographic areas to apply
• Every effort will be made to represent diverse
  voice from throughout the state
• Travel reimbursement is available for those not
  covered by agency travel

77
Stakeholder Meeting Information Continued

• We are looking for commitment from applicants to
  attend all three meetings and a mutually
  scheduled meeting in the spring
• Due to time constraints alternates will not be
  possible

78
Stakeholder Group Application

• Individuals interested in participation on the
  Stakeholders Work Group are invited to apply
• You may access the application
• Via the ITEIP Website
• http//www1.dshs.wa.gov/iteip/whats_new.html
• Contact Linda Jennings at
  jennill_at_dshs.wa.gov or [phone] to obtain
  an application via email, mail or fax
• See the attachment on email to call
  participants
• Applications should be sent to Linda Jennings at
  ITEIP, on or before Tuesday, September 5, 2006
• Please share this information with parents and
  others who may be interested assisting us

79
Participant Questions and Comments

• Jean Dauphinee, Project Coordinator

80
Closing Remarks

• Sandy Loerch Morris, Program Director
• We look forward to many applicants and our future
  work together
• This is a work in progress
• It will take all of us to get there, together
• Thanks to each of you for joining us and
  committing this time out of your busy schedules