Developmental Disabilities Program Independent Evaluation DDPIE Project

1
Developmental Disabilities Program Independent
Evaluation (DDPIE) Project

• Jennifer Johnson
• Lynn Elinson
• Cynthia Thomas
• AUCD Annual Meeting
• October 31, 2006

2
Purpose of the Independent Evaluation

• Demonstrate impact of DD Network programs on
• Individuals
• Families
• Service providers
• State systems
• Provide feedback to ADD to help improve the
  effectiveness of its programs and policies
• Promote positive achievements of DD Network
  programs
• Promote accountability to the public

3
DDPIE Project

• Independent evaluation
• 2 phases
• Phase 1 development and testing of tools
• Phase 2 full-scale evaluation
• Westat contracted by ADD to implement Phase 1

4
Evaluation Tools

• Measurement matrices
• - standards
• - indicators (structures, processes, outputs,
• outcomes)
• - performance levels
• Data collection instruments

5
Evaluation
Standards
Indicators
What do we hope to achieve?
What do we observe (measurement of indicators)?
Comparison
Are there differences/discrepancies?
What is the nature and extent of the differences?

What action needs to be taken?
6
Framework of indicators (RFP)

• To organize and guide the development of the
  performance standards and related measurement
  matrices, the following framework of indicators
  of program impact should be used
• - Structural indicators adequate and
  appropriate settings and infrastructures,
  including staffing, facilities and equipment,
  financial resources, information systems,
  governance and administrative structures, etc.
• - Process indicators activities, procedures,
  methods, and intervention supporting practices
• - Output indicators results of the DD Networks
  policies, procedures, and services
• - Outcome indicators intermediate results

7
Open Systems Model
Effectiveness
Structure (Input)
Process
Output (Product)
Outcome
Efficiency
8
Measurement Matrices

• Tools in the evaluation that will organize
  indicators, standards, and performance levels for
  each key function within each DD Network program
  and collaboration
• Developed and pilot-tested in Phase I
• Used in Phase II

9
(No Transcript)
10
Basic Evaluation Approach

• Performance-based approach interested in
  outcomes
• Development of standards and indicators
• Development of measurement matrices that contain
  standards, indicators, and performance levels
• Collection of data
• Measurement of indicators to determine level at
  which standards are being met
• Determination of overall performance at the
  national level

11
Key Assumptions

• State programs vary on their level of performance
  across the standards.
• Consistently high performance across the
  standards is related to better outcomes.
• Consistently low performance across the standards
  is related to poor outcomes.

12
Validation

• Advisory Panel
• Working Groups
• Validation Panels
• Pilot Study
• Further analysis

13
Role of Advisory Panel

• To provide balance, impartiality, and expertise.
• To provide advice on
• DDPIE process
• Standards, indicators, performance levels, and
  data collection
• Measurement matrices
• Pilot study
• Synthesis of findings and recommendations

14
Composition of Advisory Panel

• Individuals with expertise on
• DD population
• Policies and services for the DD population
• Evaluation research
• DD Network programs
• Other evaluations

15
Advisory Panel

• Self-advocates
• Family members
• Representatives from 3 programs Richard Carroll
  from Arizona UCEDD
• Child/disability advocates
• Evaluation expert
• Federal representative (for PAIMI evaluation)

16
Working Group Members Criteria for Selection

• Have broad overview of all aspects of UCEDD
  (Director, Associate Director)
• Rural/urban state
• Geographic distribution
• Placement of UCEDD in University

17
Validation Panels

• Role To endorse the contents of the measurement
  matrices.
• Composition
• - Stakeholders (consumers, advocates)
• - DD Network program staff
• - DD Council members
• - ADD staff
• - Evaluation experts

18
Pre-test and Pilot Study

• States randomly selected
• Pre-test 1 state
• Pilot Study 4 states

19
Progress to Date

• Reviewed background materials.
• Conducted preliminary/background interviews.
• Established and met with Advisory Panel twice.
• Established and met with Working Groups.
• Identified key functions of each DD Network
  program.
• Discussed structures, processes, outputs and
  outcomes of each key function.
• Conceptualized measurement matrices.
• Developed early drafts of standards and
  indicators for each key function.

20
Project Tasks to Do

• Complete draft matrices.
• Share process with state programs.
• Validate matrices (Validation Panels)
• Develop data collection instruments.
• Obtain OMB clearance (ADD).
• Conduct pilot study.
• Analyze and synthesize data.
• Write report and recommendations.

21
DDPIE Project Milestones
22
Key Functions

• A collection of activities that are intended to
  achieve particular results
• Examples
• - PA individual advocacy, outreach/public
• education
• - DD Councils systemic advocacy, development
• of community capacity
• - UCEDDs training, community
• service/technical assistance, research,
• dissemination (from DD Act)

23
UCEDD Working Group Members
Collaboration Working Group
24
Working Group Meetings

• Orientation by telephone March, 2006
• Telephone meetings (full group) in spring
• In-person meeting in spring, 2006 coinciding
  with the national meeting
• Telephone focus groups (teams) summer, 2006
• In-person meeting in fall, 2006

25
UCEDD Key Functions

• Provision of Interdisciplinary Pre-Service
  Preparation and Continuing Education of Students
  and Fellows
• Provision of Community Services
• Conduct of Basic or Applied Research
• Dissemination
• Governance and Management?
• Priority Setting?

26
Members of UCEDD Working Subgroups

• Basic or Applied Research
• David Mank
• Gloria Krahn
• Dissemination
• Carl Calkins
• Gloria Krahn

• Pre-Service Preparation
• Fred Palmer
• Lucille Zeph
• Community Services
• Tawara Goode
• Fred Orelove

27
Working Group Teams

• Described goals for each key function
• Explained the main activities
• Identified outcomes
• Discussed variability across UCEDDs
• Provided Examples
• Helped formulate SPOO tables, standards

28
Interdisciplinary Pre-Service Preparation...

• Goal
• Develop a cadre of individuals with knowledge,
  skills, attitudes and values to increase the
  capacity of states to provide services and
  supports in a culturally and linguistically
  competent manner for people with developmental
  disabilities and their families.

29
Interdisciplinary Pre-Service Preparation...(cont
d)

• Function includes
• Developing and teaching courses in the core
  curriculum
• Developing and/or teaching disability content for
  courses in other departments
• Developing and teaching classes that offer
  continuing education credits
• Outcomes Include
• Graduates from the preservice programs who
  demonstrate knowledge, skills, and attitudes and
  values consistent with . the principles of the
  DD Act.
• Students who complete courses with disability
  content and demonstrate attitudes and values .
• Trainees who complete continuing education
  classes ..

30
Following Steps

• Development of written descriptions of each key
  function
• Development of draft indicators
• Full working group reviews descriptions,
  standard, indicators