READY A Youth Development Outcomes Measure

1
Rochester Evaluation of Asset Development for
Youth

• READY - A Youth Development Outcomes Measure

2
Project Team

• United Way of Greater Rochester
• Kathy Lewis
• Elizabeth Ramsay
• Rochester-Monroe County Youth Bureau
• Chris Dandino
• Univ. of Rochester Medical Center, Div. of
  Adolescent Medicine
• Jonathan D. Klein, MD, MPH
• Melissa Matos Auerbach, MA
• Shannon M. Smith
• Sheryl Ryan, MD
• Cheryl Kodjo, MD, MPH
• Premini Sabaratnam, MPH
• Representatives from youth serving agencies in
  Rochester

3
Participating Agencies included

• Baden Street Settlement
• Big Brothers, Big Sisters of Greater Rochester
• Boy Scouts of America, Otetiana Council
• Boys Girls Club of Rocehster
• Center for Youth Services
• Charles Settlement
• Community Place of Greater Rochester
• Girl Scouts of Genesee Valley
• Hochstein Music School
• Metro Council for Teen Potential
• Southwest Area Neighborhood Association (SWAN)
• Urban League of Rochester
• YMCA of Greater Rochester

4
Youth Development

• Philosophy or approach - a set of principles
  emphasizing active support of the growing
  capacity of young people by individuals,
  organizations, and institutions
• Characterized by a positive, strength building
  orientation
• Occurs at home, in school, among peer groups, and
  in community-based programs
• Has gained importance nationally, in states, and
  in local communities

5
Community Outcomes

• Local funders are increasingly concerned with
  demonstrating effective progress toward outcomes
• Existing measures of youth development are
  lengthy and complex

6
Youth Development Outcomes Measurement Project

• GOAL
• To develop an evaluation tool for YD programs
  that met the following criteria
• Easy to Use
• Easy to Administer
• Applicable to a Variety of Youth Development
  Programs
• Useful for Assessment of Impact of Program on
  Youth Development of Participants

7
Rochester, New York

• City in Monroe County, Western New York
• Population 219,773
• 52 non-white
• 37 ages 19 or less
• Person under age 18 in 34 of households
• 32 of families with children under 18 below
  poverty level

Source US Census, 2000
8
Youth-Serving Agencies

• Serve the Youth of Rochester through
• case management
• counseling
• homework assistance
• sports programs
• life skills building
• leadership programs
• music lessons
• provision of safe, open recreational spaces

9
Three Phase Project

• Phase I Instrument development via a consensus
  process
• Phase II Piloting to test validity and
  reliability of instrument
• Phase III Field tests and dissemination

10
Phase I Instrument Development - Dec. 2000 -
May 2001

• Meetings with representatives from youth-serving
  agencies and funders
• Identification of core and optional outcome
  measures and questionnaire items
• Establishment of face validity of core measures
  and measurement strategies

11
Identification of Core Outcomes

• Three meetings resulted in list of 54 indicators
  and 10 outcomes
• peer and adult relationships
• constructive use of leisure time
• basic social skills
• community service
• health maintenance
• decision making process
• responsibility
• understanding boundaries/rules
• positive identity
• independent/daily skills

12
Narrowing List of Outcomes

• Agency representatives were asked
• What impact does your program have?
• What would you like to learn to improve the
  quality of your program?

13
Programs Wanted to Know

• Effectiveness of staff
• Effectiveness of services they provide
• Impact on youth and their families
• Impact of youth involvement in more than one
  program
• Youth Development philosophy of staff
• Gaps and what programs can do about them

14
Consensus Process

• Programs used nominal iterative process to
  identify consensus priority areas for youth that
  they could impact
• First Round 18 constructs
• Second Round 7 constructs
• Top 2 retained
• Third Round 4 constructs
• Top 2 retained

15
Outcomes for Operationalization

• Basic Social Skills
• Caring Adult Relationships
• Decision Making Process
• Constructive Use of Leisure Time

16
Candidate Questions

• Questions adapted from instruments by
• Add Health
• Boys and Girls Club of America
• Girl Scouts of America
• Metro Council for Teen Potential
• Worcester Youth Development Initiative
• YMCA

17
Phase II Piloting the Draft Instrument - May
2001 - March 2002

• Piloting in two phases
• Cognitive interviews to test validity of items
• Field test of internal consistency of items and
  feasibility

18
A. Cognitive Interviews

• 48 urban and suburban adolescents aged 10 to 17
• Mean completion time 11 minutes
• 70 had no suggestions
• 67 reported survey was easy to complete
• 81 understood everything in the survey
• 98 did not mind answering the survey
• Items re-worded to increase readability (now at
  4th grade level) and to simplify concepts

19
B. Field Test

• 389 urban and suburban adolescents
• Ages 10 to 19
• Large drop-in programs and smaller, structured
  programs
• Findings
• Feasible for program staff to administer
• Large groups required more staff time
• Easy for older adolescents
• Some issues remain for younger adolescents

20
Field Test Results

• Youth more attached to programs did better on
  measures
• Four constructs have several good factors for
  program use in evaluation
• Instrument consists of six factors, corresponding
  to three outcomes
• Internal reliability scores (as from .5782 to
  .8557)

21
Factor Analysis Core Outcomes

• Self Control
• Empathy Basic Social Skills
• Communication
• Staff Relationships Caring Adult
  Relationships
• Program Effect
• Decision Making Decision Making
• Constructive Use of Leisure Time

22
Factor Analysis - all participants
23
Factor Analysis - 13-19 year olds
24
Factor Analysis - Reliability
25
Phase II Cont

• Slight revision of individual items and
  rearrangement of questions leading to final
  instrument
• Pencil and paper survey
• 40 questions addressing four core outcomes,
  program participation, connectedness to program,
  and sociodemographic information
• Requires between 10 and 15 minutes to complete
• Development of training modules, scoring
  templates, and score report generating software

26
Phase III Dissemination in Rochester - May
2002 - May 2003

• Summer and Fall 2002 - 11 youth serving agencies
  in the Rochester area were trained to use the
  instrument and the report generating software
• TA provided to agencies to develop appropriate
  sampling plans
• During the program year of 2002-2003, over 1,000
  youth participating in YD programs in the
  Rochester area completed surveys

27
Current Steps May 2003 - present

• Software and score reports revised based on
  qualitative feedback from Year 1 implementation
• Continued training and TA to current users
• Dissemination to various other youth development
  programs through ACT for Youth Center of
  Excellence
• Validation studies

28
Additional sites

• Alaska - Residential School System served as a
  beta test site in school year 02-03
• Erie County, NY - over 75 programs including
  youth bureaus and youth boards, and UW and
  Department of Youth Services funded programs
• Hawaii - Childrens Alliance of Hawaii
• Oswego County, NY - Oswego City-County Youth
  Bureau funded programs
• Salamanca, NY - 21st CCLC program
• Syracuse, NY - Catholic Charities of Onondaga
  County sites
• Queens, NY - Queens Child Guidance Center Beacon
  OST sites

29
Use of Data by Programs

• Internal quality improvement. Examples include
• Reviewing and discussing score reports with staff
  and with Boards
• Comparing program scores within one agency to
  identify opportunities for improvement
• Reviewing curricula and current programming
  strategies
• Discussing program strategies with other similar
  programs
• Identifying training and technical assistance
  needs
• Reporting to funders
• Proposal writing
• Sharing data back allowing the creation of an
  aggregate community level score report using
  de-identified data

30
Training and Technical Assistance

• Training, TA, and use of the READY Toolkit are
  available to interested users
• Fees are based on the number of users and the
  level of training and TA required
• Options include
• Training and TA provided directly to end users
• One time training provided for end users, and
  continuous training and TA provided to a lead
  agency which then agrees to provide first line TA
  to end users

31
READY Toolkit

• READY Toolkit includes a CD which contains
• A Personalizable Instrument Template
• READY Analysis Program
• Toolkit Instructions Manual
• Users Agreement
• Instrument template may be personalized to
  contain program names and staff titles
• READY Analysis program allows community programs
  to enter their own survey data, and generate a
  score report
• Score report contains summary measures for core
  YD outcomes and frequencies for all survey items

32
SAMPLE Score Report Pg 1 of 17
33
SAMPLE Score Report Pg 2 of 17
34
Publications

• Klein JD, Sabaratnam P, Matos Auerbach M, Smith
  SM, Kodjo C,
• Lewis K, Ryan S, Dandino C.
  Development and factor structure of a
• brief instrument to assess the impact
  of community programs on
• positive youth development The
  Rochester Evaluation of Asset
• Development for Youth (READY) tool.
  Journal of Adolescent Health 2006
• 39 252-260.
• Sabaratnam P, Klein JD. Measuring youth
  development outcomes for
• community program evaluation and
  quality improvement Findings from
• dissemination of the Rochester
  Evaluation of Asset Development for
• Youth (READY) Tool. Journal of Public
  Health Management and
• Practice 2006 6(suppl) S88-S94.

35
For more information about the READY tool, please
contact

• Premini Sabaratnam, MPH
• Sr. Health Project Coordinator
• Div. of Adolescent Medicine, University of
  Rochester
• [phone]
• Premini_Sabaratnam_at_urmc.rochester.edu
• or
• Jonathan D. Klein, MD, MPH
• Associate Professor of Pediatrics and of
  Community Preventive Medicine University of
  Rochester
• [phone]
• Jonathan_Klein_at_urmc.rochester.edu

36
Or visit...

• The University of Rochester, Division of
  Adolescent Medicine, Leadership Education in
  Adolescent Health website at
• www.urmc.rochester.edu/gchas/div/adol/leah/resourc
  es.htm
• or
• The ACT for Youth, Center of Excellence website
  at www.actforyouth.net