Presentation Overview

1
Presentation Overview

• Introductions and Overview
• Quality Assurance on Investments in OST
  Experimental Evidence and Emerging Policies
• Youth Development Programs Do You Know Quality
  When You See it?
• Campfire USA
• YWCA of Minneapolis Girls and Youth Programs
• Framing Quality for Systems Level Use

2
Quality Assurance on Investments in OST
Experimental Evidence and Emerging Policies

• Grantmakers for Children, Youth and Families,
• Minneapolis, MN
• September 23, 2009

For information contact Charles Smith,
charles_at_cypq.org
3
Agenda

• What is Quality and Why Does it Matter
• New research on quality improvement
• Quality Assurance for Investments
• Case Example Cost of Quality Assurance

4
Part i

• What is Quality and Why Does it Matter

5
National Research Council (2002) Qualities that
promote positive development

• Physical and Psychological Safety
• Appropriate Structure
• Supportive Relationships
• Opportunities to Belong
• Positive Social Norms
• Support for Efficacy and Mattering
• Opportunities for Skill-Building
• Integration of Family, School and Community
  efforts

6
Durlak Weissberg (2007) Evidence on Quality
and Child Development

• Meta-analysis of 73 afterschool impact
  evaluations
• Programs with SAFE features promoted school
  achievement, feelings and attitudes, pro-social
  behavior
• Programs that lacked SAFE features DID NOT
  produce positive effects on any of these outcome
  measures.
• SAFE program features are

• Sequential use a sequenced set of activities to
  achieve skill objectives
• Active use active forms of learning
• Focused at least one component focused on
  developing personal or social skills
• Explicit targeted on specific personal
  or social skills

7
How and WhereSpecification of quality that
will translate to practice

• What does research suggest?
• Affect Active Learning Metacognition
  (Education)
• Relatedness Autonomy Competence (Psychology)

Make it simpler... Relationship Task
Increasing complexity
Where to look...? Program offerings, defined by
the continuity of staff, youth and learning
purpose.
8
Quality defined as staff practices during
offerings
9
Quality in a Multi-level Systems ContextHow do
policies and management influence instruction?
Engage
Interaction
Support
Safety
10
Video example of planning What did you see?
11
Youth have opportunities to make plans.
12
Part II

• New research on quality improvement - Youth
  Program Quality Intervention Study (YPQI) Funded
  by the William T. Grant Foundation

13
YPQI Logic Model Assess ? Plan ? Improve
Annual self / external assessment
14
Study Characteristics

• Cluster randomized field trial designed to test a
  scalable intervention
• 100 youth programs in 5 networks (4 states)
  Diverse sample by urbanicity, age, risk,
  content, staff education
• 18 month intervention sequence
• Outcome change in staff practice at PLC and POS

15
YPQI Major Findings Summary

• PLC-level mgmt practices were higher with
  assignment to Tx
• POS-level practices were higher quality with
  assignment to Tx, especially in focal areas
• YPQI was equally effective across diverse
  networks
• Level of implementation matters
• YPQI Cost per site per year, lt3,000 per site

16
YPQI Findings Closer Look Profiles of Management
Practice in PLC
17
YPQI Findings Closer Look Profiles of Staff
Performance at POS
18
Findings for the Tightly Coupled PLC

• Major findings again
• 65 of Tx managers fully implemented PLC
  management practices, only 18 of controls
• 42 of Tx offerings fell into the PYD profile,
  only 20 of controls
• Testing the overall theory Were Hi-PLC and
  Hi-POS paired more frequently when assigned to
  the Tx-group?
• Yes. Hi-PLC pairings with Hi-POS profiles were
  9.3 times more likely in the Tx group Lo-Lo
  pairings were 14 times more likely in the control
  group

19
Reflections on the Impact of YPQI

• Staff
• Our quality level top to bottom has improved. I
  dont think anyone didnt get something out of
  it. It helped us discuss what we do here so we
  got everybody on the same page. As little as
  recognizing a specific kid.. We now recognize
  the hierarchy of the pyramid, if you dont have
  safety, you cant have anything else. Everyone
  has been looking out for safety, to make the kids
  open up more. Weve seen the kids being more
  comfortable here.
• Manger
• Its all about evaluating your effectiveness and
  constant improvement as an individual. It was
  very helpful for all my staff and myself to be
  observers to sit in a group and not
  interfere-get an outside perspective in our own
  program. More youth buy-in, they are taking
  control over the program and the activities, they
  are the driving force instead of the staff being
  the driving force.

20
Customer Satisfaction

• New Mexico 21st CCLC Pilot Year
• 93.3-Report it was worth time and effort
• 86.2-Report their program improved
• Rhode Island 21st CCLC Year 2
• 84 of Executive Directors/Program Administrators
  said quality improvement planning process was
  helpful.
• Michigan 21st CCLC Year 5
• Since 2003, YPQA used for annual program self
  assessment (96.7 of grantees report that YPQA is
  a good use of time!)
•  

21
Part III

• Quality Assurance for Investments

22
YPQA and other quality assessment tools are
reviewed in Measuring Youth
Program Quality , A Guide to Assessment Tools,
Second Edition
23
Scale of CLASS (as of June 2009)

• Nationwide implementation with
• Head Start
• States in blue have some combination of state-,
  district-, and site-level implementations.

Hawaii
23
24
Scale of YPQA
7 Full-scale interventions

• 7 Statewide implementations (green)
• 12 States with city- or county-level
  implementations (red)

14 Pilot interventions
24
25
Diversity of Sectors in 18 Networks/Systems
26
21st CCLC State Projects 2009-10
1US Department of Education, http//www.ed.gov/abo
ut/overview/budget/statetables/index.html
27
Value of YPQI InterventionScaled to 25 sites
per network (est.)
28
A Hypothetical OST Network (200 sites)
n20 sites
n40 sites
n120 sites
n200 sites
29
Increase coherence, intentionality and impact of
current resource use
Policy/Funding
Evaluation and professional development
Accountability reporting requirements
30
Reading

• Yohalem, N., A. Wilson-Ahlstrom, et al. (2009).
  Measuring youth program quality A guide to
  assessment tool, second edition. Washington, DC,
  The Forum for Youth Investment.
• Durlak, J. A. and R. P. Weisberg (2007). The
  impact of after-school programs that promote
  personal and social skills. Chicago, IL.,
  Collaborative for Academic, Social, and Emotional
  Learning.
• Smith, C., T. Akiva, et al. (2009). Quality and
  accountablility in the out-of-school time sector.
  New Directions for Youth Development Defining
  and measuring quality in youth programs and
  classrooms. R. Granger, K. Pittman and N.
  Yohalem. San Francisco, Jossey-Bass. 121.
• Raudenbush, S.W. (2009). The Brown legacy and the
  OConner challenge Transforming schools in the
  images of childrens potential. Educational
  Researcher, 38(3), pp.169-180.

31
Youth Development Programs Do You Know Quality
When You See it?
Presenter Deborah Moore Operations
Director Youth Work Institute
32
Agenda for the session

• What are we doing across Minnesota to strengthen
  youth program quality?
• How has the Youth Work Institute supported youth
  programs in the search to improve quality?
• What has changed growth and lessons?
• Q A and Table Dialogues

33
Youth Program Quality A snapshot of what is
being commonly accepted across MN stakeholders
and practitioners groups

• Quality matters
• We need to improve quality levels (and we can)
• We need to align systems and stakeholders around
  the use and purpose of quality measures
• We must provide resources for programs to get the
  support they need

34
What we are doing to strengthen program quality
in MN

• Leading state convening efforts An afterschool
  advocacy/policy network (YCC), U of M Extension
  Center for Youth Development
• Adopting a shared research agenda YPQI, Cost
  Study, Workforce Study
• Connecting and aligning training and assessment
  support MNSACA, YIPA, YWI, Mentoringworks
• Uniting systems and policy action State
  appropriation, Governors Summit, funders
  convening

35
What is the Youth Work Institute doing to
strengthen youth program quality?

• The YWI is dedicated to expanding the knowledge
  and strengthening the practices of out-of-school
  program staff and volunteers.
• Public training and workshops
• Contracted services
• Research
• Observational Assessments
• Social networking opportunities
• Scholarship

36
At the core of our program quality services is
Quality Matters

• Quality Matters is a year long learning project
  that provides participants with information,
  skills, and hands-on support to improve the
  quality of their youth programs and provide
  positive experiences for youth.
• With the support of project coaches and the
  Youth Program Quality Assessment (YPQA) tool, a
  team of participants learn more about the
  essential components of a high-quality youth
  program and how to create environments that are
  positive places for young people to develop.
•  
• How it works
• Classroom Instruction with cross program
  participant groups 15 hours
• External assessment using the YPQA
• On-site coaching through a reflection and
  planning process from YPQA data
• Action plan and YPQA report left behind
• Often includes customized coaching and assistance
  after year 1

37
From Soup to Nuts YWI Program Quality Related
Services
38
QM Growth Pattern and Stats

• Starting in 2006 with 6 regional forums on
  quality research
• Ending in 2008 with 45 classes/sessions
• Quality Matters related services accounts for
• 65 of the total participants attending a YWI
  learning event (or 2,200 participants annually)
• 140 youth programs enrolling in 1-3 years of
  YWI support to improve quality
• Significant change in levels of quality

39
What Quality Improvement Looks Like

• Empowered and aligned staff at all levels
• New sense of POS responsibility on fire
• Increased interest in training, education,
  practical tools and other learning resources
• Increased activity and innovation in youth
  engagement strategies in particular

40
Evidence of Effect in MN

• Significant changes in YPQA Scores for QM sites
• Catalyzing a quality research and assessment
  effort for National 4H
• Uniting local systems and funders moving in
  shared vision, efforts and purpose for quality
  improvement (United Way, McKnight, Department of
  Education, Sheltering Arms, Minneapolis
  Foundation, Best Buy)
• Most large metro youth organizations are
  investing in 3-5 year systems change efforts
  (Community Ed, YWCA, YMCA, Boys and Girls Club,
  city of St. Paul, 21St Century, Beacons,
  Campfire, Parks and Rec) and some greater MN
  orgs (4H, Campfire
• Northern Star, Duluth YWCA, PACT 4)

41
Ideas to Chew on

• How do we support effective accountability (the
  youth outcomes vs. quality debate)?
• How do we strengthen learning and improvement in
  an under resourced field?
• How much quality is enough?
• How do we increase the public value of
• youth programs?

42
Web Resources

• Youth Work Institute Web Page on Youth Program
  Quality at
• http//www.extension.umn.edu/YouthWorkInstitute/
• Quality.html
• The Forum for Youth Investment (the Forum
• http//www.forumfyi.org/
•  
• National Institute on Out-of-School Time
• http//niost.org/
•  
•  
• Harvard Family Research Project
• http//www.gse.harvard.edu/hfrp/
• Contact info for
• Deborah Moore YWI
• ddm2_at_umn.edu or 612-625-7813

43
2007 Quality Assessment

• Selected to participate in YPQI study
• Targeted two specific sites
• Initial YPQA training and observation

44
Camp Fire USA MN Council2008 Quality Assessment
Report
4.27 4.36 3.35 2.64
45
Youth Quality Performance Assessment Quality
Pyramid
46

• Implementing Quality Improvements
• Rigorously train staff
• Continual observation, coaching feedback
• YPQA incorporated in all programs (outdoor
  community programs)
• Best Practices
• Implementing at management level
• Continuous training, coaching feedback
• Understand flexibility of tool and use it to work
  for you
• Reflection is KEY

47
YWCA of MinneapolisGirls and Youth Programs
48
YWCA of Minneapolis

• Overview of Girls and Youth Programs at YWCA
• Not typical center based approach
• 8 programs - offered within Minneapolis Public
  Schools and Community Centers
• 1500 youth annually
• School day and out of school time programs
• School year and summer programs

49
The Initial Journey

• Why Quality Matters?
• History Understanding the process
• Great concept, excited about potential of
• Building capacity within Girls and Youth team
  and within the organization
• Use tool to improve our work with youth
• Question from CEO How to work with more youth?
• Chose to embark on a Strategic Thinking Process
• Allocate training dollars into one larger quality
  initiative
• Incorporating Quality Matters into our strategic
  thinking process

50
The Process

• Five teams went through quality matters workshop
  and observation
• Initial Resistance Being observed
• Created options for staff
• To score or not score
• Offer feedback individually and/or feedback
  within the group
• Option to be observed a second time
• Staff began to own the process
• Managers suggested all staff be observed
• Staff wanted feedback excited about hearing
  strengths

51
Quality Assessment Impact

• Resulted in
• Increased capacity within the organization
• -ROCK STARS- 7 Very High Scorers
• -Recognized new capacity within particular staff
• We knew we were strong in Staff Interaction and
  Engagement, but realized we were not as strong in
  YOUTH Interaction and Engagement
• New focus on the YOUTH needs and strengths

52
Successes and Action Steps

• Quiet Staff Scored all 5s on 56 out of 60
  indicators on the YPQA
• Important We didnt know she was a ROCK STAR,
  she is quiet and didnt tout her strengths and
  methods
• Moving from Resistance to Actions
• One team was SUPER resistant initially
• Created 12 month action plan, bringing together
  talents of each of the four staff within that
  team
• Staff building internal capacity
• Creating staff observation tools to utilize the
  YPQA
• Using YPQA as supportive language to build skills
• Toolkits Energized and supported new staff
• Engaged and motivated staff

53
Framing Quality for Systems Level Use

• The Roles of Funders, Professional Development,
  and Policy-makers

54
Helping Systems Better Support Quality Practice
55
Developing a Quality Framework for the
State Minnesotas Approach

• Three Quality Retreats
• Hosted by Youth Community Connections (YCC)
• Funders private, corporate, state, United Way
• Professional development, membership
  associations, research
• Youth work professionals, tribal leaders,
  educators.
• Retreat Format
• High levels of engagement
• Robust dialogue and debate
• Contribute to the field by pushing for
  innovation
• Goals of the Retreats
• Define a Quality Framework for Minnesota
• Recommend key actions to move the Quality
  Framework ahead statewide
• Commit systems and individuals to advancing the
  Quality Framework
• Outcomes of Retreats
• White Paper Once We Know It, We Can Grow It.
• Small work groups built around embedding
  framework into grant-making, policy, education
  and

56
Broad Outcomes Across Multiple Systems

• Outcomes for the Field
• Quality is measurable, malleable, and
  marketable
• Quality is a concept that is broadly known and
  valued
• Unifies field that values variety as a virtue
• Outcomes for Funders
• Optimizes impact of grants and return on
  investment
• Unites and aligns investments from diverse
  funding streams
• Easily embedded in grant guidelines,
  communications, evaluation, and reporting.
• Outcomes for Youth Workers
• Increases satisfaction and retention of youth
  workers
• Develops reflective professional practice and
  data-driven program improvement
• Outcomes for Youth
• Increases youth participation, engagement, and
  positive results across all areas of development
• Poor quality programs are not neutral. Could be
  detrimental.