Prevention and Early Intervention Pilot program Client Outcomes Studies

1
Prevention and Early Intervention Pilot program
Client Outcomes Studies

• Australasian Evaluation Society Lunchtime Seminar
  February 2007
• Presented by Dr Terrence OBrien

2
Evaluation

• Evaluation should not be seen as an end product
  or one off event.
• Evaluation is a participatory process.
• Evaluation is an essential tool for continuous
  quality improvement.

3
Client Outcomes Evaluation

• Application to human services is challenging
• Changes in familys behaviours due to
  interventions needs to be measured, but not easy
• The question is more about how rather than if it
  can be done or not..

4
What is an Outcome evaluation?

• Outcome evaluations/studies look to the extent to
  which program participants experience benefits or
  changes in their behaviour and life circumstance
  and indications that these changes are related to
  the services received. (Mikka 2001)

5
Client Outcomes Evaluation

• Previous difficulty in evaluating clients
  outcomes.
• Outcomes in humans services hard to define and
  measure.
• Sensitivity to professional and client needs.
• Human Services not conducive to rigorous
  experimental approaches.
• Research in this area is becoming more popular.
• Growing trend of outcome measures being used to
  monitor funded NGOs.

6
Outcomes Evaluation

• Help to inform the choice and validity of
  intervention, ensuring best/evidence based
  practice principles are followed
• Allow the organisations to be outcome focussed.
• Strengthen existing services.
• Target effective services for expansion.
• Identify staff and volunteer training needs.
• Develop and justify budgets.
• Prepare long-range plans.
• Focus board members' attention on program issues.

7
PEI Pilot Programs Client Outcomes Studies

• Putting Families First
• Future Directions Initiatives PEI 1,2,4, 5, 7
  and 8
• PEI and Indigenous Family Support Trials and
  Pilots
• Anecdotal evidence of positive outcomes from
  trials (one year)
• Client Outcome Studies needed to be developed for
  the pilot evaluation (3 years)

8
PEI Pilot Programs

• 25 Trial Programs
• 18 Pilot Programs
• 12 Communities and 6 Child Safety
• Evaluation Processes
• Demographic and Case Management and
• PEI Principles Survey
• Client Outcomes Studies

9
MISSION AUSTRALIA-PROJECT CIRCUIT
BREAKEROUTCOMES EVALUATION PROJECT CONCEPT

• When my service does (our services do) -
  intervention
• Strengths based early intervention and
  prevention-
• With children, parents, families who-
• Consist of families with children between the
  ages of 10 and 16 years who live in the Northern
  Suburbs of the Department of Communities Brisbane
  City region.
• When the context is-
• Referrals of families with students in secondary
  school who are experiencing relationship
  difficulties and other issues such as truanting
  and anti-social/deviant behaviour.
• What change is there in (what change am I wanting
  to see)
• Stability and quality of family relationships and
  communication

10
Project concept cont.

• And how does this impact on the child by
  improving their
• Level of positive engagement with education
• In terms of -
• School attendance
• Behaviour at school

11
OUTCOMES EVALUATION PROJECT DESIGN

• Organisation Mission Australia - Project
  Circuit Breaker
• 1. What are your Questions?
• When my service provides strength based, solution
  focused case management,
• is there improvement in the young persons
• school attendance,
• behaviour at school and
• behaviour at home during and after the
  intervention.
• Are there improvements in the quality of the
  family relationships during and after the
  interventions?
• What are the individual outcomes that can be
  attributed to each intervention strategy that is
  implemented with a family?
• 2. How will you find out?
• Methodology
• Type Descriptive (Survey Design)/pre and
  post Interventions and after 3 months.
• 4 case studies. (Using the NCFAS Child
  Behaviour Checklist and other questionnaires
  devised by Project Circuit Breaker staff)
• Over what time 3-6 month intervention periods
  over a total of 2 years

12

• 3. Sample
• Type of sample Purposive sample - Case Study
  sample consenting families
• How many families in the sample 30 Dependent
  on number of referrals received
• Description of the sample Families and young
  people referred from Secondary Schools within
  Project Circuit Breakers catchment area.
  Families experiencing difficulties with young
  persons behaviour. Young person experiencing
  difficulties with relationships parent/s and at
  school.
• 4. What will you do about
• Confidentiality and privacy issues Information
  about the purpose, duration and the role that
  families play in the study are given to families
  prior to request of information. Families will
  not be identified codes will be attributed to
  each member of family.
• Informed consent Forms to be signed
• Ethical issues Using AES ethical principles
• 5. Tools and Techniques
• Types Standardised tests Pact c, NICFAS
  Quality of Family interactions, Child Behaviour
  Checklist for ages 6 18 (home/school/self
  report) attendance records (3 months prior/3
  months during intervention) and questionnaires
  devised by PCB staff, GO, and Parent/young
  person. Case Studies developed from case notes
  semi structured interview at 3 months after
  intervention.
• Key characteristics to measure Changes in
  presenting issues, Family interactions, Changes
  in behaviour at school, School attendance.
• When will you use them? Pre and post
  intervention and 3 months after intervention

13
Client Outcomes
14
Client Outcomes
15
Client Outcomes
16
Client Outcomes
17
Survey design
18

Survey design
19
Case Studies
20
Case Studies
21
Case Studies
22
Narrative Inquiry
23
Narrative Inquiry
24
Parents YES Outcomes Study

• relationships
• parenting skill and knowledge
• connections to support networks and
• participation in life.
• Improvements in positive child/parent
  relationships and interactions
• Children are reaching their development
  milestones
• Parenting skills and knowledge has increased
• Connections to support networks are strong
• Changes have been seen in life participation
• An increase in stable, safe, secure and adequate
  housing

25
Project Circuit Breakers, Client Outcomes Study

• a young persons attendance and academic
  performance at school
• general changes in the young persons behaviour
  and
• movement towards improvement in family
  functioning
• all families perceived there was significant
  change to the presenting issues due to
  intervention
• they reported the influence of intervention on
  goal achievement
• they perceived that progress towards goals was
  due to intervention
• all young people maintained or showed improved
  grades in at least 3 out of 4 school subjects
  after intervention, yet their school attendance
  showed little change between initial and closure
  measurements and
• all young people scored lower (behaviour
  improvement) after intervention on at least one
  of the oriented scales scores (Child Behaviour
  Checklist and the Youth Self Report).

26
Red Crosss Intensive Family Support program
Client Outcomes Study

• the childrens school attendance and behaviour
  and
• educational achievement.
• improvements in school attendance and positive
  improvements in behavior for children both at
  home and at school
• positive feedback from the childrens teachers
  regarding educational achievement and attitude
  and
• most families made perceived and made positive
  progress towards their goals.

27
The Client Outcomes study for the Munujali Jymbi
Centre

• improvement in family and community resilience
  and well being (abbreviated)
• establishment of sustainable self determining
  family and cultural support systems and
  structures
• reduction in the levels of representation of
  Aboriginal people and Torres Strait Islander
  people in statutory systems and crisis services.
• That there is increased community acceptance of
  the Centre and a high level of involvement in
  their programs
• 5 local people have been trained and are now
  employed by the centre
• The service has provided mentors for young
  people, children and their families and
• The community reports that they have increased
  self confidence and direction and have a broader
  vision of the future.

28
The Eaglehawk Indigenous Family Support Programs
Client Outcomes

• family resilience and self-esteem
• household budgeting and management skills and
• community engagement and contact with support
  networks.
• both families exhibited improvements in family
  resilience
• the social and emotional wellbeing of the
  families improved
• a direct correlation between budgeting /
  household management and secure housing / food
  supply, with both families showing improvement in
  each
• external relationships and health management
  improved. Improvements in positive child/parent
  relationships and interactions