Assessment of Intervention Outcomes Workshop: Thursday 7th February IYW Conference, Llandrindod Wells

1
Assessment of Intervention Outcomes Workshop
Thursday 7th FebruaryIYW Conference, Llandrindod
Wells

• Dr Dave Daley Dr Tracey Bywater

2
Overview of workshop

• Rationale for conducting your own evaluations
• How to manage the numbers
• Suggested evaluation material

3
Why evaluate clinical work?

• NHS Trusts need require you to conduct evaluation
• It is imperative to be able to demonstrate
  improvement associated with your groups
• Helps to maintain funding and/or win new funding

4
Is it difficult to evaluate outcome?

• The most important decisions to make when
  considering evaluation are
• 1) Design
• 2) Evaluation measures

5
1. Design

• It is very important that you have baseline
  measures (before) and outcome measures (after)
• It is also important that you have the same
  measures on everyone

6
What can we do with our evaluation scores I?

• Even simple spreadsheet programmes like Excel
  will allow you to conduct simple statistics
• For evaluation purposes you are most interested
  in change from start to end

7
Pros Cons of change scores

• Change scores are most useful
• Easy and simple way of evaluating change
• However recent suggestion that change scores on
  SDQ are less sensitive
• However formula is complicated, has not yet been
  fully validated and is not recommended for
  evaluations of small groups

8
What can we do with our evaluation scores II?

• Change scores should demonstrate improvements in
  child behaviour outcome
• You can also begin to examine whether your groups
  are more effective for particular types of parents

9
What can we do with our evaluation scores III?

• Rather than do lots of fancy statistics, it is
  also interesting to examine scores at outcome
• Most evaluation measures will have scores which
  indicate clinical concern
• It is very easy but also meaningful to examine
  the percentage of children and/or parents who
  score above cut off scores before intervention
  and after intervention

10
2. Evaluation Measures

• Which ones?
• Pathfinders project uses
• Demographic Questionnaire
• Beck Depression Inventory
• Eyberg Child Behaviour Inventory
• Strengths Difficulties Questionnaire x 2
• Parenting Scale
• Overt Hostility Scale
• Teacher Questionnaire/Report

11
Demographics

• What do you need to know?
• Do you want to compare outcomes of-
• older versus younger parents?
• Boys versus girls?
• Different areas?
• Any other ideas?

12
The Eyberg Child Behaviour Inventory

• 36 items, ages 2-16 years, 10 minutes to complete
• Problem score has a minimum of 0, maximum of 36
  (yes/no answers, problem or not)
• Intensity scale - minimum score of 36, maximum
  252 (scale of 1 7, where 1 never and 7
  always)
• Clinical cut-off scores, 127 intensity, or 11
  problem
• Purchase from http//www3.parinc.com/

13
Strengths Difficulties Questionnaire

• FREE, website http//www.sdqinfo.com/b1.html
• 10 minutes to complete, 25 items
• Emotional symptoms
• Conduct problems
• Hyperactivity
• Peer relationship problems
• Prosocial behaviour
• An additional impact supplement
• Parent teacher
• Normal, borderline atypical ranges/cut-off

14
Beck Depression Inventory

• 21-item elf-report inventory measuring the
  severity of characteristic attitudes symptoms
  associated with depression
• Each item contains four possible responses which
  range in severity from 0 ( I do not feel sad) to
  3 ( I am so sad or unhappy that I cant stand it)
• Score of 10-18 mild to moderate depression
• Score of 19-29 moderate to severe depression

15
Statistics with little maths

• Excel
• Graphs
• Mean SD
• Change scores
• Effect sizes

16
Welsh Sure Start Data
17
Effect sizes
Interven fup1 Control fup1 Difference pooled SD effect sizes
BDI 10.33 13.66 3.33 10.23 0.33
ECBI-P 9.55 13.68 4.13 8.11 0.51
ECBI-I 116.19 142.98 26.79 32.25 0.83
Cohens 1988 guidelines difference between means
divided by pooled SD. 0.3 clinically useful
change, 0.5 medium effect, 0.8 large effect
18

• Any questions?????