National Evaluation of the Childrens Fund NECF

1
National Evaluation of the Childrens Fund (NECF)
Talk to RSS Social Statistics Section /
Birmingham Local Group
Ian Plewis Institute of Education, University of
London 10 March 2005
2
What is the Childrens Fund?
Essentially it is a pot of money (initially 450m
in total) disbursed disproportionately to each
Local Authority in England. Each Local Authority
has generated Partnerships that, in turn, have
developed services aimed at children and young
people between 5 and 13. These services are
intended to be preventative and based on
partnership working and the participation of the
young people and their families.
3
What are the aims of the Childrens Fund?
To help overcome poverty and disadvantage More
specifically, to 1. Reduce truancy 2. Increase
educational attainment. 3. Reduce crime and
anti-social behaviour. 4. Improve health.
4
How is the National Evaluation of the Childrens
Fund organised?

• It is a joint project between the University of
• Birmingham and the Institute of Education in
  London,
• directed by Professor Anne Edwards in Birmingham.
  It
• started in January 2003 and, as things stand, is
  due
• to end in March 2006.
• There are two main research strands
• (i) Strand A in London, using a quantitative
  approach to measuring implementation and
  estimating impact.

5
How is the National Evaluation of the Childrens
Fund organised?

• (ii) Strand B in Birmingham, using a mostly
  qualitative and case study approach to learning
• and understanding more about the operation of
  the Childrens Fund at a local level in terms of
  the Partnerships activities in relation to
  multi-agency working, participation and their
  views about prevention.
• In addition, each Partnership has a small budget
• either to commission or to carry out themselves
  local
• evaluations.

6
Strand As research questions

• Can any improvements in the outcomes introduced
• earlier be attributed to the operation of the
  Childrens
• Fund? In other words, has the Children Fund had
  an
• impact?
• However, answers to question 1 can only be
• understood in terms of answers to a prior
  question
• How many, and what kinds of children and
  families,
• have actually used the services provided under
  the
• Childrens Fund umbrella? In other words, how
• successful has the implementation of the
  Childrens
• Fund been?

7
Strand As main methodological problems

• 1. Self-selection.
• 2. Heterogeneity of services.
• 3. Secondary analysis of datasets collected for
  other purposes.

8
Basic evaluation model
9
Millennium Cohort Study (MCS)
MCS First Survey Fieldwork

• Birth dates
• England Sept 2000 to Aug 2001
• Fieldwork child 9-10 months old
• England Jun 2001 to Aug 2002

10
Millennium Cohort Study (MCS)
MCS2

• The second MCS survey took place around the time
  of the childrens third birthdays
• The survey follows all those 11,533 families who
  took part in MCS1 in England plus some 500 New
  families who were missed by DWP last time
• Fieldwork started in September 2003 in England
  and will finish early in 2005.

11
MCS Target Sample, Sweep 1
All children living in the selected wards
Observed mean cluster size across the UK is 47
but the range is from 7 to 403.
12
Millennium Cohort Study (MCS)
MCS2 and the Childrens Fund

• 76 English wards used in the second sweep of the
  MCS were also wards with Childrens Fund services
• In these wards families with children aged
  between 4 and 15, in addition to their 3 year old
  cohort child, were asked information about (up to
  2) older siblings as part of the NECF.

13
Millennium Cohort Study (MCS)
Older Siblings (4 - 15 Year olds)

• Use of services
• Involvement with services
• Mother report on Strengths and Difficulties
• Mother report on spare time activities for
  children aged 4-9 years
• Siblings aged 10-15 do self-completion on own
  activities including truancy, smoking, substance
  use, etc
• Information collected about the schools attended

14
Millennium Cohort Study (MCS)
Possible Analysis

• Data collected at one point in time so although
  the MCS itself is longitudinal, the data on older
  siblings are cross sectional.
• We can report descriptive statistics on what type
  of people use Childrens Fund services and
  compare them to those who do not very helpful
  to policy makers who are interested in issues of
  targeting.

15
MCS3

• The third sweep of MCS will take place around the
• time of the childrens fifth birthdays, after
  they
• have started compulsory schooling.
• Fieldwork will start in November 2005 and is due
  to
• end in May 2006 in England and Wales.
• The original intention had been to measure the
• older sibs again in MCS3 but, at this point,
  funding
• for this is still uncertain.

16
PLASC / NPD

• Pupil Level Annual Schools Census and National
  Pupil Database
• Annual survey from 2002
• The purpose of this administrative dataset is to
  provide a census of pupils in England and Wales

17
PLASC / NPD

• Background Variables

• Free School Meals
• SEN
• Ethnicity
• English as a second language

18
PLASC / NPD

• Outcome variables

• PLASC/NPD is rich in detailed information on
  educational outcomes such as KS scores and
  examination results.
• Results at KS1, KS2, KS3 and KS4
• Exclusion from school

19
PLASC / NPD

• Analysis

• Schools with school-based programmes will be
  matched with similar schools without such
  programmes and the relative progress of pupils
  with the same characteristics in the two groups
  of schools will be compared.
• The 2003 data is longitudinal, containing prior
  Key Stage attainment. KS1 will be used as a
  control we do not have data on foundation stage
  profiles.

20
Families and Children Survey (FACS)

• FACS is a refreshed panel survey, which started
  in 1999. FACS started life as the Survey of Low
  Income Families (SOLIF). Therefore for the first
  two waves it considered only low-income families.
  
• We will be using FACS from 2002 onwards.
• 2003 fieldwork was carried out September 2003 to
  January 2004 this will provide us with an
  outcome measure.

21
Families and Children Survey (FACS)

• Service Use (2003 only)

• Library
• Parent and toddler group
• Playgroup
• Nursery
• Leisure Centre/Pool
• Park/playground
• GP
• Community Health Services
• Satisfaction as well as use

22
Families and Children Survey (FACS)

• Outcomes Education, Delinquency and Health
  (2002 and 2003)

• Absence from school
• Behavioural problems and exclusion
• Homework completion
• Smoking, drinking, drugs
• Trouble with the police
• Qualifications (GCSE etc)
• Parents subjective view of how child is doing
• Health

23
Families and Children Survey (FACS)

• Analysis

• We will use FACS to compare outcomes in
  Childrens Fund wards and non Childrens Fund
  wards. We will analyse the links between living
  in a Childrens Fund area, take up of services,
  and relevant outcomes.
• The advantage of FACS is that we have individual
  child level measures for both 2002 and 2003. The
  disadvantage is that exposure to services by 2003
  will not have been great.

24
Heterogeneity
Plewis, I. and Mason, P. (2005) What works and
why combining quantitative and qualitative
approaches in large-scale evaluations. Internation
al Journal of Social Research Methods
(forthcoming). Plewis, I. (2002) Modelling
impact heterogeneity. JRSS(A), 165, 31-38.
25
Heterogeneity
Suppose our outcome of interest, a measure of a
young persons criminal behaviour, can be assumed
to be measured on an interval scale, and suppose
that the sample members are clustered into areas,
some of which are intervention areas and some are
controls. We label the outcome yij where j (j
1..J) is the subscript for area and i (i 1..nj)
is the subscript for the young person. We label
the variable defining the intervention as xj
with, for now, x 1 for the intervention areas
and 0 for the control areas.
26
Heterogeneity
We want to introduce heterogeneity into our model
and we can partition this heterogeneity into four
classes heterogeneity that is a characteristic
of (1) individuals and (2) the areas in which
they live, and heterogeneity that is created by
(3) the intervention at the individual level and
(4) at the area level. We can represent these
four classes by variables z, using the
superscript (x) for variables pertaining to the
intervention
27
Heterogeneity
(1) zij are essentially fixed characteristics of
the target individuals - age, sex, social class
and so on, including a pre-intervention measure
of the outcome. The question here is does the
intervention effect vary according to these
fixed characteristics at the individual level
(level one)? (2) zj are fixed characteristics
of the area (and, therefore, constant for
individuals within areas) - urban or rural area,
proportion unemployed and so on. Now the
question is is the intervention effect
different for different types of areas?
28
Heterogeneity
(3) zij(x) are characteristics of the
intervention as experienced and perceived by the
individual, notably whether a target family
actually uses the service. (4) zj(x) are
characteristics of the intervention as it is
delivered at the area level organisations and
their characteristics delivering the
intervention and partnership arrangements behind
them, level of user participation in service
design and delivery, forms of delivery such as
how established the service is, and so on. These
are key variables for answering the what works
question measures of many of them are derived
from qualitative investigations.
29
Heterogeneity
(1a)
(1b)
(1c)
(1d)
with the intervention variable (i.e. xj) at level
two (areas).
30
Heterogeneity
The important parameters are ?01 in (1b) this
shows, in essence, whether the intervention
affects mean outcome at the area level, having
controlled for all the z variables. In other
words, a negative value of ?01 shows that
criminal behaviour was, on average, lower in the
intervention areas. The parameters ?03, ?04, ?11
and ?21 tell us about variability around this
mean (which might be zero). Thus ?03 a
non-zero value indicates that the intervention
varies in its effect according to the fixed
characteristics of the area. For example, the
intervention might be more effective in rural
areas.
31
Heterogeneity
?04 a non-zero value indicates that the
intervention varies in its effect according to
the way the service is delivered at the area
level. For example, an intervention might be more
effective if young people were involved in its
design. ?11 in 1(c) this tells us how much
heterogeneity there is in terms of fixed
individual family characteristics. For example,
an intervention might be more effective for young
people living in single-parent families. ?21 in
1(d) this tells us how much heterogeneity there
is in terms of characteristics of the
intervention as experienced by individuals. For
example, we would expect an intervention that is
used regularly to be more effective than one that
is only used sporadically.