NDC Household Survey 2006

1
NDC Household Survey 2006

• An overview
• Rachel Williams, Jessica Vince and Leon Page

2
Session objectives

• Background information about this years
  household survey
• Review data outputs available and give practical
  tips on reading and interpreting data
• The Extranet (www.ipsos-mori.com/ndc)
• Sources of help at Ipsos MORI

3
Background to the household survey
4
Overview

• Survey aims to provide information on the key
  outcomes from the NDC programme and to show
  change over time
• Third survey in the series
• initial survey in 2002
• 2004 and 2006 surveys are longitudinal going
  back to as many of the original 2002/4
  respondents as possible plus a cross-sectional
  top-up sample
• Ipsos MORI/GfK NOP conducted 15,792 interviews
  between May and October 2006 c.400 in each NDC
  area
• National benchmarks - from existing surveys and
  new Omnibus survey are NDC areas catching up?

5
Changes for 2006

• Sample size reduced to 400 interviews
• Longitudinal and top-up addresses issued in the
  same proportion as 2004
• Questionnaire length reduced
• 25 minute core questionnaire
• Partnership specific questions optional
• No research among movers, beneficiaries or
  business

6
Questionnaire
7
Core questionnaire asked in all NDC areas

• Core questionnaire covers each theme in
  national evaluation
• Housing and the local environment
• Crime
• Worklessness
• Education
• Health
• Plus profile information, basic income questions
  and general perceptions of area/community

8
Core questionnaire 2

• Wherever possible questionnaire kept identical to
  2002/04 to allow changes in views to be compared
• Designed with NET following outcome review and
  review following each wave
• Full pilot in similar regeneration areas each
  wave
• Some questions follow-up answers given by
  respondents in 2004 e.g. Last time you were very
  worried about being burgled now you are not very
  worried why have your views changed?
• Includes national benchmarks to compare against
  the country as a whole
• This year each partnership could add 5 minutes
  worth of their own questions

9
Sampling
10
Why do we sample?

• Not feasible to question all residents of a
  particular area e.g. 17,000 residents in the
  East Brighton NDC area alone
• Interview a smaller number on the basis that they
  share the same characteristics as the population
  generally

• Works but relies on strict procedures

11
Bigger is betterbut size isnt everything
Sample size
12
Household survey two samples
13
Longitudinal sample
Longitudinal
Return to individuals interviewed in 2004 and
attempt to interview them
If named respondent has moved or died then
randomly select a substitute
10,770 successful interviews at original
addresses
14
Topup sample
Top-up sampling involves a number of steps
Means every person has an equal chance of being
interviewed
15
Sample structure - example
Interviewed 283 people at these addresses 238
original respondents 45 new respondents
16
Fieldwork
17
Fieldwork

• Fieldwork May - October 2006
• Full briefings for all interviewers (28 sessions)
• dummy interviews before starting work
• where possible using the same interviewers as
  2002/4
• Advance letter and FAQ sheet sent to respondents
• Interviews conducted using Computer Assisted
  Personal Interviewing (CAPI)
• Minimum of 6 calls, including two at weekend or
  evenings, plus further re-issues.
• Quality control - back-checking and accompaniment
  
• Over 370 interviews conducted in other languages
• Response rates improved on 2002 and 2004

18
Household survey outputs
19
Household survey outputs

• Range of outputs provided by Ipsos MORI team
• Key indicators (most accessible)
• Topline questionnaires
• Computer tabulations (simple sub-group analysis)
• SPSS dataset (most technical)
• Will suggest how and when you might want to use
  these
• Extranet

20
What are key indicators?

• Show summaries of some key questions from the
  survey
• Easy to read if showing to non-researchers in
  your NDC
• Useful if you want a quick overview of the
  results and change over time
• Can be used to create a summary of findings for
  internal use e.g. Board meetings etc.
• Cover all the theme areas (Crime, Housing etc)

21
What are key indicators?
Columns giving partnership and aggregate data for
all three waves, and national benchmark
Theme heading
Summary of all those very/fairly satisfied with
accommodation. Does not show dissatisfied
22
What are toplines?

• Show all of the questions from the survey in the
  order these were asked on the questionnaire
• All responses to each question are shown (e.g.
  very and fairly satisfied/dissatisfied) not just
  summaries
• Questions are grouped by theme to make them
  easier to find e.g. CR questions (CR1, CR2 etc)
  are from the Crime theme
• Use the marked-up questionnaire if you want
  overall results but do not want to analyse by
  sub-groups e.g. men vs. women
• Fairly easy to read for those not experienced in
  research

23
What are toplines?
Theme heading and question number
Full question text
List of all responses to the question
24
What are toplines?
Filter an indication of who is asked the
question
Base a definition of who was asked the question
and how many people this includes
National source most recent benchmarking
figures available (mostly for England)
25
Toplines - Interpretation

• Number of pitfalls to be aware of when reading
  and interpreting toplines
• Check the bases underneath the question is
  everyone being asked or not?
• Some demographic questions are based on all
  household members rather than all respondents.
  Gives more accurate information on age, gender
  and work status
• Read the introductory page!

26
Toplines - functionality

• Topline document is now in Excel rather than Word

• Advantages
• Display only question text
• Display only benchmarked questions
• Quickly calculate change between years using
  formulas IF(H17"",0,H17) etc
• Run charts from the data
• Will be available in PDF version
• Working on formulas to produce automated
  significance testing (TBC)

27
Statistical reliability - 1

• Not all differences between 2002, 2004 and 2006
  data will be significant
• E.g. if satisfaction with area has increased by 2
  percentage points between 2004 and 2006 this will
  not mean that there has been a significant
  increase in satisfaction
• Whether changes are significant also depends on
  how many people were asked a question
• The fewer people asked a question, the greater
  the difference will have to be for this to be
  significant
• E.g. more difficult to find significant
  difference where 50 respondents have been asked
  compared with where 400 have been asked

28
Statistical reliability - 2

• Where all respondents have answered a question
  need differences of
• 8 percentage points or more between 2002, 2004
  and 2006 figures to be sure they represent actual
  change and are NOT due to chance
• 6-7 percentage points between individual NDCs and
  the aggregate or national benchmark figures
• Can use our ready reckoner available on the
  Extranet
• An example

29
Statistical reliability calculator
Testing change in proportion feeling unsafe
between 2004 and 2006 for Partnership X
30
What are computer tabulations?

• All responses to the survey are broken down by
  around 40 different sub-groups
• Sub-groups are respondents who are grouped
  together in the data because they have something
  in common. This may be demographic (e.g. their
  age) or attitudinal (e.g. they want to move out
  of the area)
• Examples of sub-groups are
• Men (demographic)
• Those who earn less than 100 per month
  (demographic)
• Those who have heard of NDC (attitudinal)

31
What are computer tabulations?

• Why would you want to use these?
• You may want to see if all sections of the
  population are benefiting equally from NDC
• E.g. fear of mugging may be going down overall,
  but are any groups being left behind?
• Can compare 2006 results with 2004 results
  young people may be 2 less likely to be mugged
  than in 2004 compared with a drop of 15 overall
• Can also use for baseline/monitoring data are
  Black residents more likely than average to be
  earning 100 a month or less?

32
What are computer tabulations?
Question on tenure who are most likely to be
private renters?
16-34 year olds significantly more likely to rent
privately (text shown in bold)
33
What are computer tabulations?
BUT Crucial not to compare sub-groups with small
bases as these are unreliable
34
Computer tables - significance testing

• Letters used to identify which results are
  significantly different from each other

35
Computer tables - significance testing

• Each column is tested against the total to see
  whether the differences in results are
  significantly different
• a letter under a result indicates it is

36
A few things to watch

• Some tables run over two pages because the list
  of possible answers is so long i.e. 4 pages per
  question (e.g. WO1)
• As well as the actual answers that can be given,
  also include derived summaries at the end of the
  list (eg WO1 ILO Unemployed)
• There may be tables for the same question but
  with different bases. E.g. at WO1 there are
  tables based on all respondents and also on all
  working age respondents
• Sometimes have summary tables where a battery
  of questions are asked
• There are a few derived tables e.g. HO3
  cross-tabulated by HO4

37
SPSS data

• A software program that allows you to conduct
  your own analyses (Statistical Package for the
  Social Sciences)
• Allows you to do a wide range of analyses that
  are not included on the Computer Tables
• E.g. You could find out the characteristics of
  Trapped residents those who want to move but
  do not think they will, by cross-tabulating two
  survey questions against each other
• Every NDC has their own dataset
• File contains some additional derived variables
  eg workless households
• BUT your NDC may not have bought SPSS software
  so you may not be able to use this data (LA may
  have it though )
• Ask the Ipsos MORI team if you need any help with
  using SPSS

38
A note on weighting

• Two sets of weights applied to the survey data
• Selection weighting to correct for unequal
  selection probabilities (every person should have
  equal chance of selection)
• Profile weighting to ensure that survey is
  representative of the population as a whole
  (based on Census)
• All data presented in the survey outputs is
  WEIGHTED
• EXCEPT, the final column on the computer tables -
  this presents the raw data
• Correct weights must also be applied to the SPSS
  data
• NB weighted samples are less accurate need to
  bear in mind when checking for significant
  differences

39
Extranet
40
Extranet - 1

• You may have used this with the 2002 and 2004
  data
• Analysis tool taken off-line recently to add the
  new 2006 data
• Can still access survey documentation
• 2006 data will be available in February next year

41
Extranet - 2

• Extranet allows you some of the flexibility of an
  analysis programme like SPSS but easier to use
• Can analyse your partnerships results by
  demographic factors such as age or gender
• Can compare your results against those of other
  relevant partnerships and against the aggregate
  data set
• Can filter results e.g. the findings just for
  women aged 16-24
• Can compare change in data over time

42
Extranet - 3

• An on-line resource containing all NDC data
• Run your own tables on issues of interest
• Download toplines, key findings and computer
  tables
• Data available for all partnerships
• Possible to compare yourself with other local
  NDCs or those with similar characteristics
• Each partnership has their own log in
• Benchmarking data and links to useful sites
• Information about the household survey
• Background to each wave
• Glossary of research terms and training slides

43
Some on-screen examples
44
Using the data
45
Using the data

• Background information/reality check along with
  other sources of data
• eg if feelings of safety have increased how does
  this relate to crime statistics/what the police
  are saying locally. 
• Partnership specific questions used to evaluate
  specific projects
• Post coding data to provide neighbourhood level
  information
• Wider area benchmark surveys

46
Thank you
For further information or help please
contact jessica.vince_at_ipsos-mori.com (020 7347
3148) leon.page_at_ipsos-mori.com (020 7347
3152) hayley.muller_at_ipsos-mori.com (020 7347 3252)