The impact of extended call regimes on sample representativeness

1
The impact of extended call regimes on sample
representativeness
ACSPRI Social Science Methodology
ConferenceDecember 2006
2
Overview

• Main focus
• The impact of using extended call routines on
  sample representativeness
• The bias towards higher socioeconomic status
  groups in telephone survey research

3
Surveys used in this analysis

• The 2005 Victorian Population Health Surveys
  (VPHS)
• Victorian Department of Department of Human
  Services.
• A high quality annual survey of 7,500 Victorians
  aged 18 years.
• Used to inform and support planning,
  implementation and evaluation of adult health
  services and programs throughout Victoria.
• The 2006 Community Attitudes to Violence Against
  Women Survey (VAWS)
• VicHealth.
• Full qualitative development phase
• 2,000 general community surveys
• 800 surveys with persons of selected CALD
  backgrounds background.

Initiatives to address areas where women
experience inequity,under-representation and need
4
Surveys used in this analysis
Surveys used in this analysis

• The 2004 International Crime Victimisation Survey
  (ICVS)
• Australian Institute of Criminology / United
  Nations Organisation.
• 6,000 general community surveys on crime
  victimization issues. Australian component of a
  large-scale international survey,
• Community Attitudes to Road Safety (CAS)
• Australian Transport Safety Bureau
• Long running survey program
• An annual community attitudes survey of 1,500
  persons aged 15 years and over regarding
  attitudes to road safety and road safety counter
  measures

5
Extended call cycles - Interviews by call
attempt
Call attempt process
Proportion of 18 to 44 years olds by call attempt
(a)
a) 55 of persons aged 18 years and over b) The
base for the ICVS is persons aged 16 to 44 years
6
Contactability

• The differences between those interviewed early
  and late in the call cycle are not aged-related,
  as such, but a function of contactability.
• Therefore we would expect to see other
  differences between those interviewed early and
  late in the call cycle

7
Socioeconomic and demographic characteristics
by call attempt
Denotes significant difference between estimates
obtained from calls 1 to 3 at the 95 confidence
interval.
8
Selected findings by call attempt
Denotes significant difference between estimates
obtained from calls 1 to 3 at the 95 confidence
interval.
9
Selected findings by call attempt
Denotes significant difference between estimates
obtained from calls 1 to 3 at the 95 confidence
interval.
10
Subgroup differences

• While we have observed broad attitudinal,
  behavioural and demographic differences between
  those interviewed early and late in the call
  cycle it is of interest to establish whether or
  not there are differences within selected
  population subgroups.

11
Selected findings by call attempt Subgroup
analysis
103
Denotes significant difference between estimates
obtained from calls 1 to 3 at the 95 confidence
interval.
12
Extended call routines
The profile of respondents interviewed late in
the call cycle differs substantially from those
interviewed early in the call cycle.

• Representativeness is increased by
• Extended call routines
• Carefully controlled sample release
• Longer interviewing periods

When budgets are tight probably best to invest in
an extended call cycle rather than refusal
conversion activity.
13
Bias towards higher socioeconomic status
groups
Continuing with the theme of trying to ensure our
survey estimates reflect the population on which
they are based as accurately as possible Even
with optimal sample utilisation and response
maximisation techniques and traditional weighting
methods, we often still see a bias towards higher
SES groups

• Community Attitudes to Violence Against Women
  32 tertiary qualified
• Victorian Population Health Survey 30 tertiary
  qualified
• The International Crime Victimisation Survey
  29 (est.) tertiary qualified
• Community Attitudes to Working Age Income Support
  Customers 25 tertiary qualified
• ABS national population benchmark 19.6

The question is Does this matter, and, if so,
what can be done about it?
14
Impact of educational attainment on selected
attitudes and behaviours
We know selected attitudes and behaviours vary by
educational attainment
All results are statistically significant at the
95 confidence interval.
15
Original weighted estimates and revised
estimates (adjusted for educational attainment)
Adjusted Est
Original Est
Selected attitudes and behaviours
78.8
79.2
More needs to be done to help the unemployed
17.9
16.9
Daily smoker
82.5
83.7
Could raise 2,000 within 2 days, if needed
51.7
52.3
Victim of crime in the last 12 months
38.1
39.2
High gender equality rating
16
Concluding Remarks

• Depending upon the level of precision required,
  it seems it is worth considering applying some
  sort of educational attainment (or broad
  socioeconomic status weighting factor) to some
  survey estimates
• The solution is more complicated than applying a
  post weight a rim weighting or raking procedure
  would need to be developed
• Our scan of the literature does not reveal a lot
  of work in this area
• Further work needed and warranted?
• www.srcentre.com.au
• darren_at_srcentre.com.au