Realising Value from Follow-up of Non-response in ABS Business Surveys

1
Realising Value from Follow-up of Non-response
in ABS Business Surveys

• Authors Matt Ashton, Louise Gates,
• Dina Neiger, Greg Griffiths
• Presenter Greg Griffiths

2
Total Survey Design

• Linacre Trewin (JOS 1993) A key question for
  a NSO is to determine how available funds are to
  be allocated over the full range of activity
  required.
• This requires a full understanding of fitness for
  purpose, of how the quality of collection outputs
  change due to increasing effort and an
  understanding of process and error
  interdependencies.

3
Practising Total Survey Design

• Need to further our knowledge of the survey
  process and the quality/cost interactions between
  its component activities.
• Within the ABS the Establishment of the Economic
  Statistics Data Centre has provided an
  opportunity to
• Separate the design of field operations from
  sample design and data compilation
• Establish systems for the collection of paradata
  (data about process)
• Collection of data on current processes is not
  enough however, there needs to be a good
  understanding of the costs of alternative
  processes.
• A consequence of process complexity and the
  developing state of our knowledge is that we are
  still frequently grappling with developing our
  understanding of component sub-processes.

4
Intensive Follow-up (IFU)

• Standard follow-up procedures for ABS business
  surveys include
• Reminder letters
• Phase 1 IFU an initial phone contact for all
  outstanding units starting at a fixed time after
  the due date to determine the live/dead status of
  the unit, to confirm that they received the form
  and to ascertain a date for the forms return
• Phase 2 IFU phone follow-up targeted at
  significant units. During this phase all
  outstanding units are likely to be contacted,
  however, unlike Phase 1, the contact is
  prioritised so that significant units are
  contacted first
• Phase 3 IFU phone follow-up of critical
  non-respondents
• Ongoing follow-up action from direct contacts and
  in-coming calls.

5
Current ABS IFU practice

• Significance of units determined in many
  different ways within and across collections
• strata with 'low' response rates or high
  contribution from imputation
• units with specific business structures or
  activity, high complexity or special processing
  requirements
• units according to their contribution to estimate
  
• units due to their history (past non-response or
  new-on)
• consequential contact of units with common
  ownership
• strata based on sample design properties (units
  in completely enumerated strata, units whose
  responses will or will not be augmented by
  administrative data)
• strata based on potential change in variance
• units based on some definition of unit 'size' eg
  unit sample weight compared to total weight at
  some level of aggregation.
• non respondents in publication cells with very
  low target sample numbers (for example to
  alleviate potential confidentiality issues in
  final data)
• units with high weighted expected absolute
  difference between imputation and response
  (McKenzie ICES II).

6
ABS Projects under way

• To understand from the viewpoint of statistical
  theory how units should be prioritized so as to
  minimize mean square error and develop unit
  scores.
• To identify best practice and cost-efficient
  approaches to cost-effective collection of data
  for the economic statistics program
• To understand relationship between cost of
  follow-up and survey response examination of
  telephony patterns and processes to identify cost
  effective practice

7
Number of Outbound contactsSingle collection
cycle for Annual Business Collection
8
Number of Outbound contactsSingle collection
cycle for Quarterly Business Collection
9
All outbound contacts by type of contactAnnual
and Quarterly Survey compared
10
Percentage contribution to estimates by number
of contacts- Annual survey one cycle
11
Percentage contribution to estimates by number
of contacts - Quarterly survey one cycle
12
Time slice dataQuarterly survey one cycle
13
Time slice dataQuarterly survey one cycle
14
Hourly variation from mean contact success rate
15
Future work on contact data

• further surveys and further work on the contact
  patterns
• impact of reminder letters
• impact of pre-approach letters
• impact of significance flag
• difference between faxed and mailed reminders
• Experiment with intervening phone call between
  first and second reminders

16
Wrap up

• We are still at the stage of trying to understand
  components of the overall data gathering and
  dissemination process.
• Need to ensure systems and data stores are
  designed with the collection of paradata in mind,
  
• Paradata collection for any one process be done
  with a view to upstream downstream processes.
• Cost/Effort versus Benefit models are still a
  cottage industry and much analysis of paradata
  remains to be done.