JOINT EUROPEAN COMMISSION OECD WORKSHOP ON INTERNATIONAL DEVELOPMENT OF BUSINESS AND CONSUMER TENDEN

1
JOINT EUROPEAN COMMISSION OECD WORKSHOP ON
INTERNATIONAL DEVELOPMENT OF BUSINESS AND
CONSUMER TENDENCY SURVEY

• Per aggiungere alla diapositiva il logo della
  società
• Scegliere Immagine dal menu
  Inserisci
• Individuare il file con il logo della società
• Scegliere OK
• Per ridimensionare il logo
• Fare clic su un punto qualsiasi del logo. Attorno
  all'oggetto verranno visualizzati i quadratini di
  ridimensionamento.
• Per ridimensionare l'oggetto, utilizzare i
  quadratini.
• Per conservare le proporzioni dell'oggetto da
  ridimensionare, trascinare i quadratini di
  ridimensionamento tenendo premuto MAIUSC.

• Task Force on Improvement of Response Rates and
  Minimisation of Respondent Load
• Part 1 Relationship between response rates and
  data collection methods
• Bianca Maria Martelli,
• Institute for Studies and Economic Analysis -ISAE
  Rome, Italy
• BRUSSELS, 14-15 November 2005

2

TERMS OF REFERENCE

• 1. DATA COLLECTION METHODS
• which is the most effective for
• Improving response rates
• Reducing burden
• Impact on costs
• Sectoral effects
• Size effects
• 2. COMMUNICATION METHODS
• most effective and cost efficient for
• Compulsory surveys
• Voluntary participation

3
ACKNOLEDGEMENTS

• The present research has been realised with the
  substantial contributions of the members of the
  EC/OECD task force on RR who supported me with
  extremely useful material , references,
  information, personal experience and suggestions.

4
WHY IS THE RESPONSE RATE IMPORTANT?

• Response Rate (RR) one of the most important
  factors to ensure a high quality of the BTS
  results (Etter, 2002)
• Non response rate (NR) is one of the components
  of non-sampling error that may affect the survey
  and the most efficient strategy for reducing
  measurement errors is to eliminate possible
  causes of such errors during the whole survey
  design stage (OECD 2003)

5
WHY FIRMS/CONSUMER RESPOND?

• Conceptual framework for COS/BTS Survey
  participation (proposed by Willimack 2002,
  reported by Petroni 2004)
• Businesses weight response burden against
  business goals when considering the decision to
  participate in a survey. Both burden and goals
  may be impacted by
• factors relating to the external environment, the
  business and the respondent, which are not under
  control of the survey organisation
• survey design features, which are under the
  survey organisation control

6
FACTORS INFLUENCING RESPONSE RATES
Kind of Survey

• Subject

Reporting, Sampling, Response Unit
Target Universe
RR
Questionnaire
Frame
Sampling design
Data Collection Mode
7
WHAT IS (NON) RESPONSE RATE?

• CONFUSION ON THE TOPIC
• people deals with RR/NR, but not explain what
  really apply
• NEED OF CLEAR RR/NR DEFINITION
• The response rate is the number of
  complete/incomplete interviews with reporting
  units divided by the number of eligible reporting
  units in the sample (AAPOR 2004)
• SEVERAL KIND OF RR/NR
• UNIT / ITEM NR

8
KINDS OF NON RESPONSE RATES

• OECD DEFINITIONS (2003) (1/3)
• where n number of enterprises which did not
  submit useable information
• n number of enterprises in the survey

9
KINDS OF NON RESPONSE RATES

• OECD DEFINITIONS (2003)(2/3)
• In case of unequal sampling fraction but uniform
  weights a proper measure is the following

10
KINDS OF NON RESPONSE RATES

• OECD DEFINITIONS (2003) (3/3)
• In the most general case, of unequal sampling
  fraction and reporting units with different
  weights

11
KINDS OF (NON) RESPONSE RATES

• OTHER LITERATURE ON RR/NR DEFINITIONS (AAPOR
  2004)
• COOP Co operation rate
• Proportion of all cases interviewed of all
  eligible units ever contacted
• REF Refusal Rate
• Proportion of all cases in which a respondent
  refuses to do an interview, or breaks-off an
  interview of all potentially eligible cases
• CON Contact Rate
• Proportion of all cases in which a respondent
  refuses to do an interview, or breaks-off an
  interview of all potentially eligible cases

12
KINDS OF (NON) RESPONSE RATES

• OTHER LITERATURE ON RR/NR DEFINITIONS
• OVERALL COMPLETION RATE (Lozar Manfreda, 2003)
• of responses among all sent invitations or
  exposed to invitation/eligible, including partial
  and complete respondents
• The author assumes that most often researchers
  when dealing with RR actually refer to it
  (similar to OECD NR1)
• FULL COMPLETION RATE
• Considers only the of complete (unit)
  respondents
• DROP-OUT RATE
• Referring only to item non response
• CLICK-OUT RATE
• For Web surveys ( of accessing the Web
  questionnaire among all invited)

13
RECOMMENDATION ON (NON) RESPONSE RATES

• Clearly specify which kind of response rate is
  applied
• According to the specific work purposes of this
  paper
• OECD NR1 (or OVERALL COMPLETION RATE)
• is useful in the first steps of the survey design
  to check the robustness of the data collection
  process.
• FULL COMPLETION RATE , DROP OUT RATE, CLICK
  TROUGH are also useful for analysing the DCM

14
DATA COLLECTION MODES TECHNIQUES

• FACE TO FACE
• Historically one of the first applied (since
  1912)
• Mainly for COS
• High quality, High costs
• CAPI/CASI
• Computer Assisted Personal Intervieving/Computer
  Asssisted Self- Interviewing
• Computerised version of face-to face
• MAIL
• The first historically applied (1788 as reported
  by De Leeuw, 2005)
• Implies the self-administration of questionnaire
• Visual approach
• Low costs, relatively low RR

15
DATA COLLECTION MODES TECHNIQUES

• FAX
• Similar to MAIL
• Widely used for BTS
• TELEPHONE
• Useful as supporting measure
• Aural (instead of visual)contact risk of
  loss of context
• Primacy/recency effects (for COS)
• Primary role of interviewer
• CATI (Computer Aided Telephone Interview)
• Widely used for COS, less for BTS (Italy, CB)
• Allows to quickly collect information
• Allows immediate control of inconsistencies
• Expensive

16
Focus CATI AT ISAE for BTS

• Since 1988 for Business (since 2002 completely)
• Since the beginning (1992) for Services

• plus
• Rapidity
• Reliability
• High response rate
• Less sample biases
• Easy substitution

• minus
• Costs (in comparison with postal tecnique)
• Possible subjective biases

17
Focus CATI AT ISAE for BTS
18
DATA COLLECTION MODES TECHNIQUES

• E - MAIL
• Visualapproach, similar to MAIL and FAX
• Decreasing trend
• Low cost
• Confidentiality problems (not allowed at INSEE )
  
• TOUCH-TONE Data Entry/VOICE RECOGNITION Entry
• For COS
• Low RR
• Low cost

19
DATA COLLECTION MODES TECHNIQUES

• WEB/CAWI
• Increasing relevance for BTS (less for COS)
• Low cost
• Coverage errors
• Risk of loss of context in the questionnaire
• Higher Unit NR/Lower Item NR (vs MAIL, Annaud,
  2005, Etter)
• Self selection bias
• Only a part of participants (30-50) freely
  choose this mode (Etter, Stangl, Kershoff)

20
DATA COLLECTION MODES PROCESS
DCM often is a MULTISTAGE PROCESS and for any
step more than one mode can be applied

• STEPS
• (timing of interaction)
• CO OPERATION / PRE SURVEY NOTIFICATION
• OVERALL RESPONSE STAGE
• FULL COMPLETION
• REMINDINGS
• FOLLOW UPs

MODES the different modes can be applied -
CONCURRENT and/or in a - SEQUENTIAL way
21
DATA COLLECTION MODE MIXED MODE (De Leeuw,
2005)

• MULTIMODE SYSTEM
• OPTIMIZE data collection procedures
• REDUCE total survey ERRORS within the available
  time and costs

22
MIXED MODE DATA COLLECTION
EFFECTS ON BURDEN, COSTS, RR

• FIRM
• REDUCES BURDEN
• REDUCES PANEL ATTRITION (TIME IN SAMPLE)
• INCREASES VOLUNTARY PARTICIPATION

INSTITUTION INCREASES RR INCREASES
BURDEN INCREASES COSTS ERRORS DUE TO MODE
EFFECTS
23
SECTOR AND SIZE EFFECTS

• SECTOR
• Firms specific effect is present
• Differences (for both Mail and Internet) (Etter,
  2002)
• Differences (for both CATI and FAX), Italy
• ..but limited
• Retail, Services lower RR (but also smaller
  size)

• SIZE
• Generally inverse correlation between RR and Size
• medium/large firms respond more regularly (Etter,
  2002)
• Burden is greater for small firms (Petroni, 2003)
• Some exception
• South Africa (Kershoff, 2005), Latvia do not
  notice remarkable influence

24
COMMUNICATION METHODS (CO OPERATION STAGE)

• Action useful in getting co operation (and
  reducing NR)
• Tailoring contacts to the characteristics of the
  establishment
• Contacting the right person (mid level
  management)
• Overcoming the lack of awareness (of the Survey /
  Institution)
• Building the initial relationship (personal
  contact advisable)
• Knowledge of survey characteristics
• Survey pre notice

25
COMMUNICATION METHODS INTERVIEWERS (CO
OPERATION STAGE)

• It is strongly advisable
• Use of interviewers
• in the initial step, even if the ongoing DCM is
  different, increases RR (BLS Survey)
• Training of interviewers
• Mainly for BTS, less for COS
• Overcome gate keepers difficult first contact
  does not necessarily mean unwillingness to
  participate

26
COMMUNICATION METHODS (CO OPERATION STAGE)

• all the above mentioned actions are essential
  also for effectively communicating
• COMPULSION (OF RESPONSE)
• as it may increase the (perceived) burden of
  respondent
• (Mixed ) Effects of compulsions
• Increases the Overall Completion Rate (lowers the
  Unit NR)
• May increase the Drop Outs (Item NR)
• Positive effects in France, mainly for large
  firms
• Positive effects in Italy (Retail)

27
COMMUNICATION METHODS (CO OPERATION STAGE)

• Voluntary participation
• most effective approaches
• Sending the business survey to named individuals
  and use telephone follow-up methods to encourage
  responses (Paxon, 1995)
• Lowering the response burden (Linn 2003)
• Increasing the perceived relevance of the survey
  to business goals (Linn, 2003)
• Adopting a Mixed DCM

28
OVERALL RESPONSE STAGE PANEL ATTRITION

• Maintain contacts with participant
• Renewing interest
• Using simple and plain questionnaires
• Avoiding open ended and difficult to answer
  questions
• Question wording
• accuracy, mainly when using concurrent DCMs
• Remindings
• Follow ups

29
CONCLUSIONS AND RECOMMENDATIONS

• Efforts should be made in improving preliminary
  contacts and remindings / follow up actions
• All the modes have plusses and drawbacks
• The starting point of view should be the firm (to
  a lesser extent the consumer) adopting a
  company-centric point of view
• Efforts should be made to allow surveys
  participants to choose the mode they prefer

30
CONCLUSIONS AND RECOMMENDATIONS

• Although Internet (online) seems to be the mode
  of the future, a large part of firms still prefer
  the most traditional mail questionnaire and a
  significant part of respondents prefer fax (habit
  persistence)
• The future appears to be even more bound to a
  mixed data collection mode
• Mixed mode data collection should be carried out
  both concurrently and in a multi stage way (as
  far as possible bound to BTS and COS timeliness)