Cognitive Interviews For Business surveys

1
Cognitive Interviews For Business surveys

• Marcel Levesque
• QDRC
• Statistics Canada

2
Structure of the presentation

• Cognitive interviewing defined.
• How is an interview constructed?
• How is an interview conducted?
• Cognitive interviewing applied to business
  surveys.
• Examples of issues tested in the UES survey.
• Conclusion.

3
Cognitive interviewing defined

• A process to explore
• a respondents understanding of questions
• their strategies to retrieve the information
  requested by a survey
• their judgment of the adequacy of the retrieved
  information
• their ability to report the required information

4
How is the interview constructed?

• Meeting with survey manager to discuss the
  issues to be tested
• The appropriateness of questions, concepts and
  terminology
• Respondents willingness and ability to respond
  to survey questions
• Respondents understanding of the questions and
  what to report

5

• Respondents use of external sources of
  information such as financial or administrative
  records and their need to consult other
  individuals who can provide the information
  requested
• The compatibility of questions and response
  categories with respondents record keeping
  practices
• Difficulties respondents may encounter in
  retrieving information and completing the
  questionnaire

6
Designing an interview guide

• Based on the issues discussed with the survey
  manager, a guide is designed to test the
  questions and issues identified.
• Comprehension, information retrieval, judgment
  and communication are usually issues to be
  explored.

7
Comprehension

• Difficulties encountered by a respondent in
  understanding a question may be explored
• Instruction content - inaccurate or conflicting
  instructions, instructions separate
  from the question
• Question wording - ambiguous terms, critical
  definitions missing

8

• Question structure several questions in
  one, Question /answer mismatch
• Question content - complex topic, topic
  carried over to another section
• Navigational instructions - inaccurate
  instructions (move to the
  wrong place), confusing flow

9
Information retrieval

• Difficulties encountered by a respondent in
  gathering the requested information may be
  explored
• Multiple sources - information may be
  distributed in multiple sections
  /departments in the organization
• Record retrieval - records in multiple sources,
• record access
  issues

10

• Memory retrieval- reference period too long,
  recall problems and telescoping errors.

11
Judgment

• Difficulties encountered by a respondent in
  assessing the relevance of the retrieved
  information to the data requested may be
  explored
• Data incompatible with existing records
• Need to evaluate and synthesize multiple sources
  of information
• Potentially sensitive information, strategic or
  proprietary issues may prevent revealing certain
  information

12
Communication

• Difficulties encountered by a respondent in
  reporting the response to a question may be
  explored
• Missing response categories
• Mismatch with technical language
• Response terminology critical definitions may
  be missing

13
How is the interview conducted?

• Respondents for cognitive interviews are usually
  recruited from Statistics Canadas Business
  Register or from another frame of establishments
  provided by the industry-specific program areas.
• Respondents are identified based on the project
  teams specifications, designed to ensure a
  selection of business establishments for each
  industry sector to be tested.

14

• Appointments are made with the person within the
  business establishment who usually completes
  questionnaires for Statistics Canada.
• The ideal respondent is the person in the
  business who is most knowledgeable about the data
  requested, who has access to this data and who
  also has the authority to release it.

15

• Respondents are provided with an explanation of
  the purpose of the cognitive interview with
  emphasis on the fact that Statistics Canada is
  consulting with them to obtain their feedback and
  opinions.
• The importance of their participation is
  emphasized and they are given assurances of
  privacy.

16

• A consultant from the QDRC conducts a one-on-one
  interview at the respondents place of business
  and observers from the client industry sector
  take notes during the interview.
• The interviewer walks through the questionnaire
  with the respondent and discusses issues as they
  arise.
• The interview may be recorded with the
  respondents acceptance.

17
Cognitive interviewing applied to business surveys

• The Unified Enterprise Survey (UES) began in 1997
  with a pilot of a small number of industries and
  has continued to the present time where it now
  encompasses the majority of business surveys
  conducted by Statistics Canada.
• The UES is basically a single questionnaire with
  a variant for each industry sector. There is a
  set of core questions asked of every industry and
  industry specific questions for each different
  sector.

18

• Core questions cover such topics as the reference
  period covered by the survey (financial year of
  the establishment), revenues and expenses,
  personnel, sales according to types of clients
  and their location and, if appropriate, questions
  on international transactions.
• Industry specific questions refer to the main
  commercial activity of the establishment and
  industry characteristics, namely sales activity
  specific to the sector.

19

• The purpose behind UES is to develop a common
  approach with concepts harmonized across all
  industries to
• facilitate data collection and data capture
• reduce and better manage response burden
• to increase response and data quality.

20
Examples of issues tested in the UES

• Comprehension Question structure
• One of the difficulties encountered with the
  2005 survey was in relation to a mismatch between
  the question and one of the answer categories.

21
F- Industry characteristics automotive repairs
and maintenance

• Sales of goods and services
• Repairs and maintenance labour costs
• ________________________________________
• Repairs and maintenance- parts and accessories
• used for repairs and services ___________________
  ____________________
• Sales of merchandise, parts and accessories for
• resale in their original state
• ________________________________________

22

• This section referred to sales of goods and
  services, but the first answer category asked to
  record labour costs. Consequently, respondents
  were confused.
• The intention behind this question was to obtain
  the sales value of labour costs billed to the
  client but respondents failed to understand the
  question in this way. The difficulty here was a
  mismatch between question and answer, which led
  to confusion.

23
Recommendation

• Replace the expression
• labour costs
• by
• sales resulting from labour costs billed to
  clients.

24
Comprehension Question wording

• Another difficulty encountered was in relation to
  the question wording in the French version of the
  questionnaire, resulting from the translation
  from English to French.

25
E- expenses

• Cost of employment and personnel agencies
• ___________________________________
• Coûts des bureaux de placements et des
• services de location de personnel
  ______________________________

26

• The expression personnel agencies translated
  into services de location de personnel did not
  make sense to the French respondents since the
  word location in French means rental or
  leasing.
• Respondents did not understand the meaning of
  the expression services de location de
  personnel. As a result, many respondents did not
  know what costs to report in this column.

27
Recommendation

• Since bureaux de placement was a more common
  expression
• leave out the reference to services de location
  de personnel.
• include only this category in the question

28
Retrieval multiple sources and record retrieval

• 2006 survey on entertainment and leisure
• Some respondents were involved in the health
  fitness industry offering a variety of services
  such as a spa, massage, tennis courts, pool and
  exercise rooms. Some of the expense items listed
  in the questionnaire did not correspond to their
  books.

29
E- expenses

• Wages and salaries
• ________________________________
• 5. Contract workers ___________________________

30

• Item 1 in the questionnaire asked for salaries in
  Canadian dollars.
• Item 5 asked for the dollar value attributed to
  contract workers.
• Some employees are hired on contract but the
  costs are listed under salaries in item 1.
• Respondents would not have reported any dollar
  value in item 5 for contract workers and there
  was no way for them to indicate that these
  expenses were all under item 1 on salaries.
  Consequently we would have ended up with an
  unanswered question without knowing why.

31

• Since expenses are part of the core questions
  applicable to all industry sectors, it is quite
  difficult to have answer categories correspond to
  all industry sectors.
• RecommendationAdd a space for comments under
  the section on expenses to allow respondents the
  possibility of explaining the items they are
  unable to provide as listed.

32
Judgment

• Potentially sensitive information, strategic or
  proprietary issues may prevent respondents from
  revealing certain information.
• 2005 survey on service industries
• Some respondents involved in real estate,
  leasing and property management said they would
  not provide information on revenues because their
  company had a policy on confidentiality which
  would not allow the release of such information.
  They did, however, mention that they would be
  willing to provide estimates.

33
Recommendation

• Provide revenue categories to encourage
  respondents to report information on revenues
  they would not give if exact figures were
  requested.

34
Communication

• One of the difficulties encountered in the
  communication phase of the process for the 2006
  Service industry survey was related to response
  categories.

35
F- Sales according to product type

• Architectural products
• Residential construction projects
• ______________________________
• 2. Office construction projects
  ______________________________
• 3. Construction projects for retail
• businesses and restaurants
• ______________________________

36

• 4. Construction projects for hotels
• _____________________________
• 5. Construction projects for hospitals
  ______________________________
• 6. Construction projects for educational
  institutions
• ______________________________
• 7. Construction projects for industrial
• buildings
• _______________________________

37

• Where would you report construction projects
  for airports?

38
Recommendation

• Add significant categories identified by
  respondents i.e. construction projects for
  airports.

39
Conclusion

• In a business environment, a respondents main
  preoccupation is the success of his business.
• Few respondents will consider spending time
  researching information if it is not readily
  available or spending time trying to understand
  questions that may not be clear.

40

• As indicated in this presentation, many issues
  may contribute to response burden and have a
  negative impact on the quality of the results of
  a survey.
• Cognitive interviewing
• provides us with a greater understanding of some
  of these issues
• allows us to improve a questionnaire
• allows us to increase the clarity of the
  questions, reduce response burden and result in
  increased data quality.