GROUPING QUESTIONS INTO SECTIONS OF THE SURVEY

1
GROUPING QUESTIONS INTO SECTIONS OF THE SURVEY

• Relevance, ease, interest, answerability.
• Internal logic of questionnaire smooth
  progression through questionnaire.
• Keep questions on same topic together.
• Keep questions with same response options
  together.
• First sections should appear fairly easy
  important (e.g., personal experiences or
  opinions, topics extensively covered in the mass
  media, local or regional issues, etc.).
• Watch for context redundancy effects.

2
MORE ON CONTEXT AND REDUNDANCY EFFECTS

• Context effects occur when answers to one
  question influence an answer to a subsequent
  question. (e.g., How worried are you about
  crime in your neighborhood? followed by Do you
  know of anyone who has been a victim of a crime
  in your neighborhood?)
• Redundancy effects occur when respondents are
  influenced by detailed answers when asked to
  provide a more general answer. (e.g., a series
  of questions about course content, time tabling,
  housing, safety at Nipissing University followed
  by a general or overall evaluation of Nip. U.)

3
GROUPING OF QUESTIONS IN A HYPOTHETICAL CRIME
SURVEY

• Perceptions on crime in general.
• Perceptions on specific types of crime.
• Personal experiences with crime.
• Personal fear about crime.
• Actions taken to protect self, friends, family.
• Perceptions about Criminal Justice System in
  general.
• Perceptions about specific components of the
  Criminal Justice System (courts, policing,
  corrections).
• Demographics.

4
QUESTIONNAIRE LENGTH

• Telephone surveys (10-20 minutes).
• Face-to-face interviews (30 minutes - 60 minutes)
• Mailed surveys (3 - 6 pages)
• Internet surveys (varying lengths depending on
  how many questions per computer screen).
• Major Determinants money resources
  information required expected interest/motivation
  of respondents strengths/limitations of mode
  of administration.

5
MORE ON QUESTIONNAIRE LENGTH

• What motivates respondents?
• Interesting topic questionnaire?
• Important topic?
• Actions resulting from study?
• Questionnaire appears professional, easy,
  straightforward?

6
GUIDELINES FOR MAILED QUESTIONNAIRES.

• Maximum 5-8 pages.
• lower case questions UPPER CASE RESPONSES.
• Number response options.
• Space questions answers.
• Questions horizontal responses vertical.
• Brief, simple instructions when changing topics
  and/or response options.
• Different typefaces for transitions (section
  headings).
• Arrows indicate skip instructions.

7
AVOIDING OTHER FLAWS IN MAILED SURVEYS

• Minimize open-ended questions.
• Avoid complex words.keep them simple
• - Is the word found in everyday usage (e.g.,
  newspapers, TV)?
• - Is there a simpler word that conveys the same
  meaning?
• - If you must use a specialized wordcheck its
  meaning with experts.
• - Pretesting self-administered questionnaire
  allows feedback on ambiguous or difficult
  questions.

8
SELECTING THE SURVEY METHOD.

• THREE MAJOR SURVEY TYPES MAIL, TELEPHONE,
  FACE-TO-FACE.
• INTERNET SURVEYS.RAPIDLY GROWING!
• GROUP-ADMINISTERED SURVEYS. MINOR TYPE OFTEN
  USED IN COMBINATION WITH A MAJOR TYPES.

9
EVALUATING THE PROS AND CONS OF THE 3 MAIN
METHODS

• THREE CATEGORIES OF FACTORS
• 1. ADMINISTRATIVE/RESOURCE FACTORS.
• 2. QUESTIONNAIRE ISSUES.
• 3. DATA-QUALITY ISSUES.
• REFER TO THE HANDOUT..

10

• ADMINISTRATIVE/RESOURCE ISSUES ( HOW MUCH
  TIME MONEY?)
• QUESTIONNAIRE ISSUES (HOW MANY WHAT KIND OF
  QUESTIONS).
• DATA QUALITY ISSUES (RESPONSE RATE, ACCURACY
  COMPLETE ANSWERS).

11

• THE FIRST 3 QUESTIONS WE NEED TO ASK OURSELVES
  ARE
• 1. WHO ARE SURVEY RESPONDENTS?
• 2. IS RESEARCH QUESTION MORE AMENABLE TO ONE
  METHOD?
• 3. HOW MUCH TIME MONEY DO I HAVE?

12
MAIL SURVEYS

• COVER LETTER QUESTIONNAIRE.
• COVER LETTER INCLUDES PURPOSE, SPONSOR,
  INSTRUCTIONS, CONFIDENTIALITY, CONTACT
  INFORMATION.
• THE QUESTIONNAIRE IS TOTALLY SELF-EXPLANATORY.IT
  MUST BE CLEAR SIMPLE!.
• OPTIMUM RESPONSE RATES REQUIRE REMINDER LETTERS
  AND A SECOND QUESTIONNAIRE.

13
ADVANTAGES OF MAIL SURVEYS

• ONE OF THE CHEAPER METHODS.
• MOST COSTS ARE FIXED AND PREDICTABLE.
• CAN CONSULT HOUSEHOLD RECORDS OR VISUAL AIDS.
• GOOD METHOD FOR SENSITIVE TOPICS.
• TIME TO COMPLETION IS PREDICTABLE.
• RESPONSE RATES CAN BE GOOD.

14
DISADVANTAGES OF MAIL SURVEYS

• RESPONSE BIAS.
• QUESTIONNARIE LENGTH APPEARANCE.
• QUESTIONNAIRE MUST BE SELF-EXPLANATORY.
• NO CONTROL OVER ORDER OF ANSWERS.
• NO CONTROL OVER WHO ANSWERS.
• POOR FOR OPEN-ENDED QUESTIONS.

15
ADVANTAGES OF TELEPHONE SURVEYS

• TELEPHONES RANDOM DIGIT DIALING TECHNOLOGY
  PROVIDES EXCELLENT POPULATION COVERAGE.
• DATA QUALITY IS GOOD.ESPECIALLY FOR SURVEYS
  THAT TAKE LESS THAN 30 MINUTES FOCUS ON RECENT
  ATTITUDES OR BEHAVIORS.
• USUALLY INTERMEDIATE IN COST BETWEEN MAIL
  FACE-TO-FACE INTERVIEWS.

16

• INTERVIEWERS, WORK CENTER, COMMUNICATIONS
  MONITORING EQUIPMENT, SAMPLE, SUPERVISERS.
• INTERVIEWERS ARE A MIXED BLESSING.THEY SPEED UP
  SURVEY BUT MUST BE SELECTED, HIRED, TRAINED,
  SUPERVISED.
• LOW SAMPLING FRAME BIAS (WITH RDD).
• GENERAL WILLINGNESS TO BE INTERVIEWED.
• RESPONSE RATES HIGHER THAN MAIL SURVEYS.

17

• USUALLY THE FASTEST SUVEY METHOD.
• INEXPENSIVE SAMPLING FRAMES.
• GOOD INTERVIEWERS ADD VALUE TO RESPONSES.
• CAN USE COMPLEX QUESTIONNAIRE FORMATS.
• GOOD INTERVIEWERS CAN CLARIFY QUESTIONS.
• CONTROL ORDER OF QUESTIONS.
• RAPPORT CAN REDUCE NONRESPONSES AND HELP WITH
  SENSITIVE QUESTIONS.

18
DISADVANTAGES OF TELEPHONE SURVEYS

• MUST USE SHORT, SIMPLE QUESTIONS.
• MUST USE FEW, SHORT, SIMPLE ANSWER CHOICES.
• NO VISUAL AIDS.
• LIMITED CONTROL OF INTERVIEW SITUATION.
• OPEN-ENDED QUESTIONS.

19
FACE-TO FACE SURVEYS

• INTERVIEWS DONE IN PLACE TIME CONVENIENT FOR
  RESPONDENT.
• INTERVIEWS ACCOUNT FOR MUCH OF THE EXPENSE
  ASSOCIATED WITH THIS METHOD.
• PREFERRED METHOD FOR CERTAIN GOALS.

20
FACE-TO FACE SURVEYS ADVANTAGES

• HIGHEST RESPONSE RATE OF ANY METHOD (80 ).
• MORE CONTROL OF INTERVIEW SITUATION.
• RAPPORT PRODUCES TOP-QUALITY DATA.
• COMPLEX, LONG QUESTIONS.
• VISUAL AIDS.
• CONTROL QUESTION ORDER.

21
FACE-TO FACE SURVEYS ADVANTAGES

• BEST FOR OPEN-ENDED QUESTIONS.
• PROBE FOR ANCILLARY INFORMATION.
• MORE TIME IS POSSIBLE.
• CONSULT RECORDS/FILES.

22
FACE-TO FACE SURVEYS DISADVANTAGES

• MOST EXPENSIVE METHOD.
• LONGEST METHOD.
• NEED GOOD INTERVIEWERS TO HANDLE SENSITIVE
  QUESTIONS.
• SOCIALLY DESIREABLE POLITICALLY CORRECT
  RESPONSES.

23
ADVANTAGES ON ONLINE SURVEYS

• LOW COST (Can be costly to collect data with some
  web survey hosts).
• SPEED (can add time deadlines to qualify for
  incentives).
• VARIETY OF QUESTION TYPES CONTENT OPTIONS
• POTENTIALLY HUGE AUDIENCE
• GOOD FOR OPEN-ENDED QUESTIONS
• GOOD FOR SENSITIVE QUESTIONS (if e-mail addresses
  are not recorded).

24
COMBINING SURVEY METHODS

• COMBINATIONS CAN BE USED TO OFFSET DISADVANTAGES.
• COMMON COMBINATIONS ARE MAIL TELEPHONE OR
  INTERNET FACE-TO-FACE SELF-ADMINISTERED
  QUESTIONNAIRE.
• COMBINING SURVEY METHODS DEPENDS ON YOUR RESEARCH
  PROBLEM AND YOUR AVAILABLE RESOURCES.

25
Basic Concepts in Samples and Sampling

• POPULATION
• SAMPLE
• SAMPLING UNIT
• CENSUS
• SAMPLING EFFOR
• SAMPLING FRAME
• SAMPLING FRAME ERROR

26
SAMPLING.

• 1. PROBABILITY NONPROBABILITY SAMPLING.
• 2. WITH PROBABILITY SAMPLING WE KNOW THE
  PROBABILITY OF SELECTION FOR ANY ELEMENT
• IN OUR POPULATION.
• 3. PROBABILITY SAMPLING ALWAYS INVOLVES SOME
  KIND OF RANDOM SELECTION PROCEDURE.

27

• PROBABILITY OF SELECTION (P.O.S.) IS THE
  LIKELIHOOD THAT AN ELEMENT WILL BE SELECTED FROM
  THE POPULATION FOR INCLUSION IN THE SAMPLE.
• WITH A POPULATION CENSUS THE P.O.S. IS 1.0
  AS SAMPLE SIZE DECREASES, SO DOES THE P.O.S.
  INTO THE SAMPLE (YOUR P.O.S INTO A SAMPLE OF
  25,000 CANADIANS IS GREATER THAN YOUR P.O.S INTO
  A SAMPLE OF 100 CANADIANS).
• WITH RANDOM SAMPLING, CASES ARE SELECTED INTO THE
  SAMPLE ONLY ON THE BASIS OF CHANCE.PARADOXICALLY,
  THIS REQUIRES CAREFUL CONTROL OF THE SAMPLING
  PROCESS.

28

• EVEN WITH RANDOM SAMPLING WE MAY END UP
  WITH AN UNREPRESENTATIVE SAMPLE IF WE
  HAVE.
• (1) AN MATERIALLY INADEQUATE OR INCOMPLETE
  SAMPLING FRAME (E.G., A TELEPHONE DIRECTORY
  FROM ANY LARGE CITYUNLISTED PHONE NUMBERS CAN
  RANGE FROM 10 - 30 OF THE POPULATION).
• (2) AN INADEQUATE RESPONSE RATE (E.G., BE
  CAUTIOUS IF YOUR NONRESPONSE RATE EXCEEDS 30 OF
  THE SAMPLE).

29
PROBABILITY SAMPLING

• KNOWN, NON-ZERO CHANCE OF SELECTING EACH
  POPULATION ELEMENT INTO THE SAMPLE.
• NO MAJOR SYSTEMATIC BIAS.ONLY CHANCE
  DETERMINES WHICH ELEMENTS ARE INCLUDED.
• USE WHENEVER A MAJOR RESEARCH GOAL IS TO
  GENERALIZE SURVEY FINDINGS TO A LARGER
  POPULATION.
• IS STILL SUBJECT TO RANDOM SAMPLING ERROR.

30
MORE ON RANDOM SAMPLING ERROR...

• SAMPLE SIZE AND POPULATION HOMOGENEITY
  (POPULATION SAMENESS) AFFECT THE AMOUNT OF
  RANDOM SAMPLING ERROR.THE PROPORTION OF THE
  POPULATION THAT THE SAMPLE REPRESENTS DOES
  NOT!

31
.

• THE LARGER THE SIZE OF THE SAMPLE, THE MORE
  CONFIDENCE WE CAN HAVE IN THE SAMPLES
  REPRESENTATIVENESS ( lt RANDOM SAMPLING ERROR).
• THE MORE HOMOGENOUS OUR POPULATION, THE
  MORE CONFIDENCE WE CAN HAVE IN THE SAMPLES
  REPRESENTATIVENESS (LESS RANDOM SAMPLING
  ERROR).
• SAMPLING FRACTIONS HAVE LITTLE IMPACT ON OUR
  SAMPLES REPRESENTATIVENESS ( OR ON RANDOM
  SAMPLING ERROR).

32

• DIFFERENT TYPES OF PROBABILITY SAMPLES VARY
  IN THEIR RANDOM SAMPLING ERROR.
• 4 MAJOR TYPES
• SIMPLE RANDOM.
• SYSTEMATIC.
• STRATIFIED.
• CLUSTER.

33
SIMPLE RANDOM SAMPLING.

• REQUIRES A PROCEDURE FOR ASSIGNING UNIQUE
  NUMBERS TO ALL ELEMENTS IN THE SAMPLING
  FRAME, AND THEN IDENTIFYING CASES STRICTLY
  ON THE BASIS OF CHANCE
• .BY USING A RANDOM NUMBERS TABLE,
• .OR A COMPUTER PROGRAM THAT GENERATES
  RANDOM NUMBERS.
• OR RANDOM DIGIT DIALING (FOR TELEPHONE
  INTERVIEWS OR WHENEVER AN ADEQUATE SAMPLING
  FRAME IS UNAVAILABLE).

34

• PROBABILITY OF SELECTION IS EQUAL FOR EACH
  ELEMENT.
• IF n 500 and N 17,000, than
  p n/N 500/17,000 .03
• IF n 2000 and N 30,000,000, than
  p n/N 2000/30000000 .00006
• SIMPLE RANDOM SAMPLING IS AN EPSEM METHOD
  (Equal probability of being selected method).

35
(No Transcript)
36
SYSTEMATIC RANDOM SAMPLING

• VARIANT OF SIMPLE RANDOM SAMPLING.THE FIRST
  ELEMENT IS SELECTED RANDOMLY FROM A LIST
  AND THEN EVERY nth ELEMENT IS SELECTED.
• CONVENIENT WHEN ELEMENTS ARE ARRANGED
  SEQUENTIALLY.

37
THREE STAGES IN SYSTEMATIC RANDOM SAMPLING

• TOTAL NUMBER OF CASES DIVIDED BY THE
  DESIRED SAMPLE SIZE PROVIDES YOU WITH YOUR
  SAMPLING INTERVAL ( I N / n)where I is
  your sampling interval N is the population
  size and n is the sample size.
• A NUMBER FROM 1 TO I (YOUR SAMPLING
  INTERVAL) IS SELECTED RANDOMLY.
• AFTER THE FIRST CASE IS SELECTED, EVERY
  Ith CASE IS SELECTED FOR YOUR SAMPLE.

38
(No Transcript)
39

• SYSTEMATIC RANDOM SAMPLING TYPICALLY PROVIDES
  SAMPLES THAT ARE AS GOOD (REPRESENTATIVE)
  AS SIMPLE RANDOM SAMPLES.
• AVOID USING SYSTEMATIC RANDOM SAMPLING IF
  THERE IS SOME UNDERLYING PATTERN OR
  PERIODICITY IN YOUR SAMPLING FRAME.SEE THE
  HANDOUT.
• PERIODICITY IS RARE SO DONT BE PARANOID
  ABOUT IT!

40
STRATIFIED RANDOM SAMPLING

• STRATIFIED RANDOM SAMPLING USES POPULATION
  INFORMATION (E.G., CENSUS DATA) TO MAKE
  SAMPLING MORE EFFICIENT EASY.
• SAMPLING IS EFFICIENT WHEN YOU CAN OBTAIN
  GOOD STATISTICAL ESTIMATES EASIER SAMPLING
  IS EASIER WHEN IT REQUIRES LESS TIME, ,
  OR PRIOR INFORMATION.

41

• ALL ELEMENTS IN THE SAMPLING FRAME ARE
  IDENTIFIED BY STRATA (E.G., GENDER, ETHNICITY,
  RELIGION, ETC.).
• ELEMENTS ARE SELECTED (WITH SIMPLE OR
  SYSTEMATIC RANDOM SAMPLING) FROM WITHIN EACH
  STRATA.

42
(No Transcript)
43

• WHY IS STRATIFIED RANDOM SAMPLING MORE
  EFFICIENT?
• SIMPLE OR SYSTEMATIC RANDOM SAMPLING CAN
  PRODUCE DISPROPORTIONATE SUB-GROUPS IN YOUR
  SAMPLE (SEE THE HANDOUT).
• PROPORTIONATE STRATIFIED SAMPLING CAN
  ELIMINATE THIS SOURCE OF RANDOM SAMPLING
  ERROR.
• DISPROPORTIONATE STRATIFIED SAMPLING ENABLES
  YOU TO DO STATISTICAL ANALYSES WITH
  UNWEIGHTED DATA AND STATISTICAL ESTIMATION
  WITH WEIGHTED DATA.

44

• WHY WOULD YOU WANT TO USE DISPROPORTIONATE
  STRATIFIED SAMPLING?
• TO ENSURE THAT ENOUGH CASES ARE INCLUDED
  IN SMALL STRATA SO THAT MEANINGFUL ANALYSES
  AND COMPARISONS CAN BE PERFORMED.

45
CLUSTER SAMPLING

• REQUIRES LESS PRIOR INFORMATION THAN
  STRATIFIED SAMPLING.
• USEFUL FOR SURVEYS INVOLVING A LARGE,
  DISPERSED POPULATION AND DEVELOPING
  SOCIETIES..WHERE ADEQUATE SAMPLING FRAMES ARE
  HARD TO CONSTRUCT.

46
CLUSTER SAMPLING

• A CLUSTER IS A NATURALLY OCCURRING GROUP OF
  ELEMENTS IN A POPULATION (E.G.,
  UNIVERSITIES, CITY BLOCKS, ETC.)
• EACH ELEMENT APPEARS IN ONE AND ONLY ONE
  CLUSTER.
• DRAWING A CLUSTER SAMPLE IS AT LEAST A 2
  STAGE PROCESS.CAN INVOLVE SEVERAL STAGES
  DEPENDING ON NUMBER OF CLUSTERS USED.

47
CLUSTER SAMPLING

• FOR EXAMPLE.
• FIRST STAGE CLUSTER COULD BE A RANDOM
  SELECTION OF CITY BLOCKS.
• SECOND STAGE A RANDOM SELECTION OF
  HOUSEHOLDS FROM THOSE CITY BLOCKS.
• THIRD STAGE A RANDOMLY SELECTED PERSON FROM
  EACH HOUSEHOLD.

48
(No Transcript)
49
CLUSTER SAMPLING

• AS A RULE, RANDOM SAMPLING ERROR WILL BE
  MINIMIZED, AND PRECISION OF STATISTICS
  MAXIMIZED, IF....
• THE NUMBER OF CLUSTERS SAMPLES IS
  MAXIMIZED.
• THE NUMBER OF ELEMENTS WITHIN EACH CLUSTER
  IS MINIMIZED.
• UNFORTUNATELY, THIS STRATEGY ADDS TO THE
  COST OF CLUSTER SAMPLING.

50
CLUSTER SAMPLING

• SAMPLING ERROR IS USUALLY HIGHEST IN
  CLUSTER SAMPLING COMPARED TO THE OTHER THREE
  PROBABILITY METHODS.
• ERROR INCREASES AS THE NUMBER OF CLUSTERS
  DECREASE.
• ERROR DECREASES AS THE HOMOGENEITY OF CASES
  WITHIN CLUSTERS INCREASES.

51
ONLINE PROBABILTY SAMPLING TECHNIQUES

• SIMPLE, SYSTEMATIC, STRATIFIED, CLUSTER (FOR
  CLOSED POPULATIONS).
• SATURATION (ONLINE CENSUS)
• RANDOM ONLINE INTERCEPT SAMPLING
• INVITATION ONLINE SAMPLING
• ONLINE PANELS SAMPLING (PRE-RECRUITED)

52
ONLINE NONPROBABILTY SAMPLING TECHNIQUES

• CONVENIENCE
• VOLUNTEER
• SNOWBALL OR REFERRAL
• JUDGEMENT

53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
DEVELOPING THE SAMPLING PLAN

• Define the Population
• Define the Sampling Frame
• Define the Sampling Method and Size
• Draw the Sample
• Validate the Sample
• Resample if necessary.

57
SAMPLING DISTRIBUTIONS

• ARE HYPOTHETICAL DISTRIBUTIONS OF A
  STATISTIC (E.G., THE MEAN) ACROSS ALL
  POSSIBLE RANDOM SAMPLES THAT COULD BE DRAWN
  FROM A POPULATION.
• ANY SINGLE RANDOM SAMPLE CAN BE THOUGHT OF
  AS ONE OF AN INFINITE NUMBER OF RANDOM
  SAMPLES THAT THEORETICALLY COULD HAVE BEEN
  SELECTEDIF WE HAD THE MONEY OF BILL GATES
  AND THE PATIENCE OF JOB!

58

• WHAT DOES A SAMPLING DISTRIBUTION LOOK
  LIKE?
• CHECK OUT THE HANDOUT I GAVE YOU!
• SAMPLING DISTRIBUTIONS FOR MOST STATISTICS
  ARE NORMAL OR BELL-SHAPED.
• BECAUSE RANDOM SAMPLING ERROR PRODUCES THE
  BELL-SHAPE, WE CAN ESTIMATE SAMPLING ERROR
  STATISTICALLY!

59
DETERMINING SAMPLE SIZE

• TIME AND MONEY CONSTRAINTS INFLUENCE SAMPLE
  SIZE.
• THE LOWER YOUR SAMPLING ERROR MUST BE, THE
  LARGER YOUR SAMPLE MUST BE.
• THE MORE DIVERSE YOUR POPULATION IS, THE
  LARGER YOUR SAMPLE MUST BE.
• THE MORE COMPLEX YOUR ANALYSIS, THE LARGER
  YOUR SAMPLE MUST BE.
• THE STRONGER YOUR EXPECTED RELATIONSHIPS,
  THE SMALLER YOUR SAMPLE CAN BE.

60
SAMPLE SIZERULES OF THUMB

• LOCAL OR REGIONAL STUDIES.250 - 750.
• PROVINCIAL OR NATIONAL STUDIES1000 - 2500.
• NATIONAL STUDIES WITH MULTIPLE AND COMPLEX
  RESEARCH GOALS..10,000 - 15,000.