University of Florida Survey Research Center

1
University of Florida Survey Research Center
2
Bureau of Economic and Business Research (BEBR)

• Established in 1929
• Population Program (State estimates and
  Projections)
• Publications (Florida Statistical Abstract)
• Economic Analysis (Ad hoc studies, such as
  Florida Price of Living Index)
• University of Florida Survey Research Center
  (Established 1979)

3
Some of Our Clients
4
(No Transcript)
5
(No Transcript)
6
Selected Health Related Surveys
Survey Population Sample Type Completed Interviews Number of Attempts Total Sample Released Total Number of Calls Response Rate Refusal Rate of Refusals Converted
BRFSS Cell Phone Survey FL Cell Phone users RDD Cell Phone 300 12 7,200 24,569 0.32 0.22 3.00
College of PHHP--CAHPS Eval. Medicaid for Reform Implementation Plan Children enrolled in Medicaid from Broward, Duval, Baker, Clay and Nassau counties Listed 3091 21 14,695 68,006 0.38 0.07 14.43
FL DCF --Eval. substance abuse programs Adults who received substance abuse treatment from state Listed 422 13 3,357 13,188 0.62 0.08 6.45
FL AHCA--Report Card Children enrolled in a Medicaid HMO Listed 3714 20 18,397 72,553 0.72 0.06 16.85
FL DOH--Eval. anti-tobacco program FL children (11-18 yrs old) Listed 2205 15 20,000 94,521 0.37 0.07 2.52
FL ICHP--FL KidCare eval. Children enrolled in FL KidCare Listed 1714 31 7,859 70,600 0.57 0.15 9.64
FL ICHP--Eval. health care in Partners in Care Children enrolled in PIC(terminally ill children) Listed 140 33 352 2,904 0.76 0.12 23.75
TX STARPLUS--Eval. elderly Medicaid enrollees with disabilities Adults enrolled in STARPLUS Listed 955 34 9,128 47,618 0.26 0.14 13.80
FL DCF --Eval. substance abuse programs Adults who received substance abuse treatment from state Listed 422 11 3,953 14,507 0.66 0.07 7.69
FL AHCA--Report Card Children enrolled in a Medicaid HMO Listed 4004 20 22,209 90,889 0.66 0.12 15.08
TX STAR--Medicaid HMO health care eval. Adults in a Medicaid HMO under TX STAR Listed 2255 50 20,133 173,000 0.77 0.05 11.65
7
Organizational Chart
8
Breakdown of OPS Staff

• 11 Supervisors (13.00/hr)
• 10 Assistant Supervisors/Interviewers (9.00)
• 15 Refusal Convertors (9.50/hr)
• 25 Spanish Speaking Interviewers (9.00/hr)
• 100-150 General Interviewers (8.50/hr)
• Bonus based on productivity for interviewers and
  listening to recordings for supervisors

9
Refusal convertors

• Our most skilled interviewers
• Higher pay, higher productivity standard
• Allowed to call our most difficult cases
• Promotion process
• Minimum of 208 hours of interviewing
• Maintained higher productivity standards
  consistently for last 40 hours
• No penalty boxes in last two months

10
Shift Staffing

• 1 Shift Supervisor
• 0-4 Assistant Supervisors
• 0-5 Mentors

11
Hours of Operation

• Mon-Friday 9am-12am
• Saturday 12pm-9pm
• Sunday 3pm-12am

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
FACILITIES
20
Recently expanded location
21
Dedicated network backbone
22
Security

• Building security guards and alarm system
• Dedicated on-site servers kept in a locked room
• Off-site backups to prevent data loss
• Virus and malware detection
• All network infrastructure protected with
  filtered power and battery backup
• Unauthorized access detected 24 hours a day,
  seven days a week by IT staff

23
Telephone system

• 93 telephone interviewing stations
• 69 outgoing lines with VOIP overflow option
• Extra server in case first server fails
• Asterisk VOIP software
• Reducing background noise
• Headsets
• Noise absorbent dividers and ceiling

24
OTHER SERVICES

• Focus Groups
• Data entry
• Transcription

25
Software

• Wincati Sample management software (Sawtooth)
• Ci3 Questionnaire authoring software (Sawtooth)
• Scheduler Lab management software

26
WINCATISample Management Software
27
Survey Specifications
28
Example Record
29
Disposition Report
30
Response Rate

• Proportion of completed interviews out of all
  eligible respondents
• What is eligible?
• Listed sample -- All cases may be eligible
• RDD sample There are rules to determine which
  cases to count

31
AAPOR RR1

• I Complete interview
• P Partial interview
• R Refusal and break-off
• NC Non-contact
• O Other
• UH Unknown if household/occupied housing unit
• UO Unknown, other

32
Monthly survey response rates by county
33
Ci3 Questionnaire Authoring Software
34
Lab Management and Quality Control
35
Scheduler
36
Interviewers receive feedback
37
Reviewing recorded calls
38
Reviewing call recordings

• Automatically select random calls targeted at
  mistakes and policy violations
• Supervisors listen to and evaluate calls
• Determine whether interviewer correctly described
  the outcome of the call
• Check for policy violations

39
Real-time call monitoring

• Assistant supervisors listen to calls during a
  shift
• Evaluate six skills verbatim, probing,
  clarifying, feedback, pace, and voice
• Meet with interviewer during same shift to
  discuss their performance

40
Real-time call monitoring
41
Miscoding Penalty Box

• Receive penalty box for one call with low rating
  on call skills
• Can also receive penalty boxes for frequently
  choosing the wrong description of a calls
  outcome
• May be terminated for third penalty box

42
Productivity
43
Top performer
44
This interviewer needs help
45
Measuring employee productivity

• Compare completed interviews per hour to average
  for survey
• Average across time spent on each survey
• Interviewer score calculated over large 45-day
  window to smooth out bad days
• Survey average calculated over smaller 15-day
  window for higher accuracy

46
Rewarding top performers

• Productivity incentive for keeping score above
  100
• Incentive proportional to hours worked and how
  far above 100
• Some top performers earn over 600 in
  productivity incentives for a 2-month incentive
  period

47
Penalty box

• Required to keep productivity score above
• 84 for general interviewers
• 91 for spanish-speaking interviewers
• 107 for refusal convertors
• 117 for spanish-speaking refusal convertors
• Hold most skilled to higher standard
• Meet with supervisor after second box
• Terminated or demoted after third box

48
How Much Does A Survey Cost?
49
Negotiating a survey

• Estimate productivity per interviewer hour
• We try to get 26 per interviewer hour
• A survey where we complete one interview per hour
  costs 26/complete
• A survey where we complete one interview every
  two hours costs 52/complete

50
Relation between hours worked and cost per hour
51
Survey Cost Structure

• Major factors that influence Survey Costs
  include
• Population
• Mode
• Length
• Sample
• Subject/Salience

52
Survey Mode Characteristics
53
2010 Surveys by Mode
54
Results from AASRO survey
55
Effect of Population on Cost

• How easy is it to identify, locate and/or select
  the group being studied?
• Survey of the Elderly (Elder) Statewide study of
  Florida Seniors Selection screening
  simple 38/complete
• Oral Cancer Awareness (Port9) Oversamples
  African Americans, age 25 in selected
  countiesSelection screening challenging
• 55/complete

56
Effect of Length on Cost

• Generally, a shorter survey lower costs
• Survey of Star Plus program members (Star)
• 36 min 53/complete
• Consumer Confidence Index (CCI)
• 11 min 26/complete

57
Effect of Mode on cost

• Type of survey telephone, mailout, etc.--
  affects survey cost.
• Dept. of Fine Arts Evaluation Web survey
  10/complete
• FL Dept of Transportation Survey of Drivers
• Telephone 28/complete

58
Effect of Subject/Salience on Cost

• What is the survey about? How important is it to
  the respondent? Surveys with threatening
  questions (disability, alcohol abuse, etc.) may
  increase costs.Florida BRFSSSubject Health
  practices careInc. threatening
  questions 75/complete CAHPSSubject
  Health careNo threatening questions 23.50/compl
  ete

59
Effect of Sample on Cost

• Sample type (Random Digit Dial vs. Listed
  sample) and quality can substantially affect
  costs.
• HMO Report Card Survey (HMO)
• Listed sample 3.2 recs/comp.
• Consumer Confidence Index (CCI)
• Random Digit Dial 10 recs/comp.

60
Sampling
61
Telephone survey sampling schemes

• RDD random digit dialing
• Numbers are created using information about
  existing area codes (GENESYS)
• Best for obtaining a representative sample
• Less productive than listed sample

62
SAMPLING SCHEMES

• Targeted RDD sample
• Overlay U.S. census tracts with working telephone
  banks
• Targets specific subgroups
• Regions
• Ethnic groups
• Levels of income

63
SAMPLING SCHEMES

• Predetermined lists
• Efficient
• Cost effective
• Quality depends on the frequency of updates

64
Sample size estimate
65
Do Not Call Lists

• National Do Not Call Registry (www.donotcall.gov)
• Only calls that solicit the sales of goods or
  services are covered
• Surveys and charities are exempt
• Respondents usually do not know that

66
Common sources of lists

• Telephone numbers and households listed in
  telephone directory
• Can pull national sample
• Unlisted numbers vary a lot by geographic area
  and respondent characteristics
• Drivers licenses from state Department of Motor
  Vehicles
• Must select samples by state, and states vary in
  laws regarding drivers licenses
• Data may be old as people move without informing
  Department of Motor Vehicles
• Nor every one drives and there are biases (old
  and young, people in urban settings with public
  transportation and high insurance costs)

67
Common sources of lists (continued)

• Voter Registration
• Potentially more updated than drivers license
  database
• Not everyone votes potentially very biased
  unless survey concerns potential voters
• Lists from behavioral surveys and credit card
  evaluation
• Usually expensive
• Can often select people with particular
  characteristics (e.g. smokers)
• Potentially biased based on source
• Member and User Lists such as patient records,
  HMO membership, recipients of Temporary
  Assistance for Needy Families (TANF)
• Source is often variable in maintaining records
  (e.g. HMOs do not have common database practices
  for recording membership data)

68
Telephone sample options and costs

• 300 minimum on all orders
• Random Digit Dialing (RDD) uncleaned
  .04/record
• RDD with business purging from yellow pages
  .05/record
• RDD with business purge and attended dialing
  using automated detection .09/record
• Claritas Modeled data
• Experian Behavior Bank .35/record

69
ExampleConsumer Confidence Survey
70
What is Consumer Confidence?

• Telephone survey based measure calculated each
  month from the responses of approximately 500
  Florida households
• Designed to predict consumer spending which
  accounts for 70 of Gross Domestic Product (GDP)
• Barometer of public opinion about economy

71
How is consumer confidence measured?

• Consumer Sentiment from the University of
  Michigan

• Consumer Confidence from the Conference Board

• 500 Random Digit Dial telephone interviews a
  month across the U.S.
• Preliminary release second Friday, final release
  last Friday

• Approximately 3,500 mail out surveys each month
  across the U.S.
• Released last Tuesday

The University of Florida Survey Research Center
uses the University of Michigan method for
Florida, but releases the results the last
Tuesday of each month
72
Comparison of Index Questions
University of Michigan Index of Consumer Sentiment Conference Board Consumer Confidence Index
Current Conditions Current Conditions
Overall Buying Conditions Business Conditions in Area
Current Personal Financial Condition Current Job Availability in Area
Future Expectations Future Expectations
Business Conditions Over Next 12 Months Business Conditions in 6 Months
Business Conditions Over Next 5 Years Job Availability in Area in 6 Months
Personal Financial Condition in 12 Months Total Family Income in 6 Months
73
Comparison of U.S. Consumer Sentiment and
Consumer Confidence
74
Comparison of Florida and U.S. Consumer Sentiment
75
What are some drivers of Consumer Confidence?

• Economic Shocks
• Prices
• Stocks
• Employment

76
Our Survey Design

• Approximately 5,000 Random Digit Dial phone
  numbers released each month
• Phone numbers are released proportionate to
  households by county
• A preliminary consumer confidence release is made
  the last Tuesday of each month with at least 400
  completed interviews
• By the end of the month we collect at least 500
  interviews
• Faculty can add questions to our monthly survey

77
News sources that have used our data