Conducting Statewide Customer Satisfaction Surveys

1
Conducting Statewide Customer Satisfaction Surveys

• Brad Sickles
• USDOL / ETA
• ETA Performance Accountability Team

2
Measuring Customer Satisfaction

• Participant Customer Satisfaction
• Employer Customer Satisfaction

Customer ratings on each of the three core
questions regarding satisfaction are weighted to
account for regional differences and reported on
a 1 to 100 scale. Computation of the ACSI is
discussed in TEGL No. 6-00. The American
Customer Satisfaction Index (ACSI) is used to
report statewide results to the Department.
3
The Core ACSI Questions
1. Utilizing a scale of 1 to 10 where 1 means
Very Dissatisfied and 10 means Very
Satisfied what is your overall satisfaction with
the services provided from ________? 2.
Considering all of the expectations you may have
had about the services, to what extent have the
services met your expectations? 1 now means
Falls Short of Your Expectations and 10 means
Exceeds Your Expectations. 3. Now think of the
ideal program for people in your circumstances.
How well do you think the services you received
compare with the ideal set of services? 1 now
means Not very close to the Ideal and 10"
means Very Close to the Ideal.
4
Participant Customer Satisfaction

• The weighted average of participant ratings on
  each of the three questions regarding overall
  satisfaction reported on a 1 to 100 scale.
• Who Exiters
• When 1to 60 days after date of hard exit or
  date a soft exit was determined
• How Telephone survey
• Required completions 500 statewide
• Required response rate 50 first year 70
  following years

5
Employer Customer Satisfaction

• The weighted average of employer ratings on each
  of the three questions regarding overall
  satisfaction reported on a 1 to 100 scale.
• Who Employers who receive a substantial
  service.
• When 1to 60 days after date of the completion
  of a service. For employers who listed a job
  order where no referrals were made, contact
  should occur 30 to 60 days after a job order was
  listed.
• How Telephone survey
• Required completions 500 statewide
• Required response rate 50 first year 70
  following years

6
(No Transcript)
7
Capturing Customer Satisfaction Results to Assess
Local WIBs

• States may use the ACSI questions and approach to
  measure satisfaction at the local level
• States may also develop their own customer
  satisfaction instruments and approach for
  assessing local WIBs
• Approaches must be applied
  uniformly and consistently
  to all
  WIBs
• TEGL 6-00 outlines parameters
  for collecting
  customer satisfaction results

8
Revisions to TEGL No. 6-00
Surveys to Obtain State Level Results

• Response rates 50 for the first year of data
  collection 70 for following years.
• Response levels below the specified minimums will
  invalidate the results.
• States are required to determine the appropriate
  sample sizes. States are no longer required to
  draw minimum sample sizes of 1,000.
• Respondents must be told responding to the survey
  is voluntary and the information will be kept
  confidential.
• States must create a process for creating an
  up-to-date customer list, capturing information
  on each customers address and telephone number.

9
Revisions to TEGL No. 6-00
Surveys to Obtain State Level Results

• Disregard earlier guidance that stated the
  sampling frame only included participants and
  employers eligible for the surveys who had valid
  contact information.
• A survey is complete when valid answers are
  provided by the respondents for each of the core
  ACSI questions. Valid answers are responses 1
  through 10.
• The calculation of the ACSI score includes only
  the results from complete surveys.
• The introductory statements for both the
  participant survey and the employer survey were
  revised to better set the context for the
  interviews.

10
Revisions to TEGL No. 6-00
Surveys to Obtain State Level Results Additions
to Ensure Adequate Response Rates

• Local program staff must inform the customer
  during the registration process about the
  importance of satisfying customers and the
  possibility of being contacted for information on
  his or her experience with the services.
• The interview should be limited to 15 minutes or
  less.
• A minimum of five follow-up attempts is
  required, involving various times of the day
  before closing the record.

11
Revisions to TEGL No. 6-00
Surveys to Obtain State Level Results Additions
to Ensure Adequate Response Rates

• Local programs should collect alternate contact
  information.
• A letter in advance of the survey should be sent
  out informing the customer that he or she can
  expect to be contacted about his or her
  satisfaction with the services.

12
Creating Questions to Supplement the ACSI
Customer Satisfaction Not the Same as Customer
Service

• Customer service is measured by standards set by
  you or your agency.
• Customer satisfaction is measured
  by the customers standards for
  the services, regardless of
  whether they make
  sense.

13
Drivers of Satisfaction
Customer Service Characteristic
Service Quality Measures

• Convenience
• Accessibility
• Ease of use
• Timeliness
• Safety
• Reliability
• Accuracy
• Thoroughness
• Fairness
• Appropriateness
• Attractiveness and cleanliness

• Courteous
• Professional
• Attentive
• Friendly
• Helpful
• Knowledgeable
• Prompt
• Informative
• Honest
• Candid

14
Characteristics of Firms With High Customer
Satisfaction

• Customers define quality
• Variety of services
• Customization of services
• Convenience of services
• Timeliness of services
• Continually identify factors that influence
  satisfaction
• Continually identify expectations and set
  customer service standards well above these
  expectations (to delight customers)

15
Writing Effective Questions

• Remember your survey's purpose to
  provide you feedback on the
  level of satisfaction.
• If you have doubts about a question,
  discard it.
• State your questions simply.
• Stay focused when writing questions avoid vague
  areas.
• If a question can be misinterpreted by the
  respondent, chances are that it will be.
• Include only one subject per question.
• Avoid questions that lead respondents.
• Consider optional ways to ask questions that deal
  with sensitive areas.

16
Writing Effective Questions

• Create questionnaire items to determine the key
  drivers of satisfaction

17
Constructing Effective Response Items

• Answer options need to be mutually
  exclusive and exhaustive.
• Keep open-ended questions to a
  minimum.
• Respondents interpret time-oriented
  responses differently, such as
  never, sometimes,
  and always.
• Consider a "don't know" responses when you create
  answer sets.
• Create meaningful scales for rating services.

18
Ordering Questions

• The first series of questions should be easy for
  the respondent to understand and should capture
  his or her attention and interest.
• Start with general questions and gradually shift
  to more specific questions.
• Group questions in logical sections and
  sequence sections or questions in a well
  thought out order.
• Introduce each new section with a
  summary statement so that
  participants have an
  opportunity to adjust to the
  new set of questions.

19
Ordering Questions

• Position questions about personal or sensitive
  issues toward the end of the questionnaire.
• The order of similar items on a list can bias
  results. Randomly or alphabetically order items
  and indicate in the instructions how they are
  ordered, reducing the likelihood that respondents
  will see the first items as most important of the
  group.
• Put demographic questions at the end of the
  survey form.
• Seek to minimize the number of times the a skip
  rule is followed as confusion can result with
  too many skips or when skips become overly
  complicated.

20
Resources

• Fowler, F. J., Jr. Survey Research Methods (2nd
  ed.). Thousand Oaks, CA Sage Publications, Inc.,
  1993.
• Hayes, Bob E. Measuring Customer Satisfaction
  Development and Use of Questionnaires.
  Milwaukee ASQC Quality Press,1993.
• Payne, S. L. The Art of Asking Questions.
  Princeton, NJ Princeton University Press, 1951.
• Schuman, H and Presser, S. Questions and Answers
  in Attitude Surveys Experiments in Question
  Form, Wording, and Context. New York Academic
  Press,1981.
• Simply Better! The Voice of the Customer, Second
  Edition. Washington U.S. Department of Labor,
  Employment and Training Administration,
  1998.
• Sudman, S. and Bradburn, N. Asking Questions A
  Practical Guide to Questionnaire Design. San
  Francisco, CA Jossey-Bass Inc., 1982
• Vavra, T. G. Improving Your Measurement of
  Customer Satisfaction. Milwaukee ASQC Quality
  Press, 1997.