Laurie Roshak

1
Evaluating Globally How to Conduct Remote
International Usability Research
Laurie Roshak HCI Engineer Sun Microsystems
Rocky Mountain CHI 23 Jan 2003
2
Agenda

• Overview
• Demonstration
• Remote, moderated study
• Remote, unmoderated study
• Lessons Learned
• Remote studies
• International Studies
• QA

3
Overview

• Why conduct international studies?
• Products are used globally
• Need to understand cultural differences
• Need to ensure we meet the needs of these global
  users

4
Overview (cont.)

• Why international studies typically are not
  conducted
• Aggressive development schedules
• Limited budget (travel, translations,
  consultants, etc.)
• Limited knowledge on how to conduct international
  user research
• Identifying the target languages, locations, and
  cultures for evaluation
• Recruiting scheduling participants in multiple
  time zones
• Technical limitations for global implementation
  such as language limitations in the browsers
• Some methods do not work for some cultures

5
Overview (cont.)

• Means of conducting international studies
• Establishing geographic usability research groups
• Traveling to conduct research
• Hiring consultants
• Conducting moderated and unmoderated remote
  research

6
Demonstration Remote, Unmoderated

• Objective
• Obtain internal feedback on Sun's global site
  strategy
• Pilot test materials to be used for an upcoming
  usability evaluations with customers and
  partners.

7
Demonstration Remote, Unmoderated

• Participants
• 20 business and technical Sun employees
• From Brazil, Canada, China, France, Germany,
  Italy, South Africa, Spain, and United Kingdom

8
Demonstration Remote, Unmoderated

• Study Environment
• Solaris or Windows computer with Internet
  connection
• HTML prototype
• Participant materials in English
• Survey tool
• Data logging software

9
Demonstration Remote, Unmoderated

• Procedure
• Invitation and instructions to participate were
  distributed to select locations
• Conducted in English
• Conducted anytime, anywhere
• Individual sessions

10
Demonstration Remote, Moderated

• Objective
• Obtain customer feedback on Sun's global site
  strategy

11
Demonstration Remote, Moderated

• Participants
• 28 business and technical customers
• From Brazil, Canada, China, France, Germany,
  Italy, South Africa, Spain, and United Kingdom

12
Demonstration Remote, Moderated

• Study Environment
• Solaris or Windows computer with Internet
  connection
• Telephone
• JSP prototype on external server
• Participant materials translated into
  Chinese-Simplified, English, French-Canadian,
  French-European, German, Italian,
  Portuguese-Brazilian, and Spanish-Iberian
• Internet browser (Netscape or IE)
• Application sharing software (SunForum for UNIX,
  NetMeeting for Windows)
• Data logging software

13
Demonstration Remote, Moderated

• Procedure
• Participants pre-recruited
• Conducted in participant's preferred language
• Translator used if the participant's preferred
  language differed from the moderator's language
• Conducted at the participant's preferred location
  (own office vs. Sun facility)
• Individual sessions
• Instructions distributed to local contact

14
Lessons Learned

• International remote usability studies is a cost
  effective and timely way to obtain global
  feedback
• No travel costs incurred
• No travel time incurred
• Individual can conduct study in multiple
  locations in a single day

15
Lessons Learned

• Get buy-in from all stakeholders (e.g. local
  contacts, technical advisors, recipients of study
  results) at the beginning of the study
• Research objectives
• Schedule
• Participant criteria
• Issues and dependencies
• Ensure study environment will support the
  prototype (e.g. jsp, encodings)

16
Lessons Learned

• Logistics are more complex
• Detailed project documentation is essential
• Regular status meetings helped keep project on
  track and provided a forum for discussing issues
  that needed more than email exchanges
• Provide detailed instruction to the moderator(s),
  local country translators, and participants
  helped minimize confusion

17
Lessons Learned International Research

• International research requires more resources
  and takes longer than local research
• Use a tool and system environment that will
  support all required languages. If not available,
  outsource the work.
• Determine language encoding at beginning of
  project

18
Lessons Learned International Research

• Translations took longer than expected
• Vendor issues
• National holidays
• Time zone delays
• Do not translate participant materials until
  complete country pilots
• Prototype changes
• Rewording material to accommodate cultural
  differences
• Add or delete material

19
Lessons Learned International Research

• Recruiting took longer than expected
• Lack of customer leads
• People out of the office for national holidays,
  training, illness, etc.
• Country contacts has other obligations with
  higher priority
• Delays in responses due to time zone differences
• Customer in midst of critical path
• Project team needed to rely on others to do the
  recruiting due to language issues

20
Lessons Learned International Research

• Establish company-wide recruiting process and
  supporting tools
• Reduce recruiting cycle
• Avoid multiple groups from contacting same
  customer
• Use tools (e.g. timeanddate.com) to ensure you
  have the correct time and date
• Holidays vary by location
• Time zone change may result in different date as
  well as different time
• Daylight savings time switches are not the same
  in all locations
• Some time zones are 30 min. differences

21
Lessons Learned International Research

• Include the time and date for all applicable
  locations in your communications
• Exporting gifts is costly and time consuming
• Distribute gifts ASAP (at least 6 weeks prior to
  conducting study) to ensure they clear customs in
  time
• Have supply of gifts on hand to eliminate time
  required to order gifts
• Consider have local contact purchase and
  distribute gifts
• Work with local contact to identify appropriate
  gifts

22
Lessons Remote Studies

• Advantages of unmoderated studies
• Number of participants not limited by the
  availability of the moderator
• Avoid hassle of scheduling participants,
  especially in multiple time zones
• No time lost with "no shows"
• Participation not limited by the language of the
  moderator/availability of translator, only by the
  availability of translated material

23
Lessons Remote Studies

• Advantages of moderated studies
• Can redirect participants
• Can ask follow-up/clarification questions in real
  time
• Can diffuse participant's frustration
• Can troubleshoot problems in real time

24
Lessons Remote Studies

• Need collaboration and data logging tools that
  are
• Hardware/software independent
• Works with firewalls
• Does not require the participant to install or
  configure
• Study material need to be online for remote
  participants, so unable to use some discount
  usability methods

25
Lessons Remote Studies

• Environment variables (participant's location,
  platform, network) can make setup and
  troubleshooting difficult
• Need to teach most users how to install,
  configure, and use collaboration software
• Difficult to obtain the IP address when on a LAN
  typically need to set up for port forwarding
  which many users cannot or will not do

26
CHI 2003 Panel

• Evaluating Globally How to Conduct International
  or Intercultural Usability Research
• Vanessa Evers, University of Amsterdam
• Rolf Molich, DialogDesign
• Colleen Page, Microsoft
• Ann-Byrd Platt, Swisscom
• Laurie Roshak, Sun Microsystems
• Jared Spool, User Interface Engineering

27
Thank You!
laurie.roshak_at_sun.com 303.223.6124