OBSERVING USERS

1
OBSERVING USERS

• Shubha Tandon
• Nandita Kodali

2
Outline

• Introduction
• Goals , Questions Paradigms
• How to Observe
• Data Collection
• Indirect Observation (tracking users activities)
• Analyzing, Interpreting and Presenting Data

3
Introduction

• Watching and Listening
• User can be observed in
• 1) Controlled environment as in Usability
  Testing
• 2) Natural environments i.e the feild
  

4
Goals, Questions and Paradigms

• Goals and Questions are necessary to help the
  observers stay focused
• Goals and Questions should guide all
  evaluation studies
• Studies should also be open to modification or
  refocusing as observers learn more about the
  situation

5
What and when to observe

• Observing is done at all stages of Product
• Development
• Observers can be
• a) on-lookers
• b) participant observers
• c) ethnographers
• The degree of immersion that evaluators adapt
  varies across a broad outsider-insider spectrum.

6
Outsider-Insider - illustration

• Aim To observe WAP phones
• Scenario 1The observer joins a group which goes
  to DC .The group uses it find restaurant in the
  area ,and while waiting for a taxi call the
  restaurant and book tables.
• Observes that there are some problems with the
  interface ,but on balance the device was useful
  and the group is pleased

7
Outsider-Insider - illustration

• Scenario 2 User need to do a preplanned task in
  a usability laboratory ,which is to search for
  the phone number of a particular restaurant .
• The video recording and interaction log suggest
  that the screen is to small for the amount of
  information they need to access

8
Outsider-Insider - illustration

• Which situation does the observer have more
  control ?
• What are advantages and disadvatages of the two
  methods?
• When might each type be more useful ?

9
Outsider-Insider - illustration

• Which situation does the observer have more
  control ?
• -Second study as the task is predetermined ,
• The participant is told what to do and they are
  in a controlled environment.

10
Outsider-Insider - illustration

• What are advantages and disadvantages of the two
  methods?
• Field Study
• Advantages- Real situations,Real Problems.
• Experienced the delight expressed at
  the over all concept and also frustration with
  the interface .
• Understood what the users like and need in
  real life situations

11
Outsider-Insider - illustration

• Disadvantages
• Since the observer is an insider how objective
  can she/he be?
• The observer might be having a good time and
  might not notice some peoples annoyance
• Another study could be done to find out ,but it
  is hard to replicate the exact situation.

12
Outsider-Insider - illustration

• Laboratory Study
• Advantages- Several users perform same task
• can compare performance and
  take averages.
• Easy for the observer to be
  objective .
• Disadvantage-Study is artificial ,says
  nothing about how to use the
  device in real environment.

13
Outsider-Insider - illustration

• When might each type be more useful ?
• Depends on the goals of the study
• Laboratory study is used for examining details
  of the interaction style and correcting usability
  problems e.g. button design etc.
• Field study is used to see how the phone is used
  in real world ,how it integrate or changes users
  behavior .

14
Approaches to Observation

• Quick and dirty observation -Can occur
  anywhere anytime, to find out what's happening
  quickly and informally.
• Observation in Usability testing -controlled
  environment, video and interaction logs used .
• Observation in field studies -observer can be
  anywhere in the outsider-insider spectrum.
  Complete participants ,more marginal participants
  ,observers who also participate ,people who
  observe from the outside and do not participate

15
How to observe

• In Controlled environments
• In the Field

16
How to observe

• Basic Data collection tools direct observation
  ,taking notes, collecting videos etc.
• Laboratory-what individuals do?
• Field in what context they do it ,
• how they interact with other
• people, technology etc .

17
Observation in Controlled Environments

• Observer collects data and then tries to make
  sense of the data .
• Practical issues have to be taken care of in
  advance eg where the users are located, test
  the equipment, get consent from user etc

18
Observation in Controlled Environments cont.

• Q) How does the observer know what the user is
  thinking?
• Sol) Thinking aloud Technique This technique
  requires users to think aloud everything they are
  thinking or trying to do

19
Observation in Controlled Environments cont.

• A better solution Two people working together
  and talking to each other .It is proven to be
  more successful , as its more natural and
  revealing for people to talk and help each other
  out .

20
Observation in the Field

• Events in the field are complex and change
  frequently.
• Evaluators have Frameworks to structure and focus
  their observations .

21
Observation in the Field cont.

• Eg of a basic framework
• The Person. Who is using the technology at any
  particular time ?
• The Place . Where are they using it ?
• The thing . What are they doing with it?
• Experts prefer a more elaborate frame work with
  greater attention to detail

22
Checklist for Field observation

• Have a study goal and questions
• Select a Framework
• Decide how to record events
• Routine revisions of notes and records
• Highlight and separate personal opinion
• Refocus if necessary
• Try to gain acceptance
• Prepare on how to handle sensitive issues

23
Checklist for Field observation cont.

• 9. Try Working as a Team
• 10.Consider different perspectives

24
Data Collection

• Methods available
• Notes and still camera
• Audio recording plus still camera
• Video
• Can be used individually or in conjunction
• Which to use - decide based on context, time
  available and subject sensitivity.

25
Notes Plus still Camera

• Least technical and cheep
• Taking notes is flexible and unobtrusive
• Transcribing handwritten notes, can help organize
  and analyze data
• Downside
• writing is tedious, slow and boring
• maybe biased - only what note-taker thinks is
  important gets recorded.
• feedback to design team depends on the
  note-takers authority
• Photos can supplement written notes.

26
Audio recording plus Camera

• Inexpensive
• Provides mobility
• Relatively unobtrusive
• Provided extensively detailed audio information,
• Permanent original record - can be revisited
• More convincing than notes - incontrovertible
• Drawbacks
• Requires a lot of transcription - but depending
  on detail maybe only parts are required.
• High external noise
• Changing cassettes and microphone position maybe
  a problematic.

27
Video

• Captures audio visual information- Complete
• Reliable, unbiased, permanent data
• Critical incidents can be identified and tagged
  for analysis
• Downside
• More expensive - requires mixing, analysis
  equipment
• Obstructive - requires focusing and positioning
• Attention becomes focused on what is seen - may
  miss important details outside the focus span.
• Detailed analysis may be very time consuming -but
  may not be needed

28
Indirect Observation - Tracing Users

• When Direct observation is not possible
• Totally unobtrusive
• Two techniques
• Diaries
• Interaction Logging
• From these records evaluators reconstruct what
  happened and look for usability problems

29
Diaries

• Provide records of
• what users did
• when they did it
• what they though about interaction with
  technology
• Very useful when users are scattered and
  unreachable in person. E.g.. Internet and web
  evaluations
• Templates (like open questionnaires) can be
  created for standardization

30
Diaries - good and bad

• Disadvantages
• study needs the participants to be committed to
  remembering and completing the dairies
• Needs incentive
• Have to be very straightforward
• user observations tend to be subjective (better
  or worse/ longer or shorter) than they actually
  are

• Advantages
• helpful scattered users
• Inexpensive
• No special equipment or expertise
• suitable for long term studies
• Standardized online templates can be read
  directly into a database

31
Interaction logging

• Can be done by recording
• key presses, mouse or other device movements
• these logs can be synchronized with video and
  audio to understand how users go about the tasks
• Logs are time stamped to calculate how long users
  spend on a particular task or part of software.
• For websites
• Explicit counters to record number of visitor
• counters to how long people stayed at a site and
  which areas they visited, where they came from
  etc.

32
Interaction Logging - good and bad

• Advantages
• Unobtrusive
• large volumes of data collected automatically
• can help collect useful information about number
  of visitors used to maintain and upgrade a
  website or subsites

• Disadvantages
• Is it ethical?
• powerful tools needed to analyze the vast amount
  of data collected. (e.g. WebLog)

33
Interaction logging - Is it ethical?

• Tradeoff
• With technology data can be collected without
  users knowledge.
• If users are told they are being observed they
  may change their behavior.
• Where do you draw the line ?

34
Analysis, Interpreting and presenting the data

• Studies generate large amounts of data
• So, it is important to
• first identify goals and questions
• based on these determine which data is collected
  and how it is analyzed.
• For analysis
• eyeball data to see what stands out
• Are there patterns or significant events?
• Is there evidence that answers a question or
  supports a theory?
• Analysis according to goals and questions

35
Data Categories

• Three types of data
• Qualitative data which is interpreted - tells a
  story about what was observed
• Qualitative data which is categorized - using
  techniques like content analysis
• Quantitative data treated statistically

36
Qualitative analysis to tell a story

• Objective To construct convincing story
  illustrated with powerful example from data.
• Steps involved
• Review data after each observation session,
  identify key themes and make collections
• Record themes in a coherent, flexible form, with
  examples
• Record the date and time of each data analysis
  session.
• As themes emerge, check understanding with other
  observers and informants.
• Iterate this process till a faithful story
  emerges
• Report findings to development team, preferably
  with an oral presentation as well as report.

37
Qualitative analysis for categorization

• Three main techniques
• Looking for incidents or patterns
• Analyzing data into categories
• Analyzing discourse

38
Looking for incidents/ patterns

• Useful when extremely fine-grained analysis not
  needed
• Look for critical incidents when users were
  obviously struck
• marked by comment, silence, puzzled looks
• review these in detail, treat rest of video as
  context
• Another approach Use theory to focus on relevant
  incidents.
• Need tolls for handling data and recording
  analysi
• NUDIST
• Video-PRO, etc.
• Report from analysis is feed back to development
  team with video clips

39
Analyzing Data into categories

• Content analysis Fine grained way of analysis
  video
• Challenges
• Determining meaningful orthogonal (mutually
  exclusive) categories to code the content under.
• Deciding appropriate granularity for
  categorization
• Content categories have to be reliable so that
  analysis can be replicated. - can be accomplished
  by training two researchers in using categories
  and having both analyze the data till high
  inter-research reliability rating is obtained
  obtained.

40
Analyzing discourse

• Assumption There is no objective Scientific
  truth
• Focus analysis on the meaning of what is said,
  not the content
• strongly interpretive - different people may have
  different perspectives
• pays attention to context
• Conversation analysis (fine grained discourse
  analysis) in which semantics is analyzed is
  detail is used for analyzing discourse on
  Internet (chartrooms, bulletin boars, etc.)

41
Quantitative data analysis

• Steps involved
• Video data collected in usability laboratories is
  annotated by hidden observers
• Errors or unusual behavior is usually marked and
  remarks added
• Evaluators use the annotations to calculate
  performance times so as to compare performance of
  various prototypes
• This data is also subjected to simple statistical
  analysis - mean, SD etc..

42
Feeding findings back into design

• How to convey evaluation results to design team
• Well written report with overview at beginning
  and detailed content list
• Include anecdotes, quotations, pictures, and
  video clips
• Quantitative data may be helpful depending on
  type of study and goals
• Verbal presentation including video clips is very
  powerful
• having both qualitative and quantitative analysis
  is good - provided different perspectives.

43
Summary

• Observation in usability testing depends tends to
  be objective from the outside
• In participant observation, observers works with
  the user to understand their activities and
  problems
• Observational data collection and analysis
  depends on paradigm quick and dirty, user
  testing, field study
• For data collection a combination of video,
  audio, paper records, diaries and data logs can
  be used.

44
Summary

• Evaluators should discuss and summaries their
  findings soon after the observation session.
• Analyzing video and data logs tedious
  important to have access to appropriate tools and
  specific questions and goals to guide the process
• Helpful to flag events in real time and subject
  these key events to detailed analysis, using the
  rest as context.

45
Questions ???