Chapter 12 Observing Users

1
Chapter 12 Observing Users

• Li, Jia
• Li, Wei

2
Outline

• What and when to observe
• Approaches to observation
• How to observe
• How to collect data
• Indirect observation
• Analyzing, interpreting and presenting data

3
Goals and Questions

• Provide a focus for observation
• Example watch a group of people
• First watch what are they doing?
• Second watch more focused questions
• Necessary for all evaluation studies
• Balance between being guided by goals and being
  open-minded

4
What and When to Observe

• During early design of a product
• Help understand users needs
• During later refinement of a product
• Find out whether it meets users needs

5
Type of Observation
Field environment (i.e., natural)
Controlled environment (i.e., lab-like)
Observation
Quick and dirty In field studies
Quick and dirty In usability testing
Outsider looking on
Participant observation (e.g., in ethnography)
(Not applicable)
Insider
6
Approaches to Observation

• Quick and dirty observation
• find out what is happening quickly and with
  little formality
• Observation in usability testing
• controlled environment
• Video, interaction logs
• Watch through one-way mirror or via remote TV
  screen
• Observation in field studies

7
Observation in Field Studies

• Levels of participation
• (By Colin Robson, 1993)
• Complete participants
• More marginal participants
• Observers who also participate
• Outside observer

8
Observation in Field Studies - Outside Observer

• When an observer is interested only in the
  presence of certain types of behavior
• Example
• Observe when a computer is used by boys
  and when by girls, and how long

9
Observation in Field Studies - Inside Observer

• Participant observers
• Participate in the lives of the people under
  study
• Maintain a professional distance that allows
  adequate observation and recording of data
• Ethnographers
• Controversy
• Participant observation is virtually synonymous
  with ethnography
• Participant observation is a technique used in
  ethnography

10
How to Observe - In Controlled Environments

• Issues before observation
• Decide laboratory to do the tests where
  appropriate equipment is set up
• Test equipment to make sure it works
• Provide an informed consent form for users to
  read and sign at the beginning of the study
• Problems
• Observers do not know what users are thinking
• Think-aloud technique

11
How to Observe - In the Field

• Issues before observation
• State initial study goal and questions clearly
• Select a framework to guide activity in the field
• Decide how to record events
• Go through notes and other records as soon as
  possible
• Highlight and separate personal opinion from what
  really happens

12
How to Observe - In the Field (Contd)

• Issues before observation
• Refine goals and questions while observation
• Gain the acceptance and trust of those to be
  observed
• Think about ways to handle sensitive issues
• Consider working as a team
• Check notes with an informant or members of the
  group
• Plan to look at situation from different
  perspectives

13
Frameworks

• Why use frameworks
• Provide focus
• Organize/structure observation
• What framework to select
• A simple framework
• The person
• The place
• The thing

14
Frameworks (Contd)

• What framework to select
• Goetz and LeCompte framework
• Who is present?
• What is happening?
• When does the activity occur?
• Where is it happening?
• How is the activity organized?

15
Frameworks (Contd)

• What framework to select
• Colin Robson framework
• Space appearance and lay-out
• Actors names and relevant details
• Activities what are they doing and why
• Objects present physical objects
• Acts what are specific individuals doing
• Goals what to accomplish for actors
• Feelings mood of the group and individuals

16
How to Observe Participant Observation and
Ethnography

• Issues before observation
• Identify goal and questions with the help of
  observation framework
• Get immersed in the group to observe
• Collect a variety of data
• Get balance between broad picture and specific
  questions
• Analyze data using a holistic approach

17
Dilemmas

• When should I stop observing
• When see similar patterns of behavior being
  repeated
• When finish listening to all the main stakeholder
  groups and understand their perspectives
• How can I adapt ethnography to fit the
  development process
• Preparation
• Field study
• Analysis
• Reporting

18
Data collection

• Notes plus still camera
• Audio recording plus still camera
• Video

19
Notes plus still camera

• Advantages
• Less technical.
• Flexible.
• Equipment easy to get and not expensive.
• Unobtrusive.
• Easy to transcribe.

• Disadvantages
• Difficult and tiring to write and observe at the
  same time.
• Writing speed limited.
• Less reliable, rely on the note-takers.

20
Audio plus still camera

• Disadvantages
• Lack of visual record
• Difficult to transcribing the data.
• Have to change tapes and position microphone .

• Advantages
• Inexpensive
• Flexible
• Relatively unobtrusive
• Low disturbance

21
Video

• Disadvantages
• Equipment more expensive
• Intrusive
• Time-consuming to analyze the video data
• Need positioning and focusing camera lens.

• Advantages
• Capture both visual and audio
• High reliability
• Permanent original record that can be revisited.

22
Indirect observation tracking userss activities

• Diaries
• Interaction logging

23
Diaries (1)

• What users did
• When they did it
• What they thought about their interactions with
  the technology

• Useful when users are scattered and unreachable
• such as Internet and web evaluations

24
Diaries (2)

• Advantages
• Inexpensive
• No special equipment or expertise
• Suitable for long-term studies

• Disadvantages
• Rely on participants being reliable and
  remembering to complete the diaries.
• Participants often remember events as being
  better or worse than they really were, or taking
  more or less time than they actually did.

25
Interaction logging

• Advantages
• Unobtrusive
• Large volumes of data can be logged automatically

• Disadvantages
• Ethical concerns.
• Powerful tools are needed.
• Webtrends

26
Dilemma --- observing
without being seen

• They do not know we are watching. Shall we tell
  them?
• Tell users may react or change their behavior.
• Depends on the context
• How much personal information is collected?
• How the information will be used?

27
Analyzing, interpreting, and presenting the data

• Qualitative data that is interpreted and used to
  tell the story about what was observed.
• Qualitative data that is categorized using
  techniques such as content analysis.
• Quantitative data that is collected from
  interaction and video logs and presented as
  values, tables, charts and graphs and is treated
  statistically.

28
Qualitative analysis to tell a story

• Review the data
• Record the themes
• Record the date and time
• Check your understanding
• Iterate this process
• Report your findings

29
Qualitative analysis for categorization

• Looking for incidents or patterns
• Analyzing data into categories
• Analyzing discourse

30
Looking for incidents or patterns

• Look for critical incidents, such as times when
  users were obviously stuck.
• Theory may also be used to guide the study.
• A variety of tools are available to record,
  manipulate and search the data.
• NIDIST
• Observer Video-Pro tool

31
Analyzing data into categories

• Content analysis provides another fine grain way
  of analyzing video data.
• It is a systematic, reliable way of coding
  content into a meaningful set of mutually
  exclusive categories.

32
Analyzing discourse

• Another approach to video and audio analysis is
  to focus on the dialog.
• Analyzing discourse on the Internet has started
  to influence designers understanding about
  users needs in these environments.

33
Quantitative data analysis

• Typically this data is further analyzed using
  simple statistics such as means, standard
  deviations, etc.

34
Feeding the findings back into design

• Clearly written reports with
• Overview at the beginning
• Detailed content list
• Verbal presentation