Observing users

1
Observing users

• Presenter JungMin shin JaeYoung Lee

2
Contents

• Goals, questions, paradigms
• How to observe
• Data Collection
• Indirect Observation tracking users activities
• Analyzing, interpreting, and presenting data
• Field studies

3
Goals, questions, paradigms

• What and when to observe
• Approached to observation

4
Goals, questions, paradigms

• Goal and question provide a focus for
  observation,
  
• DECIDE framework.
• Having goal helps to guide the observation
  because
  
• there is always so much going on.
• What and when to observe
• On developing product
• starting time helps understand users needs.
• ending time examine to satisfy users needs.

5
Goals, questions, paradigms

• Role of evaluator
• Onlookers, participant observers, ethnographers
• Table 12.1 Type of observation

6
Goals, questions, paradigms

• Approaches to observation
• 1. Quick and dirty observation
• Watching and talking to users in a casual way to
  
  
• get immediate feedback.
• 2. Observation in usability testing
• Video and interacting logs capture everything.
• Can watch through a one-way mirror or via TV
• screen.
• Observational data is used to see and analyze for
  
  
• users action.

7
Goals, questions, paradigms

• Observation in usability testing System

8
Goals, questions, paradigms

• 3. Observation in field studies
• Observers may be anywhere along the outsider
• insider spectrum.
• Colin Robson(1993) participating level
• complete participants
• more marginal participants
• observers who also participate
• people who observe from the outside and do not
  participate.
  
• The goal is to cause as little disruption as
  possible.

9
Goals, questions, paradigms

• 3. Observation in field studies (Continue)
• ex)
• A study of the time spent by boy and girl using
  technology in the classroom.
  
• - observe and note at the back of room.
• A study of understanding how the computer
  integrates with other artifacts and social
  interacting.
  
• - might take of insider perspective.

10
Goals, questions, paradigms

• 3. Observation in field studies (Continue)
• participant observation evaluator
• participate with users in order to learn
  what they do and how and why they do it.
  
• fully participant observer
• a member of group.
• being present to share experience and
• learning the social conventions of the
• group.

11
Goals, questions, paradigms

• Determining goals , exploring questions Choosing
  
  
• techniques are necessary steps in the DECIDE
• framework.

12
How to observe

• In controlled environments
• In the field environments
• Participant observation and ethnography

13
How to observe

• In controlled environments
• Collect and then Make sense of the data while
• watching users in controlled environment.
• Practical issues in controlled environment
• - where users will be located
• - equipment test
• - An informed consent form

14
How to observe

• In controlled environments
• the observer doesnt know what users are
  thinking, and can only guess from what she sees.
  
• Think-aloud technique by Erikson and Simon
• Require people to say out loud everything that
  they are thinking and trying to do.
  
• ex) Im typing in http//ics.uci.edu (type)
• Now I press the enter key, right? (presses
  enter key)
  
• But if users meet some errors, they could be
  silent again...
  
• Sol) Evaluator reminds him think out loud, but
  that would be intrusive.
  
• - two people work together so that they
  talk to each other.

15
How to observe

• In the field environments
• Events in the field can be complex and rapidly
  changing.
  
• Framework
• help observers to keep their goals and
  questions in sight.
  
• Collin Robson(1993)
• Space, Actors, Activities, Objects, Acts,
  Events, Goals, Feelings
  

16
How to observe

• In the field environments
• Checklist of things to plan before going into
  field
  
• 1. State the initial study goal and question
• 2. Select the framework
• 3. Decide how to record events
• 4. Be prepared to go through your notes and other
  records
  
• 5. Make and review your notes
• 6. Be prepared refocus your study
• 7. Think about how you will get the acceptance
  and trust of
  
• those you observe
• 8. Think about how to handle sensitive issues
  (privacy)
  
• 9. Consider working as a team
• 10. Consider checking your notes with an
  informant or members
  
• of the group
• 11. Plan to look at the situation from different
  perspectives

17
How to observe

• Participant observation and ethnography
• Checklist for doing ethnography
• Identifying a problems or goal and then ask good
  questions to be answered by the study
  
• The most important part of fieldwork just being
  there to observe, ask questions, and record.
  
• Collect a variety of data (Interview,
  retrospectives interview)
  
• Be prepared to move backwards and forwards
  between the broad picture and specific questions
  
• Analyze the data using a holistic approach -
  contextualized, usually iterative

18
How to observe

• Participant observation and ethnography
• When have I observed enough?
• schedules often dictate when your study end.
• stop when you stop learning new things
• when you start to see similar pattern of
  behavior being repeated.
  
• when you have listened to all the main stake
  holder group and understood their perspectives.

19
How to observe

• Participant observation and ethnography
• How can I adapt ethnography to fit the
  development
  
• process?
• Ann Rosa Procedure
• Preparing (Understand/Familiarize/Set Goal
  /Gain
  
• permission)
• Field study (Establish/Observe and interview/
• Follow any lead/ Record)
• Analysis (Compile/Quantify/Reduce and
  interpret/
  
• Refine the goals)
• Reporting (Consider/Prepare)

20
Data collection

• Notes plus still camera
• Audio recording plus still camera
• Video

21
Data collection

• Notes plus still camera
• being difficult and tiring to write and observe
  at the same time
  
• bored, the limitation of writing speed
• (sol) working with another person
• (disadvantage) have to be transcribed
• Audio recording plus still camera
• Useful alternative to note taking and is less
  intrusive than
  
• Video.
• Drawback Lack of visual record, transcribing
  the data
  
• So, evaluators use the recording to remind them
  about
  
• important details and as a source of anecdotes
  for reports.

22
Data collection

• Video
• Advantage
• Capturing both Audio and Video Data
• Disadvantage
• Intrusive.
• Attention becomes focused on what is seen
  through the lens.
  
• Time-consuming for analyzing video data.
• ex) 1 hour video recording over 100 hours of
  
  
• analysis time

23
Indirect Observation tracking users activities

• Diaries
• Interaction logs

24
Indirect Observation tracking users activities

• When ? Impossible to observe directly
• Diaries
• A record of what users did, when did it, what
  thought about their
  
• interactions with the Technology.
• (Advantage)
• When User scattered and unreachable person.
• Inexpensive - no equipment or expertise.
• Suitable for long term studies.
• Template and Enable the data to go straight into
  database.
  
• Ex) open-ended online questionnaires
• (Disadvantage)
• Rely on participant (reliable/remembering).
• Participant remember better or worse that they
  really were.

25
Indirect Observation tracking users activities

• Interaction logs
• key presses, mouse or other device movement are
  recorded usually
  
• synchronized with A/V log.
• Has time-stamped (to calculate how long)
• Explicit counter / Recording number of visitors
• EX) In study of interact art museum (Researchers
  at USC)
  
• Server log (7 months/ analysis tool webtrends)
• when/what /how long/what browser/what country
• (Advantage)
• Unobtrusive (but this also raises ethical
  concerns)
  
• Large volumes of data.

26
Analyzing, interpreting, and presenting data

• The first things to do is to eyeball.
• Proceed to analyze it according to the goals and
• questions.
• Qualitative analysis to tell a story
• Qualitative analysis for categorization
• Quantitative data analysis
• Feeding the findings back into design

27
Analyzing, interpreting, and presenting data

• Qualitative analysis to tell a story
• Power of analyzing descriptive data - being able
  
  
• to tell a convincing story
• Interview with Sara Bly (User-Centered Design
  Consultant)
  
• At the end of each observation period
• Review their data
• Discuss what they observed
• Construct a story from their data

28
Analyzing, interpreting, and presenting data

• Qualitative analysis to tell a story
• To summarize about Interview with Sara Bly.
• Review the data.
• Record the themes.
• Record the date and time of each data analysis
• Session.
• As themes emerge.
• Iterate this process (until your story faithfully
  
  
• represents).
• Report your findings to the development team.

29
Analyzing, interpreting, and presenting data

• Qualitative analysis to tell a story
• Analyzing and reporting ethnographic data
• - ethnographers reconstruct knowledge to
  produce detailed description known as rich or
  thick description.
  
• Main activities (Fetterman-1998)
• Look for key events
• Look for patterns of behavior in various
  situations and among different players
  
• Compare sources
• Report your findings in convincing and honest
  way
  
• S/W tools NUDIST and Ethnograph

30
Analyzing, interpreting, and presenting data

• Qualitative analysis for categorization
• Looking for incident or patterns
• Common strategy is to look for critical
  incidents
  
• ex) Jurgen Koenemann-Belliveau el al(1994)
• used this form of critical incident analysis to
  examine
  
• breakdown or problems in achieving a
  programming task
  
• and also to identify possible threats of
  incidents.
  
• Using the theory helped the evaluators to focus
  in relevant
  
• Incidents.

31
Analyzing, interpreting, and presenting data

• Qualitative analysis for categorization
• analyzing data into categories
• contents analysis provides another fine grain
  way of analyzing video data.
  
• determined by the evaluation question.
• must also be reliable so that the analysis can
  be replicated.
  
• EX)
• training second person and then analyze the same
  data sample.
  
• if there are large discrepancy between each
  other. - what?
  
• Inter-research reliability rating
• - the percentage of agreement between the two
  researchers.

32
Analyzing, interpreting, and presenting data

• Analyzing discourse
• Dialog
• Strongly interpretive, pays great attention to
  context.
  
• Language is viewed as a constructive tool and
  discourse
  
• analysis provides a way of focusing upon how
  people use
  
• language to construct versions of their worlds
  (Fiske,1994)
  
• Small changes in wording can change meaning, as
  the
  
• following excerpts indicate (Coyle, 1995)
• (According to Coyle,) discourse analysis is what
  you do when
  
• you are saying that you are doing discourse
  analysis

33
Analyzing, interpreting, andpresenting data

• Quantitative data analysis
• In case of errors or unusual behavior -
• marks the video and records a brief remark.
  
  
• To use this annotated recording
• 1. Calculate performance time
• 2. further analyzed using simple statistics such
  as means, standard deviations, T-tests, etc.

34
Analyzing, interpreting, and presenting data

• Feeding the findings back into design
• The results can be reported to the design team.
• Written reports with a overview and detailed
• content list.
• Quantitative data - its value depends on the
  type
  
• of study and its goals.
• Often both quantitative data analysis are useful
  
  
• because of they provide alternative perspectives.

35
Field Study
36
Field Study

• A broad range of data gathering techniques at the
  users location
  
• There is no one best way to conduct a field
  study. It depends on the goals of your study and
  your access to users.

37
The Goal of a Field Study

• Field studies are excellent for collecting rich,
  detailed data and obtaining a holistic view of
  the process or domain.
  
• Field studies are not a good technique if you
  want to collect quantitative data from large
  samples.
  
• The goal of a field study is to make the implicit
  aspects and processes of the user environment
  explicit.

38
Field Study Methods

• Techniques are divided into three categories
• Observation only
• Interacting with the user
• Method supplements
• The Goal of the Field Study Methods
• To observe users and collect information about
  their tasks and the context in which they are
  done.
  

39
Field Study Methods

• Observation Only
• Pure Observation
• Deep Hanging-Out
• Interacting with the user
• Contextual Inquiry
• Process analysis
• Condensed ethnographic interview
• Discount User Observation (DUO)
• Method supplements
• Artifact walkthroughs
• Incident diaries
• Observing while you are not present

40
Field Study Methods

• Observation Only
• Pure Observation
• Deep Hanging-Out

41
Observation only - Pure Observation

• Synopsis
• When you are unable or dont wish to interact
  with the user, you simply observe from a
  distance.
  
• Advantages
• Flexible
• Low resources
• Level of effort
• Minimal, Place your self in a good vantage point
  and observe as many user/sites/tasks as you feel
  appropriate
  
• You continue to conduct observations until you
  feel you have a good understanding of the domain
  of areas of focus.

42
Observation only - Pure Observation

• Valuable in situations where you cannot interact
  with the end users.
  
• Users may or may not know they are being studied.
  
  
• Do not interact with the participant
• Do not distribute surveys, interview the user, or
  ask for artifacts from the user.
  
• It is essential to have a good sampling plan. The
  sampling plan should include specific information
  such as days/times you anticipate key events
  (e.g., the day before Thanksgiving, or bad
  weather at an airport), as well as normal days.

43
Field Study Methods

• Observation Only
• Pure Observation
• Deep Hanging-Out

44
Observation only Deep Hanging out

• Synopsis
• This method is similar to pure observation but
  provides more structure by suggesting focus areas
  and things to observe.
  
• Advantages
• It has more structured than pure observation so
  you can do a more detailed level of data analysis
  and compare data collected across multiple sites.
  
• Level of effort
• Because there is more structure, it takes more
  effort than pure observation.
  
• You are on at all times, which can be tiring.
• It is also valuable to become a user yourself (if
  possible) and collect artifacts along the way.

45
Observation only Deep Hanging out

• A more structured form of pure observation.
• Involves significant amounts of observation along
  with involving yourself in the process
  
• Method of Deep Hanging-Out includes
• Structured observation
• Collection of artifacts
• Becoming a user yourself
• However, you do not interview participants,
  distribute surveys, or present design ideas for
  feed back.

46
Field Study Methods

• Observation Only
• Pure Observation
• Deep Hanging-Out
• Interacting with the user
• Contextual Inquiry
• Process analysis
• Condensed ethnographic interview
• Discount User Observation (DUO)
• Method supplements
• Artifact walkthroughs
• Incident diaries
• Observing while you are not present

47
Field Study Methods

• Interacting with the user
• Contextual Inquiry
• Process analysis
• Condensed ethnographic interview
• Discount User Observation (DUO)

48
Interacting with the user - Contextual Inquiry

• Synopsis
• Interview, apprentice with, and interpret the
  resulting data with users
  
• Advantages
• Contextual Inquiry is more focused and context
  dependent than the other methods
  
• At the end, you walk away with actionable items
• Level of effort
• The effort level is higher than for
  observation-only techniques. You must develop an
  observation guide, observe users, apprentice with
  them, and discuss your observations with them

49
Interacting with the user - Contextual Inquiry

• Four main parts to Contextual Inquiry
• Context
• Partnership
• Interpretation
• Focus

50
Interacting with the user - Contextual Inquiry

• Four main parts to Contextual Inquiry
• Context You must go to the users environment
  in order to understand the context of his/her
  actions. Contextual Inquiry assumes that
  observation alone or out-of-context interviews
  are insufficient

51
Interacting with the user - Contextual Inquiry

• Four main parts to Contextual Inquiry
• Partnership To better understand the user,
  tasks, and environment, you should develop a
  master-prentice relationship with the
  participant. Immerse your self in the
  participants work and do as he/she does.
  Obviously, this is not possible with many jobs
  (e.g., surgeon, fighter pilot)

52
Interacting with the user - Contextual Inquiry

• Four main parts to Contextual Inquiry
• Interpretation Observations must be interpreted
  with the participant in order to be used later.
  Verify that your assumptions and conclusions are
  correct.

53
Interacting with the user - Contextual Inquiry

• Four main parts to Contextual Inquiry
• Focus Develop an observation guide to keep you
  focused on the subject of interest/inquiry.

54
Field Study Methods

• Interacting with the user
• Contextual Inquiry
• Process analysis
• Condensed ethnographic interview
• Discount User Observation (DUO)

55
Interacting with the user Process Analysis

• Synopsis
• Capture the task sequence for a process that may
  span several days
  
• Advantages
• Because it is more focused than Contextual
  Inquiry, it is also much faster
  
• Level of effort
• You need to stay focused on the process at hand
  to help users walk you through the process of
  interest. It may span just several days.

56
Interacting with the user Process Analysis

• A process analysis is similar to Contextual
  Inquiry. However, you enter with a series of
  questions and you do not necessarily apprentice
  with the user
• Questions
• When does the first task in the process happen?
• What triggers it?
• Who does it?
• What information does the person have when the
  task begins?
  
• What are the major steps in the task?
• What information comes out of it?
• Who is the next person in the chain of the
  process?

57
Interacting with the user Process Analysis

• Example - Process map for a travel agents
  typical day

58
Field Study Methods

• Interacting with the user
• Contextual Inquiry
• Process analysis
• Condensed ethnographic interview
• Discount User Observation (DUO)

59
Interacting with the user Condensed
Ethnographic Interview

• Synopsis
• Use the results of semi-structured interviews to
  guide observations
  
• Advantages
• This technique is considerably shorter than some
  of the other techniques, because the interviews
  scope what you observe
  
• It also limits the data you are able to collect
• Level of effort
• It requires a medium level of effort to conduct
  interviews and then observe users and collect
  artifacts

60
Interacting with the user Condensed
Ethnographic Interview

• Employs the standardization and focus of a
  semi-structured interview along with the context
  of observations and artifacts
  
• Characterized as top-down, because the
  interviews form a general framework from which to
  interpret specific observations.
  

61
Interacting with the user Discount User
Observation (DUO)

• Synopsis
• One facilitator interviews the user and takes
  notes while another facilitator takes pictures of
  everything
  
• Advantages
• You can quickly reconstruct the session through
  time stamped notes and digital pictures to verify
  your observations with users
  
• Level of effort
• It takes two investigators to focus on their
  specific tasks of either interviewing or taking
  pictures and notes
  
• It will take additional time/effort to combine
  notes with digital photos and put them in a
  multimedia presentation tools to review with
  users.
• It takes additional effort to validate
  observations with user afterward.

62
Field Study Methods

• Observation Only
• Pure Observation
• Deep Hanging-Out
• Interacting with the user
• Contextual Inquiry
• Process analysis
• Condensed ethnographic interview
• Discount User Observation (DUO)
• Method supplements
• Artifact walkthroughs
• Incident diaries
• Observing while you are not present

63
Field Study Methods

• Method supplements
• Artifact walkthroughs
• Incident diaries
• Observing while you are not present

64
Method supplements Artifact walkthroughs

• Synopsis
• Collect all the artifacts used by participants
  and determine what triggers their use, when they
  are used, and for what
  
• Advantages
• Quick and easy to conduct
• Level of effort
• Low level of effort to review artifacts with
  participants and make/collect copies of them

65
Method supplements Artifact walkthroughs

• Quick and easy to conduct but provide
  indispensable data
  
• Artifacts are objects or items that users use
  to complete their tasks or that result from their
  tasks
  
• You have to sign the companys confidential
  disclosure agreement promising that you will keep
  all data collected confidential

66
Field Study Methods

• Method supplements
• Artifact walkthroughs
• Incident diaries
• Observing while you are not present

67
Method supplements Incident diaries

• Synopsis
• Worksheets the user takes home or to work to
  collect ongoing data
  
• Advantages
• No observation is required
• Able to understand more issues than what can be
  observed in the lab or during a single visit
  
• Level of effort
• Low effort level to create and distribute
  diaries
  
• Moderate effort to analyze data across multiple
  diaries. There will be a time lag between when
  you distribute the diaries and when you receive
  the data
• You are depending on the participants to
  follow-thorough.

68
Method supplements Incident diaries

• Incident diaries allow you to collect ongoing
  data from users
  
• Participants are provided with a notebook
  containing worksheets to be completed
  
• They are given to users to keep track of issues
  they encounter while using a product
  
• The worksheets may ask users
• Describe a problem or issue they encountered
• How they solved it
• How troublesome it was

69
Field Study Methods

• Method supplements
• Artifact walkthroughs
• Incident diaries
• Observing while you are not present

70
Method supplements Observing while you are not
present

• Synopsis
• Recording users in action when space, time, or
  restrictions prevent you from being there in
  person
  
• Advantages
• If you have multiple video cameras, you can view
  several users simultaneously
  
• Level of effort
• Low effort to set up cameras and record
• Moderate effort to meet with user again and
  review tapes
  
• Moderate effort to categorize and index behavior

71
Method supplements Observing while you are not
present

• You can observe users even when you are not
  present by setting up a video camera and then
  leaving
  
• Excellent way to understand detailed steps

72
Identify the type of study to Conduct

• decision diagram

73
Thank you