End-User Research in the Informatics Process

1
End-User Research in the Informatics Process

• An Overview and Demonstration n Don Rickert,
  Ph.D. n 4/8/03

2
What do I mean by Research?

• Any set of activities where data are collected
  and analyzed for the purpose of making
  generalizations beyond the study (Hughes, 1999)

3
What Is User Research in the Technology
Development Context?

• The discipline that is concerned with
• Translating end user or other consumer insights
  into the development of new systems (or improving
  legacy systems)
• Who does this research?much interdisciplinary
  cross-fertilization, especially among
• Psychology
• Human Factors
• Information Decision Systems
• Market/Consumer Research
• Industrial Design
• Anthropology

4
Fundamental Foci of End-User Research in a
Technology Context

• Figuring out who the target users are
• Understanding the motivations of these target
  users
• Uncovering their unmet needs and desires (what is
  appealing?)
• Using the insights gained to support intelligent
  business decisions

5
Typical Pragmatic Research Questions

• Will anybody care about this product?
• Why should they care?
• Who are these people who will care?
• What should the product look like?
• How should it behave?
• Will people (the ones who care) pay enough for
  the product for us to make money?
• Bottom line Should we build it at all?

6
Faith in Reason

• Historically, the practice of Research involving
  people tends to be characterized by
• A rational view of human motivation
• Faith in linguistic measures (the Voice of the
  Consumer)

7
Risks to Gaining Useful Insights From People

• Asking irrelevant questions
• Not asking the right questions
• Asking questions in such a way as to create
  attitudes and opinions where none previously
  existed
• Assuming that what happens in a formal research
  setting applies in the real world
• Confusing what people are willing to say in a
  group interview with what they really think

8
More Risks

• Observing people doing things that they would not
  really do
• NOT observing them doing the things that they
  really would do
• Confounding peoples remembered or imagined
  beliefs, attitudes, intentions or experiences
  with actual beliefs, attitudes, intentions and
  experiences.
• Talking to, surveying or observing the wrong
  people

9
Reality Check

• Beliefs, attitudes and intensions expressed by
  people (i.e. Voice of the Consumer) are
  notoriously unsuccessful at predicting actual
  behavior.
• What people say and what they do are very often
  different.

10
Why Is This So?

• People are not consciously aware of all of their
  beliefs, attitudes and intensions.
• People don't really know what they want or need
  or how to articulate it.
• Motivation is not completely rational.
• Our thought process is different in different
  emotional states.
• Peoples memory about an experience is a VERY
  different thing than the experience itself.

11
The Role of Science

• Science is
• A form of inquiry that is capable of revealing to
  us the structure of the world and explaining what
  the world is really like. It seeks to
• Discover what is so
• Explain it
• Science myths
• It IS possible to measure the all of the key
  variables and thus explain peoples behavior
• Science is always deductive and involves
  hypothesis testing
• Good science is always quantitative
• Statistical significance is the same as
  importance
• Beyond that, there are MAJOR debates on what
  makes for inquiry that can be suitably called
  Science.

12
The Common Concerns on Which Most Scientists Agree

• The notion of Error and the importance of
  avoiding it
• Reliability
• Validity

13
Error

• Type I Error
• Failing to observe something that really is true
• Type II Error
• Concluding that something is true when it really
  isnt
• Type III Error
• Asking the wrong questions

14
Reliability

• Are your measurements good?
• E.g. can you repeat your study and consistently
  get the same results?

15
Validity

• Are you measuring what you think you are
  measuring?
• Internal Validity
• Is the effect you observe due to I stimulus that
  you have identified or something else entirely
  (that you dont know about)?
• External Validity
• Do the conclusions based on your observations
  apply in the real world?

16
Good Science boils down to

• Avoiding error i.e.
• Failing to observe something that really is true
• Concluding that something is true when it really
  isnt
• Asking the wrong questions
• By
• Developing reliable measures
• Observing the effects of stimuli on outcomes
• Ruling out threats to validity

17
All of the methodology stuff that researchers
argue about

• is really about tactics for controlling error
  furthering reliability and validity, such as
• Representative samples
• Qualitative vs. Quantitative
• Response bias
• Degrees of freedom
• Experimental control
• Situational biases
• Repeatability

18
Convergence of Method

• No research method is perfect.
• No method, by itself, can ensure reliability and
  validity
• Scientists always employ multiple methods, using
  the strengths of one or more methods to
  compensate for the weaknesses of each.
• The process is called
• Convergence of method
• Or Triangulation

19
The Most Common Methods Employed by End-User
Researchers

• Questionnaires
• Focus Groups
• Interviews
• Usability Testing
• Real World Observation, such as
• Ethnography
• Analysis of Artifacts

20
Questionnaires

• Key Points
• Acceptable for getting a pulse on gross
  opinions
• I like this.
• I hate that.
• Easy way to collect data for quantitative analysis

21
Questionnaires

• Typical purpose
• To measure peoples beliefs, attitudes,
  intentions in order to make inferences about
  their behavior
• To measure peoples experiences to assess their
  satisfaction with experiences of products
• To obtain factual information about people,
  such as their income or age.
• Why researchers use them
• Easy way to collect data for quantitative
  analysis
• Easy to get large sample sizes for statistical
  significance
• The risks
• They measure peoples remembered or imagined
  beliefs, attitudes, intentions or experiences
• The possibility
• A questionnaire with a well-documented
  relationship with actual behavior outcomes makes
  powerful quantitative analyses possible.

22
Focus Groups

• Typical purpose
• Same as questionnaires measure beliefs,
  attitudes, intentions and experiences
• Why researchers use them
• An assumption that, somehow, group discussion of
  attitudes, beliefs, intentions and experiences
  draws out the truth.
• Convenience
• Focus groups are a big business and there are
  lots of facilities.
• They look rigorous
• The risks
• They measure what people are willing to say in a
  group about their remembered or imagined beliefs,
  attitudes, intentions or experiences
• The possibility
• The discussion that occurs in focus groups can
  give researchers many insights to better explain
  actual consumer behavior that they observe in
  companion studies.

23
Interviews

• Key Points
• Can be
• In a facility or
• In the field or
• Over the telephone
• Do not generate easy to analyze data.

24
Interviews

• Typical purpose
• You guessed itmeasure beliefs, attitudes,
  intentions and experiences
• Why researchers use them
• Probing by a good interview can draw out much
  more information than questionnaires
• Conveniencecan even be done over the telephone
  or online
• The risks
• Same as questionnaires and focus groups, plus
• Asking questions can create attitudes and
  opinions where none existed before.
• The possibility
• Hearing what consumers say can add to insights
  gained from actual observation of behavior.

25
Usability Testing

• Key Points
• What people do in a controlled setting
• When you are asking the questions
• Relatively unobstrusive observation of
  semi-realistic behaviors

26
Usability Testing

• Typical things that researchers want to measure
  with usability tests
• Problems with ease of use and other conventional
  quality problems
• Why researchers use them
• A notion of scientific rigorcontrolled setting
  with well-defined usage scenarios
• They can allow relatively unobtrusive observation
  of semi-realistic behaviors
• They do a good job of uncovering egregious
  problems with conventional quality
• The risks
• Type III error
• the type of error arising from asking the wrong
  questions
• Observing people doing things that they would not
  really do and NOT observing them doing the things
  that they really would do.

27
The possibility for usability tests

• When the right research questions are identified,
  usability testing really can help in identifying
  unforseen problems with a product.
• When combined with other methods such as
  questionnaires, interviews, real-world
  observation, etc. usability testing provides
  compelling data.

28
Ethnography

• Key Points
• On peoples own turf
• Observation of what they really do
• Often supplemented by field interviews and
  quantitative
• Expensive
• Data analysis is time-consuming.
• High external validity (generalizable conclusions)

29
Analysis of Artifacts

• Usually
• Customer Service logs
• Complaint emails
• Provide good context for understanding actual
  observations of customer behaviors
• Can lead to wrong conclusions if interpreted
  stand-alone (not in the context of actual
  observation).
• I.e. should be done in concert with ethnography
  or other observational research

30
Ethics in End-User Research

• The following slides are exerpted from a refereed
  paper entitled The Ethics Problem in Technology
  Companies and What Can Be Done About It
  presented by Don Rickert and Alycen Whiddon at
  the
• The eighth annual Ethics Technology
  Conference, June 24-25, 2005 at Saint Louis
  University
• http//205.240.10.101/denver.ethicstech/past_confe
  rence.html

31
Happy workplaces are a prerequisite for becoming
ethical workplaces.

• This is all about creating workplaces that are
  conducive to the type of reflection and
  responsibility needed on the part of every
  individual in the making of an ethics-driven
  company.
• See Mihaly Csikszentmihalyis recent book, Good
  Business Leadership, Flow and the Making of
  Meaning (2003)

32
Good workplaces dont have

• Periodic staff reductions, especially when
  outsourcing is used to fill the deleted positions
• Mergers (culture clashes and staff reductions are
  almost always a result)
• Too few staff combined with aggressive
  development schedules
• Vague and ever changing goals
• Frequent reorganizations and reshuffling of staff
• Dysfunctional politics

33
Human-Centered Product Design

• Human-centered design is all about attempting to
  meet the real needs of consumers, not the
  development of new and improved manufactured
  desires.
• What is important is that all companies,
  especially technology companies, need an
  organized and empowered group, whose duty it is
  to focus on the actual needs of customers.

34
The Belmont Report

• Prescribes the essential responsibilities of
  researchers working with human beings, including
  informed consent, assessment of risks and
  benefits and selection of research participants.
• Three key ethical principles
• Respect for Persons
• Beneficence
• Justice
• A copy of the Belmont Report is inclduded with
  this presentation

35
Focus Responsibility (Making ethical
considerations everyones problem)

• While the Belmont Report is a tool for focusing
  ethical decision-making for researchers, The
  Social Impact Statement helps engineers and
  others involved in the design of new products.

36
The Social Impact Statement

• Attributed to Ben Shneiderman
• Its explicit purpose is to provide a framework
  for designers to investigate the social impact of
  the systems they are responsible for.
• Every product development team should prepare a
  detailed Social Impact Statement for whatever it
  is they produce. The report includes the
  following sections
• Executive summary
• Description of the system
• Analysis of the ethical issues (stakeholders,
  principles, risks, etc.)
• Recommendations for actions, with analysis of the
  possible outcomes
• Reader's guide to literature on the issues in
  more depth
• Appendix that describes the methods used to
  collect data and prepare the analysis.

37
Education

• What we need is true education in ethical
  principles.
• Such education happens in university programs,
  not in two-day workshops.
• The real hope is with the next generations.
• Beacons of light
• Online Ethics Center for Engineering and Science
  at Case Western Reserve (http//onlineethics.org/)
  .