Principles of Language Use for User Centered Design of Software User Interfaces

1
Principles of Language Use for User Centered
Design of Software User Interfaces

• Derek Brock
• Navy Center for Applied Research in
• Artificial Intelligence
• Naval Research Laboratory

2
A different design perspective

• Designing a software user interface is like
  writing a reference book with many chapters and
  subsections
• The Designer is the author
• Users are the readers
• Interaction starting points in the user
  interfacemenu choices, documents, tool bar
  icons, etc.are like chapters and subsections
• The authors goal in each chapter or subsection
  is to provide readers with
• Clear and useful expositions
• Pointers to dependent and related references

3
Whats the point?

• Software is designed by people to be used by
  people
• The purpose of a user interface is to coordinate
  what the designer thinks users need to
  understandand doto use the software
• But few users can read a designers mind
• Any form of coordination between people requires
  some form of communication
• Successful communication requires
• Collaboration (a willingness to work together)
• Meaning and understanding
• The use of various kinds of signals
• A shared basis for the use of those signals

4
What goes on when we read a book?

• A successful communication takes place
• The author and the reader carry out a kind of
  collaborationboth have to work at it
• The authors meaning (ideally) becomes the
  readers understanding
• A variety of linguistic and nonlinguistic signals
  are usedtext, diagrams, illustrations, etc.
• The author and the reader begin with a communally
  shared basis for the signals used
• which becomes a personally shared basis when the
  communication is complete

5
The same ideas apply to software use

• When software use is successful
• The designer and the user carry out a kind of
  collaborationagain, both have to work at it
• The designers meaning (ideally) becomes the
  users understanding
• A variety of linguistic and nonlinguistic signals
  are usedlabels, icons, actions, sounds,
  messages, etc.
• The user also makes use of input signalsmore on
  this later
• The designer and the user begin with a communally
  shared basis for the signals used
• which becomes a personally shared basis when the
  communication is complete

6
The larger picture

• Signal use is really language use
• People use language to do things with each other
• Peoples language use skills are their greatest
  resource whenever meaning and understanding are
  involved
• The design of human-computer interaction is a
  representation problem that primarily involves
  the designers meaning and the users
  understanding
• The principles of language use are a foundational
  framework for this design problem

7
Introducing the study of language use

• This is not the study of linguistics
• Linguistics is the study of the development and
  structure of languages, particularly spoken and
  written languages
• Instead, its the study of whats involved in how
  people use signals to do things together
• How do people convey and grasp each others
  intentions?
• How do people succeed in coordinating this?
• How does this work when people arent
  face-to-face?
• What are the cognitive and social processes
  involved?
• How is language used in different conceptual
  domains of action?

8
First concepts and terms of art

• Language use requires participants and raises the
  notions of speaker and addressee
• In addition to signals, language use always
  involves speakers meaning and addressees
  understanding
• Joint actions are atomic instances of language
  use
• They are not composed of disjoint actions that
  are simply carried out by speaker and addressee
  independently
• But, instead, are the coordinated result of
  actions that speaker and addressee carry out
  together, as a dueteven when this must happen
  across time and space
• Joint activities emerge from sequences of joint
  actions
• Interactive joint activities can never be fully
  planned in advance
• This is the proper level for specifying roles,
  social purposes, goal hierarchies, etc.
• Language use always takes place in a setting
• Settings shape how language is used

9
Settings

• The notion of a setting in language use combines
  notions of scene and medium
• Scenewhere the language is used
• Mediumthe manner of language use (e.g., spoken,
  written, recorded, gestural, a mixture, etc.)
• Settings range in type, in their properties, and
  in what they require of people and the roles each
  person plays
• The fundamental paradigms are
• Face-to-face settingsconversation, lecture,
  wedding
• Written settingsbooks, letters, movies, recorded
  music
• A joint activitys setting determines how joint
  actions are coordinated and what skills are
  needed
• Face-to-face settings are the most basic (or
  natural)
• Written settings are the most demanding (or
  contrived)

10
Joint actions

• Complete joint actions involve the coordination
  of a causally related ordering of cognitive,
  physical, and perceptual subactions (speakers ?
  addressees)
• Proposing an intention ? considering it
  (engagement)
• Conceptually signaling the intention (meaning) ?
  recognizing it (understanding)
• Presenting the signal ? identifying it
• Executing the presentation ? attending to it
• All coordinated joint actions between people
  require these subactions in some form some
  examples
• A handshake, paddling a canoe, dancing, making
  music, etc.
• Asking for directions, posting a notice, driving
  in traffic, etc.
• To coordinate these subactions as parallel parts
  of the whole, people depend on their intuitions
  of downward evidence and upward completion

11
But wait

• How are signals used to convey meaning and
  achieve understanding?
• How can a speaker be confident a particular
  signal will convey his or her intended meaning
• How does an addressee converge on the intended
  meaning represented by the signal
• It turns out this is a deep problem for computer
  science, but something that nature has solved
  rather well
• Meaning and understanding depend on the notion of
  common ground

12
Characterizing common ground informally

• Common ground is knowledge people intuitively
  assume they can use with each other on a presumed
  basis of shared information
• The idea is that when Alan and Zoe go to a movie
• afterwards, they can presume they had mostly
  similar experiences of its content
• and can assume that each will now understand the
  others use of this newly shared information
• Shared experience naturally becomes a shared
  basis of information
• In fact, this is exactly how people introduce and
  then use new ideas with each other
• But even more is implied by this notion of a
  shared basis

13
Awareness of a shared basis is the essence of
common ground

• Verification or inference of similar experiences
  also builds shared bases
• Alan and Zoe dont even need to see a movie
  together to use their knowledge of it with each
  otherthey can simply find out if they have both
  seen it
• Common ground acquired in this way often contains
  many discrepanciesbut all common ground contains
  discrepancies
• And note Alan and Zoes separate knowledge of a
  movie isnt common ground at all until they have
  evidence that confirms theyve both seen itthat
  their knowledge is actually shared
• In general, people can usually account for what
  they take to be common ground

14
Building common ground

• Common ground is built in joint activities
• People expect each other to reason similarly and
  to maintain a similar awareness of what they are
  doing together
• People expect each other to be engaged
• Common ground in joint activities can be divided
  into at least three parts
• Knowledge initially taken to be held in common
• Knowledge of the activitys current state
• Knowledge of what has conspicuously taken place
  so far
• When common ground is missing, meaning and
  understanding immediately break down
• But people are very good at repairing
  misunderstandings and filling in missing common
  ground
• Building common ground is a lifelong social and
  personal process

15
Using common ground in joint actions posing
coordination problems

• People implicitly justify their common ground by
  using what they take to be shared references to
  it
• Shared references are simply signals that refer
  to elements of common ground they function as
  coordination devices
• Joint actions involving meaning and understanding
  are really coordination problems people pose and
  agree to solve together
• People use coordination devices to pose
  coordination problems, focus their intent, and to
  indicate their solution
• Most coordination problems in language use are
  familiar and most are readily solved
• To make a coordination problem easy to solve
• It should have a straightforward solution when it
  is posed
• Its coordination devices should fully indicate
  the solution
• The solution should be obvious from what is
  common ground

16
For example

• What does Zoe do if she and Alan are sharing
  popcorn at the movies and she wants a sip from
  Alans drink?
• She can signal Alan either by asking him for a
  sip or by simply pointing to his drink in a way
  that draws his attention
• By asking or gesturing, Zoe makes a clear and
  obvious shared reference to her common ground
  with Alan
• Either ploy is an excellent coordination device
  because
• Zoe has a straightforward solution in mind that
  Alan can fulfill
• Either device explicitly indicates her solution
• Her solution is obvious given her common ground
  with Alan
• In agreeing to Zoes coordination problem, Alan
  correspondingly expects that
• Zoe has an intended solution in mind (what hes
  supposed to do)
• Her signal contains enough information to solve
  the problem
• He can figure out the solution on the basis of
  what is obvious in their common ground

17
More about coordination problems

• Codifying the joint premises for solving
  coordination problems
• Solvability (a straightforward solution exists)
• Sufficiency (the solution is fully indicated)
• Salience (the solution is obvious from common
  ground)
• Working out coordination problems is a dominant
  part of peoples language use skills.
• These skills have their origins in the perceptual
  richness of face-to-face settings
• In choosing the right shared basis to use as a
  coordination device, joint salience is usually
  the most important consideration.
• Appealing to this aspect of peoples attention in
  context greatly increases the likelihood of their
  converging on the intended solution.

18
Salient signals

• Signals that make salient coordination devices
  with respect to common ground include
• Actions (including sounds and gestures)
• These command perceptual attention
• External representations
• These are physical aspects of a language use
  setting that have conceptual meaning(s) for a
  joint activity for example
• Playing cards, the elements of a board game,
  ceremonial objects, etc.
• External representations enjoy perceptual
  immediacy
• Conventions of use
• Conventions are representations of standing
  solutions to frequently occurring coordination
  problems
• When a convention is used as a coordination
  device, addressees usually know how to proceed
  immediately
• Languages are full blown systems of conventions
• Conventions of use give written settings their
  power

19
Layersconceptual domains of action

• Layers are shifts in the conceptual domain of
  action (and/or its setting) that shape the focus
  of common ground for purposes of solving
  particular kinds of coordination problems in
  joint activities
• Layers depend on peoples abilities to change
  perspective
• Storytelling illustrates the basic paradigm
• Primary layer
• A storyteller and an audience use language in a
  real world setting as themselves
• Secondary layer
• The characters of the story use language in the
  story world setting
• Note that the audience has no trouble shifting
  its common ground into the story world
• Further layers within the story may arise
• Layers can involve play acting, removes in space,
  time, and reality, persons not present, changes
  in setting, etc.

20
Summary of language use principles

• Language use takes place in settings (e.g.,
  face-to-face, written) that shape its
  coordination and its required skills
• Language use is a joint activity it is
  participatory as opposed to an individual
  activity
• Joint actions are the unit of language use
• They involve cognitive, physical, and perceptual
  levels of subactions
• They are coordination problems
• Coordination devices are signals people use to
  pose, focus, and solve coordination problems
• The joint premises for solving coordination
  problems are solvability, sufficiency, and
  salience
• Salient sources for coordination devices include
  actions, external representations, and
  conventions of use
• Layers are conceptual shifts of perspective in
  joint activities that shape the focus of common
  ground

21
Language use in human-computer interaction

• Speaker and addressee in human-computer
  interaction are designer and user
• Software use is their joint activity
• This joint activity fully involves designers
  meaning and users understanding
• Software use takes place in a written setting and
  makes use of an important secondary layer
• Joint actions are the unit of software use
• An entire class of joint actions in software use
  are called user interactions
• User interactions and other joint actions in
  software use are simply coordination problems
• Designing an effective system of coordination
  devices highlights the problem of building common
  ground

22
Thinking about software as awritten setting

• As with any written setting, a software user
  interface is designed and produced in advance of
  when it is taken up by its user
• It is useful to think of a written setting as a
  type of pre-coordinated joint activity
• Conventional written settings build common ground
  through a static, serial narrative whose domain
  typically defines a secondary layer of action
• The written setting of software use differs from
  this paradigm in three important ways
• It has a non-serial narrative
• Its presentation is dynamic (non-static)
• Its secondary layer of action places the user in
  a type of face-to-face interactive setting

23
Characterizing the first two layers of software
use

• In the primary layer of software use
• The designer and the user pursue their joint
  activity (software use) as themselves in a
  written setting
• In effect the user reads the designers
  presentationthe display portion of the user
  interface
• In the secondary layer of software use
• The user participates in a face-to-face
  interaction with the computer (as opposed to the
  designer)
• Particularly because of the change in setting,
  each layer makes different demands of the users
  language use skills

24
Characterizing the primary layer of software use
in language use terms

• The design poses a fixed set of coordination
  problems
• Most software uses a cafeteria paradigm for user
  interaction
• This paradigm implies a non-serial narrative
• The design pre-coordinates the joint activity,
  but the user has opportunistic control of how the
  software use proceeds
• The user interface generalizes to a huge menu of
  starting points
• The presentation uses a language of coordination
  devices made up of a variety of linguistic and
  nonlinguistic signals,
• These include elements of natural language,
  visual artifacts, and behaviors (i.e., actions
  and procedures)
• Many elements of this language are (rightly)
  presumed to be conventions
• The presentation serves, in part, as an external
  representation of the joint activity
• Manipulable display elements are generally given
  specific conceptual meanings in the context of
  the software use

25
Building common ground in the primary layer of
software use

• In a cafeteria-style presentation
• Both the designer and the user strive to find and
  build common ground, but
• The designers control over the process of
  building a complete body of common ground with
  the user in the planned manner of a serial
  narrative is sacrificed
• The users common ground with the designer builds
  contingently (irregularly) on the basis of
• Opportunistically solving coordination problems
  posed in the interaction design
• And, more generally, opportunistically becoming
  familiar with the presentation language
• Despite the presentations non-serial narrative
• Over time, users generally develop a sense of
  where common ground is missing and will look for
  it unless the effort proves to be too costly

26
Designing coordination problems in the primary
layer of software use

• Coordination problems in a software design
  deliberately anticipate the users domain goals
• But addressees knowledge in written settings can
  never fully anticipated
• Users would benefit from an account of what the
  software can and cannot do
• All coordination problems posed by a design
  should honor users expectations of salience,
  sufficiency, and solvability
• From a practical standpoint, this is not always
  possible
• Nontrivial designs generally involve
  multidimensional conceptual dependencies
• Opportunistic user control pre-empts the
  designers control over the introduction of
  critical material
• When users are unfamiliar with relevant
  supporting concepts, they may not know what to do
  through no fault of their own
• Working solution identify where common ground
  may be missing and provide users with immediate,
  context dependent access to it

27
Other primary layer issues in software use

• Serial narratives naturally correspond to the
  third part of common groundpeoples record of
  what has openly occurred so far in their joint
  activity
• Serial narratives in hard copy are
• Intuitively indexed viewed as external
  representations, they easily represent their
  current state in this way
• Readily open to review when misunderstandings,
  questions, or lapses of memory arise
• These advantages are de-emphasized in the written
  setting of software use
• The presentation may not always represent the
  current state of the joint activity
• External representations in the presentation are
  generally dynamic (non-static) and actions are
  evanescent
• Cafeteria organization of entry points obfuscates
  the users intuitive ability to index the
  presentation and/or locate particular functions
• Little or no narrative record is kept for the
  user to consult

28
Characterizing the secondary layer of software
use in language use terms

• What we call human-computer interaction takes
  place in the secondary layer of software use
• In this layer, the user participates in a direct
  interaction with the computer (as opposed to the
  designer)
• The interaction is designed to resemble a joint
  activity in a face-to-face setting
• But this setting is not equivalent with a
  face-to-face setting between people
• The user and the computer use different
  communication languages
• and their perceptors are mismatched
• External representations in the presentation are
  technically available to the computer but are
  generally unused by the system to justify common
  ground beyond, at most, small numbers of
  interactions
• Joint actions between the user and the computer
  are designed to closely resemble joint actions
  between people

29
Digressionknowledge people accumulate about
activities in general

• Claim In any interactive activity, the knowledge
  people accumulate and are able to justify
  corresponds structurally to the three parts of
  common ground
• Knowledge initially held about the activity
• Knowledge of the activitys current state
• Knowledge of what has conspicuously taken place
  so far in the activity
• This accumulated knowledge of an activity
• Is common ground when the interaction takes place
  between people and shared informational bases
  exist to justify it as common ground
• Is only an individuals conception of the
  activity when the interaction takes place with
  elements of the environment that accrue no such
  knowledge
• Is part common ground and part individual
  conception in human-computer interaction

30
Building common ground in the secondary layer of
software use

• When the user engages the computer through a
  software user interface
• The interaction is designed to resemble a joint
  activity in a face-to-face setting
• The user actively keeps track of contingent
  interaction knowledge corresponding to three
  parts of common ground as well as informational
  bases to justify it
• At most, software keeps track of some of the same
  knowledge and shares informational bases in
  limited, ad hoc ways, such as undo mechanisms and
  other kinds of interaction histories
• Contingent interaction knowledge the software
  fails to be able to justify and use cannot be
  construed as common ground in the secondary
  layers joint activity
• Consequently, some of the users most valuable
  face-to-face language use skills go largely unused

31
Key points about language use for the design of
software user interfaces

• The written setting of software use differs from
  conventional, serially narrated written settings
  in several critical ways
• Its non-serial narrative structure builds common
  ground irregularly and introduces problems
  associated with conceptual dependencies
• Dynamic and evanescent aspects of the
  presentation and its cafeteria organization
  reduce its hard-copy utility for the user for
  purposes of reviewing the software use
• User interactions and other joint actions in
  software use are simply coordination problems
• Almost all display elements of a user interface
  should be understood as coordination devices
• Users expect all coordination problems posed by a
  design to honor the joint premises of salience,
  sufficiency, and solvability
• Strategies for the accumulation of common ground
  with the user in the secondary layer remain ad
  hoc, at best

32
Practical language use advice for user centered
design

• Focus on coordination problems from a primary
  layer perspective
• Identify coordination problems where common
  ground may be missing for the user
• Provide the user with immediate links to context
  sensitive help make it easy for the user to
  return from help to the task
• Provide explanatory drop-down labels for icons
  (tool tips) and other external representations
  do this even for cases where a terse standing
  label may be initially insufficient
• Give users a simple, point-of-interaction
  facility for annotating their software use
• Provide a hyperlinked index to the softwares set
  of functions (this can be part of a help
  mechanism)
• Provide a deep undo mechanism
• Be aware of the disorienting effect of interface
  clutter, the value of meaningful organization,
  the harm of unexplained evanescent behaviors, and
  the inevitable wrath of the user

33
Take home messages

• When the normal function of common ground in a
  language use setting is impaired, people are
  forced to work harder cognitively to accomplish
  their goals
• Designers of software user interfaces must strive
  to compensate users for a paradigm that, in
  primary layer of software use, inherently builds
  irregular common ground
• Building true common ground between the user and
  the computer in the secondary layer of software
  use awaits the future

34
Language useful references

• Brock, D. (2002). A language use perspective on
  the design of human-computer interaction.
  Proceedings Workshop on Cognitive Elements of
  Effective Collaboration. Arlington, VA Office of
  Naval Research, Code 342.
• Alterman, R. and Garland, A. (2001). Convention
  in joint activity. Cognitive Science (25)4,2001.
• Clark, H. H. (1996). Using language. New York,
  NY Cambridge University Press. (??read this if
  nothing else!)
• Hutchins, E. (1996). Cognition in the wild.
  Cambridge, MA MIT Press.
• Norman, D. A. (1992). Turn signals are the facial
  expressions of automobiles. Reading, MA
  Addison-Wesley.
• Norman, D. A. and Draper, S. W., eds. (1986).
  User centered system design. Hillsdale, NJ
  Lawrence Erlbaum.
• Winograd, T. and Flores, F. (1986). Understanding
  computers and cognition. Reading, MA
  Addison-Wesley.