User Interface Agents

1

• User Interface Agents
• Roope Raisamo (rr_at_cs.uta.fi)
• Department of Computer and Information Sciences
• University of Tampere
• http//www.cs.uta.fi/rr/

2
User Interface Agents

• A user interface agent guides and helps the user
• Many user interface agents observe the activities
  of the user and suggest better ways for carrying
  out the same operations
• They can also automate a series of operations
  based on observing the users
• Many user interface agents are based on the
  principles of programming by example (PBE)

3
Two examples of user interface
agentsEagerLetizia
4
Eager automated macro generator

• Allen Cypher, 1991
• http//www.dnai.com/cypher/Eager/
• Observes the activities of the user and tries to
  detect repeating sequences of actions. When such
  a sequence is detected, offers a possibility to
  automate that task.
• like an automated macro generator
• this kind of functionality is still not a part of
  common applications, even if it could be.

5
Eager

• Eager observes repeating sequences of actions
• When Eager finds one, it jumps on the screen and
  suggests the next phase

6
Eager

• When all the phases suggested by Eager have been
  shown and accepted, the user can give Eager the
  permission to carry out the automated task.

7
Letizia a browser companion agent

• Letizia observes the user and tries to preload
  interesting web pages at the same time as the
  user browses through the web

8
Letizia
9
Letizia

• Traditional browsing leads the user into doing a
  depth first search of the Web

Letizia conducts a concurrent breadth-first
search rooted from the user's current position
10
The appearence of agents
11
The appearance of an agent

• The appearance of an agent is a very important
  feature when a user tries to find out what some
  agent can do.
• It is a bad mistake to use such an appearance
  that the user thinks an agent to be more
  intelligent than it really is.
• The appearance must not be disturbing.

12
Computer-generated talking head

• one of the most demanding forms of agent
  presentation
• a human head suggests the agent to be rather
  intelligent
• a talking head probably is the most natural way
  to present an agent in a conversational user
  interface.

13
Drawn or animated characters

• the apperance has a great effect on the
  expectations of the user
• a paper clip vs. a dog vs. Merlin the Sorceror
• Continuously animated, slowly changing or static
  presentation

14
Textual presentation

• Textual feedback of the actions of an agent
• Concerning textual user interfaces we usually
  should avoid textual input if it is not a part of
  the main task that the agent is observing.
• Chatterbots
• e.g., Julia that is a user in a MUD world. It can
  also answer to questions concerning this
  world.http//lcs.www.media.mit.edu/people/foner/Y
  enta/julia.html
• so called NPCs (non-person characters) in
  multiplayer role-playing computer games.

15
Auditory presentation

• An agent can also be presented only by voice or
  sound, the auditory channel
• ambient sound
• beeps, signals
• melodies, music
• recorded speech
• synthetic speech

16
Haptic presentation

• In addition to auditory channel, or to replace it
  an agent can present information by haptic
  feedback
• Haptic simulation modalities
• force and position
• tactile
• vibration
• thermal
• electrical

17
Haptic output devices

• Inexpensive devices
• The most common haptic devices are still the
  different force-feedback controllers used in
  computer games, for example force-feedback
  joysticks and wheels.
• In 1999 Immersion Corporations force feedback
  mouse was introduced as Logitech Wingman Force
  Feedback Gaming Mouse
• In 2000 Immersion Corporations tactile feedback
  mouse was introduced as Logitech iFeel Tactile
  Feedback Mouse

18
Haptic output devices

• More sophisticated devices
• SensAble Technologies PHANTOM
• Immersion Corporation Impulse Engine
• Often very expensive, and non-ergonomic.

PHANTOM
VTi CyberTouch
VTi CyberForce
Impulse Engine 2000
19
No direct presentation at all

• An agent helps the user by carrying out different
  supporting actions
• e.g., prefetching needed information, automatic
  hard disk management,
• An indirectly controlled background agent
• question How to implement this indirect control?
• multisensory input the agent is observing a
  system, an environment, or the user

20
Related user interface metaphorsConversational
User InterfaceMultimodal User Interface
21
Conversational User Interfaces

• Why conversation?
• a natural way of communication
• learnt at quite a young age
• tries to fix the problems of a direct
  manipulation user interface
• Conversation augments, not necessarily replaces a
  traditional user interface
• the failure of Microsoft Bob
• Microsoft Office Assistant

22
Microsoft Office Assistant

• Office assistant tries to help in the use of
  Microsoft Office programs with a variable rate of
  success.
• The user can choose the appearance of the agent
• unfortunately, this has no effect on the
  capabilities of the agent
• A paper clip most likely is a better presentation
  for the current assistant than a Merlin character.

23
Multimodal User Interfaces

• Multimodal interfaces combine many simultaneous
  input modalities and may present the information
  using synergistic representation of many
  different output modalities Raisamo, 1999

24
Multimodal User Interfaces

• An agent makes use of multimodality when
  observing the user
• speech recognition
• reacts on speech commands, or observes the user
  without requiring actual commands
• machine vision, pattern recognition
• recognizing facial gestures
• recognizing gaze direction
• recognizing gestures

25
Multimodal User Interfaces

• a specific problem in multimodal interaction is
  to combine the simultaneous inputs.
• this requires a certain amount of task knowledge
  and intelligence
• this way every multimodal user interface is at
  least in some respect a user interface agent that
  tries to find out what the user wants based on
  the available information

26
A high-level architecture for multimodal user
interfaces
Adapted from Maybury and Wahlster, 1998
27
Modeling
Nigay and Coutaz, 1993
28
Put That There
Bolt, 1980
29
Example Digital Smart Kiosk
Christian and Avery, 1998

• Smart Kiosk was a research project at
  Compaq-Digital Cambridge Research Laboratory in
  which an easy-to-use information kiosk has been
  built to be used by all people
• Combines new technology
• machine vision, pattern recognition
• speech synthesis (DECtalk)
• speech recognition
• animated talking head (DECface)

30
Example Digital Smart Kiosk
Vision
Netscape Navigator
DECface
Touchscreen
Active vision zone
31
Example Digital Smart Kiosk
32
Example Digital Smart Kiosk
33
Example Digital Smart Kiosk
34
References

• Bolt, 1980  Richard A. Bolt, Put-that-there.
  SIGGRAPH 80 Conference Proceedings, ACM Press,
  1980, 262-270.
• Christian and Avery, 1998 Andrew D. Christian
  and Brian L. Avery, Digital Smart Kiosk project.
  Human Factors in Computing Systems, CHI 98
  Conference Proceedings, ACM Press, 1998, 155-162.
• Nigay and Coutaz, 1993  Laurence Nigay and
  Joëlle Coutaz, A design space for multimodal
  systems concurrent processing and data fusion.
  Human Factors in Computing Systems, INTERCHI 93
  Conference Proceedings, ACM Press, 1993, 172-178.
• Raisamo, 1999   Roope Raisamo, Multimodal
  Human-Computer Interaction a constructive and
  empirical study. Ph.D. dissertation. Report
  A-1999-13, Department of Computer Science,
  University of Tampere.
• http//granum.uta.fi/pdf/951-44-4702-6.pdf
• Maybury and Wahlster, 1998  Mark T. Maybury and
  Wolfgang Wahlster (Eds.), Readings in Intelligent
  User Interfaces. Morgan Kaufmann Publishers,
  1998.