Dialog design

1
Dialog design

• How do we communicate with computers?

2
Agenda

• Syllabus modification
• Part 1 due next week!
• Interaction modalities and styles

3
Dialog Design

• How does a user interact with the interface?

4
Interaction Styles

• 1. Command languages
• 2. WIMP - Window, Icon, Menu, Pointer
• 3. Direct manipulation
• 4. Speech/Natural language
• 5. Pen Mobile

5
General Issues in Choosing Dialogue Style

• Who is in control - user or computer
• Initial training required
• Learning time to become proficient
• Speed of use
• Generality/flexibility/power
• Special skills - typing
• Screen space required
• Computational resources required

6
Command Languages

• Earliest UI interaction paradigm
• Examples MS-DOS shell, UNIX, Linux
• Little or nothing is visible so
• Work primarily by recall, not recognition
• Heavy memory load
• Poor choice for novices but...

7
CL Advantages

• Advantages for experts
• Speed, conciseness
• ls (hard to beat)
• Can express actions beyond a limited set
• Flags, piping one command to another
• Repetition, extensibility
• Scripting, macros
• Easier implementation, less overhead
• Power
• Abstraction, wild cards

8
CL Dangers

• With added power, comes added responsibility and
  danger
• UNIX
• rm -r
• Deletes every file that you have, and you cant
  get them back

9
CL Design Goals

• Consistency
• Have options and arguments expressed the same way
  everywhere
• Good naming and abbreviations
• UNIX fails here because commands were developed
  by lots of different people at different
  organizations
• No guidelines provided

10
General Issues - CL

• Who is in control - user or computer
• Initial training required
• Learning time to become proficient
• Speed of use
• Generality/flexibility/power
• Special skills - typing
• Screen space required
• Computational resources required

11
Direct Manipulation

• Definition
• 1) Continuous visibility of the objects and
  actions of interest
• 2) Rapid, reversible, incremental actions whose
  effect is immediately noticeable
• 3) Replacement of command language syntax by
  direct manipulation of object of interest
  (physical actions, buttons, etc.)

Shneiderman 82
12
DM Essence

• Representation of reality that can be manipulated
• The user is able to apply intellect directly to
  the task
• The tool itself seems to disappear
• Related to
• Directness
• Engagement

13
Direct Manipulation

• See pgs 171-174 in DFAB
• Examples
• WYSIWYG editors and word processors
• VISICALC - 1st electronic spreadsheet
• CAD
• Desktop metaphor
• Video games

14
Example Homefinder
15
DM Issues

• Advantages
• Flexible, easily reversible actions helps reduce
  anxiety in users
• Easy to learn and remember
• WYSIWYG provides context instant visual
  feedback
• Exploits human use of visual spatial cues
• Limits types of errors that can be made

• Disadvantages
• visual representation not self-explanatory, no
  prompts
• Mouse ops may be slower than typing
• Screen space intensive
• Not good at
• Repetition
• History keeping
• Certain tasks (Change all italics to bold)
• Abstract elements (variables)

16
General Issues - DM

• Who is in control - user or computer
• Initial training required
• Learning time to become proficient
• Speed of use
• Generality/flexibility/power
• Special skills - typing
• Screen space required
• Computational resources required

17
WIMP

• Windows, Icons, Menus, Pointers
• Focus Menus, Buttons, Forms
• Predominant interface paradigm now (with some
  direct manipulation added)
• Advantages
• ?

18
Menus

• Advantages
• 1 keystroke or mouse operation vs. many
• No memorization of commands
• Limited input set
• Disadvantages
• Less direct user control - have to find correct
  menu / menu item
• Not so readily extensible
• Slower than keyboarding for experienced users, at
  least without accelerators

19
Menu Items

• Various types
• Flat
• Cascading
• Pop-up (contextual)
• Organization strategies
• Create groups of logically similar items
• Cover all possibilities
• Ensure that items are non-overlapping
• Keep wording concise, understandable

20
A Good Example

• Logical grouping
• Visual separation of groups
• Disabled items grayed out
• Shortcuts shown
• indicates leads to dialogue

21
Presentation Sequence

• Forms, dialogue boxes, menus
• Use natural if available
• Time
• e.g. Breakfast, Lunch, Dinner
• Numeric ordering
• e.g. Point sizes for font
• Other possibilities
• Alphabetical
• Group related items
• Frequently used first
• Most important first

22
Pointers

• Allows users to specify location parameter of a
  command
• Provides for some of that direct manipulation
• Put this file HERE
• Also displays system state info
• Tracking
• Busy
• Hints
• Modes (text cursor vs. selection pointer)

23
Other WIMP issues

• Windows management
• How to locate, move, find
• Transfer information between
• Switch attention between
• Icons
• Need graphic design attention
• Toolbars

24
Newer icons
25
Simple icons
26
Simple icons
27
General Issues - WIMP

• Who is in control - user or computer
• Initial training required
• Learning time to become proficient
• Speed of use
• Generality/flexibility/power
• Special skills - typing
• Screen space required
• Computational resources required

28
Natural input

• Universal design
• Take advantage of familiarity, existing knowledge
• Alternative input output
• Multi-modal interfaces
• Getting off the desktop

29
Speech dialogue

• Why use it?
• Hands busy
• Mobility required
• Eyes occupied
• Conditions preclude use of keyboard
• Visual impairment
• Physical limitation

30
Speech Input

• Speaker recognition
• Tell which person it is (voice print)
• Monitoring, recording
• Speech recognition
• Identify words
• IBM ViaVoice, Dragon Dictate, ...
• Natural language understanding
• (does not necessarily involve audio)

31
Recognition Dimensions

• Speaker dependent/independent
• Parametric patterns are sensitive to speaker
• With training (dependent) can get better
• Vocabulary
• Some have 50,000 words
• Isolated word vs. continuous speech
• Continuous where words stop begin
• Typically a pattern match, no context used

Did youvs. Didja
32
Recognition Example

• My flight experience uh,
• I started off as a private
• pilot and then I spent 20
• years flying as a navigator
• on C130s in the air force
• and I currently fly as a
• first officer on the SAAB
• 340 turboprop.

• my flight experience a and
• I started off as a private
• pilot and I spent 20
• years flying is a navigator
• on C-130s in the Air Force
• and I currently fly as a
• first officer on the sound
• 340 turboprop.

33
And another example

• Here here's one of the
• thoughts I'd throw in on
• that. I'd agree with
• Howard that the
• combination of checklists
• and flows is the way way
• to manage things best.

• hair and here's one of the
• outside troodon and
• an eye to agree with
• power that the
• domination checklists
• and flows is the way
• that managed things best

34
Errors

• Systems make four types of errors
• Substitution - one for another
• Rejection - detected, but not recognized
• Insertion - added
• Deletion - not detected
• So how do you recover from an error?

35
Natural Language Understanding

• Putting meaning to the words
• Input might be spoken or typed
• Holy grail of Artificial Intelligence problems

36
Eliza - a Faked Understanding

• Person Men are all alike.ELIZA In what
  way?Person They're always bugging us about
  something or other.ELIZA Can you think of a
  specific example?Person Well, my boyfriend
  made me come here.ELIZA Your boyfriend made
  you come here?Person He says I'm depressed
  much of the time.ELIZA I am sorry to hear that
  you are depressed.Person It's true. I am
  unhappy.ELIZA Do you think coming here will
  help you not to be unhappy?Person I need some
  help, that much seems certain.ELIZA What would
  it mean to you if you got some help?Person
  Perhaps I could learn to get along with my
  mother.ELIZA Tell me more about your family
• http//www-ai.ijs.si/eliza/eliza.html
• Weizenbaum, J., "ELIZA -- A computer program for
  the study of natural language communication
  between man and machine", Communications of the
  ACM 9(1)36-45, 1966

37
NL Factors/Terms

• Syntactic
• Grammar or structure
• Prosodic
• Inflection, stress, pitch, timing
• Pragmatic
• Situated context of utterance, location, time
• Semantic
• Meaning of words

38
SR/NLU Issues

• Advantages
• Easy to learn and remember
• Powerful
• Fast, efficient (not always)
• Little screen real estate

• Disadvantages
• Assumes domain knowledge
• Doesnt work well enough yet
• Requires confirmation
• And recognition will always be error-prone
• Expensive to implement
• Unrealistic expectations can generate mistrust

39
Speech Output

• Tradeoffs in speed, naturalness and
  understandability
• Male or female voice?
• Technical issues (freq. response of phone)
• User preference (depends on the application)
• Rate of speech
• Technically up to 550 wpm!
• Depends on listener
• Synthesized or Pre-recorded?
• Synthesized Better coverage, flexibility
• Recorded Better quality, acceptance

40
Speech Output

• Synthesis
• Quality depends on software ()
• Influence of vocabulary and phrase choices
• http//www.research.att.com/ttsweb/tts/demo.phpt
  op
• Recorded segments
• Store tones, then put them together
• The transitions are difficult (e.g., numbers)

41
Designing Speech Interaction

• Constrain vocabulary
• Limit valid commands
• Structure questions wisely (Yes/No)
• Manage the interaction
• Examples?
• Slow speech rate, but concise phrases
• Design for failsafe error recovery
• Visual record of input/output
• Design for the user Wizard of Oz

42
Speech Tools/Toolkits

• Java Speech SDK
• FreeTTS 1.1.1 http//freetts.sourceforge.net/docs/
  index.php
• IBM JavaBeans for speech
• Microsoft speech SDK (Visual Basic, etc.)
• OS capabilities (speech recognition and synthesis
  built in to OS) (TextEdit)
• VoiceXML

43
General Issues Speech/NL

• Who is in control - user or computer
• Initial training required
• Learning time to become proficient
• Speed of use
• Generality/flexibility/power
• Special skills - typing
• Screen space required
• Computational resources required

44
Non-speech audio

• Good for indicating changes, since we ignore
  continuous sounds
• Traditionally used for warnings, alarms or status
  information
• Provides secondary representation
• Supports visual interface
• Provides information that helps reduce error
• Tradeoff in using natural (real) sounds vs.
  synthesized noises.

45
Non-speech audio examples

• Error ding
• Info beep
• Email arriving ding
• Recycle
• Battery critical
• Logoff
• Logon
• Others?

46
Pen, Touch, Mobile interaction
47
Pen, Touch, Mobile dialog

• Stylus or finger
• Tradeoffs of each?
• Pen as a standard mouse (doubleclick?)
• Variety of platforms
• Desktop touch screens or input pads (Wacom)
• Tablet PCs
• Handheld and Mobile devices
• Electronic whiteboards
• Platforms often involve variety of size and other
  constraints

48
Mobile devices

• More common as more platforms available
• PDA
• Cell phone
• Ultra mobile tablets
• GPS
• Smaller display (160x160), (320x240)
• Few buttons, different interactions
• Free-form ink
• Soft keyboard
• Numeric keyboard gt text
• Stroke recognition
• Hand printing / writing recognition

49
http//www.blackberry.com/
http//www.oqo.com/
50
Soft Keyboard

• Presents a small diagram of keyboard
• You click on buttons/keys with pen
• QWERTY vs. alphabetical
• Tradeoffs?
• Alternatives?

51
Numeric Keypad

• You press out letters of your word, it matches
  the most likely word, then gives optional choices
• Faster than multiple presses per key
• Used in mobile phones
• http//www.t9.com/

52
Other pen text input

• Graffiti Palm mobile devices
• Unistroke recognition
• Experimental
• Cirrin
• World level unistroke

• Quickwriting
• Harder to learn than graffiti

53
Hand Printing / Writing Recognition

• Recognizing letters and numbers and special
  symbols
• Lots of systems (commercial too)
• English, kanji, etc.
• Not perfect, but people arent either!
• People - 96 handprinted single characters
• Computer - gt97 is really good

54
Recognition Issues

• Boxed vs. Free-Form input
• Sometimes encounter boxes on forms
• Printed vs. Cursive
• Cursive is much more difficult
• Letters vs. Words
• Cursive is easier to do in words vs individual
  letters, as words create more context
• Usually requires existence of a dictionary
• Real-time vs. off-line

55
Pen Gesture Commands

• Might mean delete
• Insert
• Paragraph

Define a series of (hopefully) simple drawing
gesturesthat mean different commands in a system
56
Pen Use Modes

• Often, want a mix of free-form drawing and
  special commands
• How does user switch modes?
• Mode icon on screen
• Button on pen
• Button on device

57
Error Correction

• Having to correct errors can slow input
  tremendously
• Strategies
• Erase and try again (repetition)
• When uncertain, system shows list of best guesses
  (n-best list)
• Others??

58
Free-form Ink

• Ink is the data, take as is
• Human is responsible forunderstanding
  andinterpretation
• Often time-stamped
• Applications
• Signature verification
• Notetaking
• Electronic whiteboards
• Sketching

59
Electronic whiteboards

• Smartboard and Mimio
• Can integrate with projection
• Large surface to interact with
• Issues?

http//www.mimio.com/
http//www.smarttech.com/
60
Touch tables

• Which techniques might be similar to smaller
  touchscreens?
• Which would differ?
• How similar and different from interactive white
  boards?

Microsoft Surface
61
Real paper

• Anoto digital paper and pen technology
  (http//www.anoto.com/)
• Other pens available
• Issues?

http//www.logitech.com/
http//www.epos-ps.com/
62
General Issues Pen input

• Who is in control - user or computer
• Initial training required
• Learning time to become proficient
• Speed of use
• Generality/flexibility/power
• Special skills - typing
• Screen space required
• Computational resources required

63
Other interesting interactions

• Gesture input
• Wii
• Lots of other specialized hardware for tracking
• 3D interaction
• Stereoscopic displays
• Virtual reality
• Immersive displays such as glasses, caves
• Augmented reality
• Head trackers and vision based tracking
• Tangible interaction
• Use physical objects to express input