Pervasive Interaction: Tools for Designing Anywhere, Anytime, Anydevice User Interfaces

1
Pervasive Interaction Tools for Designing
Anywhere, Anytime, Anydevice User Interfaces

• Prof. James A. Landay
• EECS Department
• UC Berkeley
• Microsoft Research China
• May 10, 2001

2
Pervasive Information Access
3
HCI in the Pervasive Computing Era

• Future computing devices wont have the same UI
  as current PCs
• wide range of devices
• small or embedded in environment
• often w/ alternative I/O w/o screens
• information appliances
• lots of devices per user
• all working in concert

• How does one design for this environment?
• What will these interfaces look like?

4
Future UIs for Information Access

• Star Trek style UI
• verbally ask the computer for info or services
• may be common in mobile/hands-free situations
• hard to get to work well since it requires
  perfect speech recognition unambiguous language
  understanding
• Put-that-there style UI Bolt, et. al. 80
• user says retrieve something like this while
  pointing
• combines speech w/ gesture to disambiguate
  (multimodal)
• These UIs are hard to design build
• how do we prototype and test new design ideas?

5
Pervasive Computing Design Challenges

• Designing the information architecture
• e.g., web site
• Designing cross-appliance applications
• e.g., calendar app. one speech-based one
  GUI-based
• model-based UI tools Foley, et. al. or UI
  description languages may help here, but still
  too much like programming
• Making different devices work together
• multiple devices, UIs modes
• which to present?

6
Info Access will be via Multimodal UIs

• Benefits
• use multiple modes to help disambiguate natural
  input
• computers could be used in more situations
  places
• UIs easier and useful to more people

7
Info Access will be via Multimodal UIs

• Benefits
• take advantage of more than 1 mode of
  input/output
• computers could be used in more situations
  places
• UIs easier and useful to more people
• Obstacles
• building multimodal UIs is hard
• often require immature recognition technology
• hard to combine recognition technologies
• programming expertise required to do design
• this was the state of GUIs in 1980

8
Approach Sketches Models

• Infer models from design sketches
• model is an abstraction of appliances UI design
• Use models to
• semi-automatically generate UIs
• dynamically adapt apps UI to changing context

9
Specifying Non-Visual UI Elements

• How do designers do this now?
• speech
• scripts or grammars (advanced designers only)
• flowcharts on the whiteboard
• Wizard of Oz -gt fake it!
• gestures
• give an example then tell programmer what it
  does
• We can do the same by demonstration

10
Defining zoomin in
Input modes
Output modes
Modifiers
Text
11
Defining zoomin in
Input modes
Output modes
Modifiers
Text
12
Defining zoomin in
Input modes
Output modes
Modifiers
CtrlZ
Text
13
Defining zoomin in
Input modes
Output modes
Modifiers
CtrlZ
Tools
Zoom
Text
zoom in
14
Defining zoomin in
Input modes
Output modes
Modifiers
CtrlZ
Tools
Zoom
Text
zoom in
zoom in to here
15
One multimodal action defined
Input modes
Output modes
Modifiers
Text
16
Defining Calculate Distance
Input modes
Output modes
Modifiers
Calculate distance
30 miles
30 miles
30 miles
Text
30 miles
Calculate distance
17
The Best Future Multimodal UIs that areAware of
Users Context

• Applications can be aware of
• location, user, what are they doing, who is
  nearby
• Examples
• services to discover available devices
• there is a wall display -gt use it for my wearable

18
The Best Future Multimodal UIs that areAware of
Users Context

• Applications can be aware of
• location, user, what are they doing, who is
  nearby
• Examples
• services to discover available devices
• there is a wall display -gt use it for my wearable
• choose interaction modes that dont interfere
• people are talking -gt dont rely on speech I/O
• users hands using tools -gt use speech I/O
  visual out

-gt UI design tools should understand context
support multimodal I/O
19
(No Transcript)
20
Summary

• Pervasive computing will be about pervasive
  information access using
• a variety of devices
• a variety of modalities
• UIs that are aware of the users context
• Designing for this environment is hard
• Our solution
• informal tools that use models
• infrastructure support for context

21
Pervasive Interaction Tools for Designing
Anywhere, Anytime, Anydevice Interfaces

• Prof. James A. Landay
• EECS Department, UC Berkeley
• http//guir.berkeley.edu