Handheld Devices for Control

1
Handheld Devices for Control
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University

• Brad A. MyersJeffrey Nichols, Jacob O.
  Wobbrock,Kevin Litwack, Michael Higgins, Joe
  Hughes, Thomas K. Harris, Roni Rosenfeld,
  Mathilde Pignol
• bam_at_cs.cmu.edu
• http//www.cs.cmu.edu/pebbles

2
Handheld Devices

• Handheld Devices for
• Communication
• As a Personal Information Manager
• As a Personal Digital Assistant
• As a PC replacement
• PocketPC
• Mobile Devices for Remote Control of other
  devices
• Remote Control of PCs
• Remote Control of appliances

3
Problem

• Appliances are too complex

4
Problem

• Too many remotes

5
Problem

• April 8, 1991

6
Approach

• Use a personal handheld device as an Interface to
  the PC and to the Appliances

7
History of Communication

• ParcTab continuous communication
• Early organizers no communication
• Newton extra cost for synchronization
• Palm HotSync once a day
• WindowsCE/PocketPC ActiveSync once a day

8
Now, Handhelds will be Communicating
Toshiba e740

• 802.11 (Wi-Fi)
• BlueTooth
• Cell-phone network (G3)
• (Infrared)
• (Wires or cradle)

HP iPaq 3870
Microsoft SmartPhone
9
Premises of our Research

• With the coming wireless technologies,
  connecting the PCs and PDAs together will no
  longer be an occasional event for
  synchronization. Instead, the devices will
  frequently be inclose, interactive
  communication.

• Brad Myers, Using Hand-Held Devices andPCs
  Together, Comm. ACM,Vol. 44, No. 11. Nov.,
  2001. pp. 34 - 41.

10
Pebbles Project

• Performed as part of the Pebbles Project
• Overall goal Use of multiple devices at the same
  time
• Palm and desktop PC
• Mobile Phone and Smart Home
• Multiple handhelds in a meeting
• Pocket PC and appliances
• Multiple users with their devices
• Single user with multiple devices

11
Pebbles is
DAs for ntry of oth ytes and ocations
from xternal ources.

• P
• E
• B
• B
• L
• E
• S

http//www.pebbles.hcii.edu/
12
Controlling a PC

• Remote Commander
• Slide Show Commander
• Semantic Snarfing
• Shortcutter

http//www.pebbles.hcii.edu/
13
Remote Commander

• Allow PDAs to control a PC
• Can be used with any application
• Uses the standard (single) cursor
• Dont have to jump up andgrab mouse
• All mouse and keyboard functions
• Use PDA like touchpad
• Graffiti or our own pop-upkeyboard
• Word prediction and completion

14
For People with Muscular Disabilities

• Using handhelds as interface to PCs
• People with Muscular Dystrophyhave fine-motor
  control but losegross motor control
• Difficulties with mouse and keyboard,but stylus
  OK
• So use Remote Commanderas PCs keyboard and mouse

15
PocketPC version

• Get PCs screen onto PocketPC
• Full view, or one-to-one zooming
• Scroll with iPaqs buttons
• Control or scribble

16
Shared Clipboard

• Transfer information among your personal devices
• E.g., between PDA and PC
• Also between multiple computers
• Connect all clipboards together
• Transfer content or reference
• Works with all applications

17
Power Point Control

• Use PC to give the presentation
• Use hand-held to control the PC
• Two-way communication
• Hand-held shows picture of slide,notes, timer

18
SlideShow Commander, cont.

• See list of slide titles
• Scribble
• Tap on on-screen buttonsand links
• Preview otherslides
• Controldemonstrations

19
Control at a Distance

• Controlling a PC fromacross the room
• Meetings, etc.
• Use a laser pointer?
• Studies
• Semantic Snarfing
• Grab contents to handheld

20
Shortcutter

• User-created panels of controls
• Create custom interfaces and extensions to PC
  applications
• And then take them with you
• Direct manipulationfor edit, then setproperties
• PocketPC or Palm

21
Shortcutter Widgets

• Buttons
• Virtual Joy stick
• Virtual Knob
• Sliders
• Mouse pad
• Gesture panel

22
Scenarios of Use

• Lean-back mail reading
• Controlling media players
• and many others

23
Controlling Appliances

• PhD research of Jeffrey Nichols

http//www.pebbles.hcii.edu/puc/
24
Problem

• Too many complex devices, eachwith its own
  idiosyncratic interface
• Stereo system
• Telephones
• ATM
• Fax machine
• Photocopier
• Hotel alarm clock
• Increasingly computerized
• Low usability

25
Existing universal controls

• Pre-programmed at the factory with a subset,
• Or, Laboriously hand-programmed by the user

26
Our Approach

• Handheld is a Personal Universal Controller
  (PUC)
• Two-way communication
• Appliance describes its functions
• Handheld PUC
• Automatically creates interface
• Controls the appliance
• Displays feedback about appliance status

Specifications
Control
Feedback
27
Automatic Generation of UIs

• Benefits
• All interfaces consistent for a user
• With conventions of the handheld
• Even from multiple manufacturers
• Addresses hotel alarm clock problem
• Can take into account user preferences
• Multiple modalities (GUI Speech UI)
• A Hard Problem
• Previous automatic systems have not generated
  high quality interfaces

28
Research Challenges

• Automatic Design of Graphical User Interfaces
• Automatic Design of Speech User Interfaces
• Connection with real devices
• Through various protocols
• X-10, AV/C, HAVi, UPnP, etc.
• Also, custom hardware and software

29
Hand-Generated Graphical Interfaces

• First, Hand-Designed PocketPC interfaces
• AIWA Shelf Stereo (Tape,CD,Tuner)
• ATT Telephone/AnsweringMachine
• Used Embedded Visual Basic
• Ensured quality with heuristicanalysis and
  think-aloud studies
• Compared with manufacturersinterfaces

30
Results of Comparison

• Using PUC, users took 50 less time made 50
  fewer errors
• All differences are significant (p lt 0.05)

31
Current PUC Specification Language

• XML
• Full documentation for the specification language
  and protocol
• http//www.pebbles.hcii.cmu.edu/puc/
• Contains sample specification for a stereo

32
Properties of PUC Language

• State variables commands
• Each can have multiple labels
• Useful when not enough room
• Typed variables
• Base types Boolean, string,enumerated,
  integers,fixed-point, floating-point, etc.
• Optional labels for values
• Hierarchical Structure
• Groups

33
Dependency Information

• Crucial for high-quality interfaces
• Expressed as ltactive-ifgt clauses
• Operations
• Equals, Less-Than,Greater-Than
• Combined Logically
• AND, OR
• Used for
• Dynamic graying out
• Layout
• Widget selection

34
Specifications

• Have working specifications for
• Audiophase stereo
• X-10 lights control
• Sony CamCorder
• Windows Media Player
• Audio ReQuest hardware MP3 player
• WinAmp Media Player
• Elevator

35
Examples of Generated GUIs

• Stereo and X-10

36
More Examples

• Elevator
• Outside Inside

37
Generating Speech Interfaces

• Universal Speech Interface (USI) project
• Prof. Roni Rosenfeld of CMU
• http//www.cs.cmu.edu/usi
• Creates grammar, language model and pronunciation
  dictionary from PUC specification
• Pronunciation from labels using phonetic rules
• Can provide other pronunciations as labels for
  fine-tuning
• Will use dependency information to help with
  disambiguation and explanation
• Supports queries and spoken feedback
• Paraphrases as confirmation

38
Adaptors

• Adaptors provide the interface to existing (and
  future) appliances
• If do not support specification language directly
• Custom hardware
• Audiophase Stereo
• Custom software
• ARQ MP3 player
• SIMA MP3 player (future)
• X-10
• Light switches, etc.

• AV/C (standard protocol)
• Sony CamCorder
• PlayStation, etc. (future)
• HAVi (current work)
• Mitsubishi HDTV 65 TV
• Mitsubishi HDTV VCR
• UPnP (future)
• ??

39
Adaptor for AV/C
PUC Java Proxy
PUC Java Library
AutomaticGUIgeneration
AutomaticSpeech UIgeneration
Hand-written spec of AV/C devices
Our Java device code
Our C code
Multiple devicecontrollers
Microsoft DirectShowlibrary
AV/C protocol
Firewire cable(IEEE 1394)
Digital VideoCamera or VCR
40
Demonstration

• X-10
• Camcorder through AV/C protocol forIEEE 1394
  (Firewire)
• Two way communication
• When state changes from appliance, GUI is updated
  and can query state with speech

41
Future Work on PUC

• Controlling User Experiences
• One command for multiple devices
• Play a DVD ? sets DVD, cable, TV
• Setting alarm clock ? sets heat, coffee, car
• Consistency for Users
• And user preferences
• More devices
• General Motors automobiles
• Etc.

42
For More Information

• Many papers on the Pebbles web site
• On PUC and other work
• Most programs available for free downloading
• SlideShow Commander waslicensed for commercial
  sale

http//www.cs.cmu.edu/pebbles
?
http//www.slideshowcommander.com
43
Conclusions

• As more and more electronics are computerized and
  communicating, mobile devices can help control
  them
• Handheld devices can improve the user interfaces
  of everything else
• Not just be another gadget to be learned

44
Thanks to Pebbles Students

• This research performed by a large number of
  students since 1997

Rishi Bhatnagar Ben Bostwick Franklin Chen Yu Shan Chuang Karen Cross Carl Evankovich Marc Khadpe Dave Kong Chun-Kwok Lee Joonhwan Lee Jennifer Li Leo Lie Jack Lin Kevin Litwack A. Chris Long Rob Miller Jeff Nichols Choon Hong Peck Mathilde Pignol Herbert Stiel Marsha Tjandra Adrienne Warmack Jake Wobbrock Jerry Yang Sunny Yang Brian Yeung
45
Thanks to Our Sponsors!

• The Pebbles research project is supported by
  grants from
• DARPA
• Microsoft
• NSF
• Pittsburgh Digital Greenhouse
• General Motors
• And equipment grants from
• Hewlett Packard
• Lucent Technologies
• Mitsubishi
• Palm Computing
• Symbol Technologies

• IBM
• SMART Technologies
• VividLogic
• Synergy Solutions
• Handango

46
Handheld Devices for Control
Thank You!
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University

• Brad A. MyersJeffrey Nichols, Jacob O.
  Wobbrock,Kevin Litwack, Michael Higgins, Joe
  Hughes, Thomas K. Harris, Roni Rosenfeld,
  Mathilde Pignol
• bam_at_cs.cmu.edu
• http//www.cs.cmu.edu/pebbles

47
Black, followed by OLD
48
Discussion of Comparison

• Our hand designed interfaces succeeded for
  several reasons
• Good organization
• Each button has one function
• Good labels
• Only available functions are active
• Others, hidden on tabs or grayed out
• Better feedback and error messages

49
PUC Architecture
50
Adaptor using Custom Hardware for Stereo

• Pretends to send IR codes
• Reads LED panel signals to decode state
• Created by Pittsburgh company Maya Design