Sensing Techniques for Mobile Interaction

1
Sensing Techniques for Mobile Interaction

• Ken Hinckley
• Jeff Pierce
• Mike Sinclair
• Eric Horvitz
• Attentional User Interfaces Project
• Microsoft Research

2
Mobile Devices vs. Mr. Cleo (An Informal
Comparison)

• How can we make Smart Computers?

• Mr. Cleo
• Aware of sounds, objects
• Knows if I walk in the room
• Selfish Inconsiderate

• My Mobile Device
• Unaware of environment
• Oblivious to my presence
• Selfish Inconsiderate

3
Sensitive Interfaces(a.k.a. Not-So-Stupid
Computing)

• Computers are very fast idiots. They are
  oblivious to the external world.
• What is a smart computer anyway???
• Can we build one? Do you really want one?
• Or one that is aware, respectful, Not So
  Stupid?
• Wed be better off if computers were as dumb as
  my cat
• Can some very dumb sensors, with straightforward
  software, help address this problem?
• Design hardware/software interfaces that are
  sensitive to the user and the surrounding
  physical environment.

4
Sensing for User Interaction

• Sense more than just explicit commands
• Simplify the interface using background
  informationthat is already there
• PointShoot Cameras

• One Button
• Many sensors

5
Can Background Sensing Enhance Mobile UI?

• Real world has high cognitive / attentional
  demand
• Even clicking a button can be hard!
• Software ignorant of the changing physical
  context
• Device should adapt to current task / situation
• Naturally occurring gestures of use are missed
• Pick up, put down, look at, walk around with
• Explosion of cheap, informative sensors
• On device or available via wireless network

6
Mobile Sensor Prototype (Casio E105)

• Tilt Sensor
• Proximity Range Sensor
• Touch Sensor
• Demos implemented as Windows CE applications
• Sensor I/O via PIC micro.
• VIDEO

7
Tilt Sensor

• Analog Devices 2-axis linear accelerometer
• Tilt relative to gravity
• But, other accelerations also affect signal
• Limitations
• Cannot sense rotation around vertical axis
• Cannot tell up from down

8
Activity Detection Tilt Sensor Example
Data
Looking at display
Walking to elevator
Walking to meeting
PocketPC held at side
9
Voice Memo Detector

• Gesture allows general-purpose PIM to have
  special-purpose context of use
• Sensor Fusion

• Must be holding device
• Tilt like a phone
• Hold close to face
• Audio is crucial to the interaction
• Relaxing stops recording

10
Voice MemoWorkflow Analysis

• I have to think about finding the button,
  pushing .it, holding it
• It was just listen for the beep

11
Sensed Memo Recording Usability Issues

• talk into it like a cell phone is enough
  instruction to use
• Quite a bit easier, I can focus
• Would use it more if it worked that way
• 6/7 Ss preferred sensed gesture to button (4.3)

• Sensed gesture not easily discoverable
• disorienting to put up to my face to talk - 1
  Ss
• False positives can occur e.g. putting into
  sweater pocket
• But, button also has false positives

12
Tracking Experiment

• Not faster, but less demand on visual attention.
• User tracks fly on monitor, using mouse
• Records Testing 123
• (S, M) gt C (plt.001)
• Sensed gesture may require less visual attention
  (p lt 0.1)

13
Portrait / Landscape Display Mode Detection

• Snow Globe
• Input controls rotated to match screen
• Easy (5.0/5.0).
• 6 Ss prefer tilting to menu 1 Ss I think it
  would drive me nuts
• Sharing w/others
• Doesnt interrupt

14
Portrait / Landscape Detection Implementation
Tilt vs. sensed orientation

• 5 display modes
• 2 Portrait
• 2 Landscape
• Flat
• Dead bands, 0.5s keep screen stable
• Put-down problem FIFO queue, look for stable
  orient.

15
Tilt for Scrolling

• Touch screen bezel to clutch
• Easy to activate / maintain state
• Too easy to hit by accident (landscape mode)
• Sets resting orientation
• Hides on-screen UI (menu, taskbar) during
  scrolling
• Several transfer functions possible
• Rate ctrl v K sgn(dA) ? max(dA - dAmin,
  0)?
• Single axis only move along 1 primary axis at a
  time
• Dual axis full panning in any direction
• Mixed axis panning, but with affinity for
  primary axes

16
Tilt Scrolling Then Some

• Contrast compensation
• Equalize apparent contrast
• Scrolling Portrait/Landscape
• Dont change P/L modes while scrolling!
• Dont change when stop scrolling, either!
• Waits to see different orientation
• Not quite right should switch after longer dwell
  (2-3 s?)
• User Testing 5 Ss, compared to built-in
  direction pad
• Good way to scroll the screen Agree (4.8 /
  5.0)
• Id rather use the direction pad Disagree
  (1.8 / 5.0)
• One-handed operation, natural, simplifies the
  movement

17
Power Management

• Automatic power-on
• Holding device in hand
• AND looking at display (flat in one axis, tilted
  fwd 20o in the other) for 0.5s
• Cant power up in purse / pocket
• Wont power up if you just touch it to push away
• Wont power up if you just grab hold at side
  (usually ?)
• There is no auto-power-OFF feature, by design
• Best case turns off user doesnt even notice
  feature
• Touch, tilt proximity sensors prevent power-off
  or screen dimming while using device

18
Other Fun Stuff

• Games
• Proximity
• For zooming fun demo, but how to select?
• Proximal UI Creepy
• Hand of frustration
• Shaking
• To switch applications (top app ? bottom)
• Turn upside down shake to erase -)
• Impl. not robust enough to turn on all the time

19
Conclusions

• New UIs with better sensitivity to the user
  the users physical environs
• Great potential to simplify enhance the UI
• ? new behaviors and services that users find
  compelling, useful, engaging, respectful
• Design, Implementation, Usability challenges
• Design must handle false positive / false
  negative cases
• Not a panacea. Only seems helpful for some tasks.
  
• Making things simpler vs. Loss of explicit
  control
• Promising area that needs more work

20
(No Transcript)
21
Proximity Sensor

• IR LED, 60o, 40kHz
• IR receiver (used in TVs)
• Gain?distance
• Some light sensitivity, e.g. sunlight

22
Touch Sensor

• Mainly so we know whenuser is holding device
• Useful to disambiguate intentional gestures
  from accidental ones
• holding and tilting, vs.
• tilted while sitting in briefcase
• Also experimented with touch buttonsaround
  screen bezel

23
Software Architecture

• Context Server whiteboard
• Shared memory to read variables
• Or, ask for Windows messages to notify of changes
• Apps can post any synthesized info back to server
• Some example context variables (events)

• Holding, Duration
• TiltAngleLR / FB
• DisplayOrientation
• Walking

• Proximity (z)
• ProximityState
• Scrolling
• VoiceMemoGesture

24
Screen Real Estate Optimization

• Touch bezel for explicit controlover tool bars
• Proximity of hand to screen?
• auxiliary UI appears when hand gets close