Ink, Audio and Video

1
Working with Natural Data Types

• Ink, Audio and Video

2
Agenda

• Questions
• Project selection
• Natural data types
• Pen, Audio, Video
• As data vs. Recognition

3
Project Selection
4
Natural Data Types

• As we move off the desktop, means of
  communication mimic natural human forms of
  communication
• Writing, Speaking, Seeing
• Ink, Audio, Video
• Understanding these data types leads to
  interesting applications
• e.g., capture applications

5
Uses of Natural Data Types

• Recognition
• Translate into other information (words or
  description of scene)
• Dealing with errors
• Raw data
• as data use for human consumption (to read,
  hear or see)
• Often linked with other context (time) in capture
  applications

6
Pen Computing

• Use of pens has been around a long time
• Light pen was used by Sutherland before Engelbart
  introduced the mouse
• Resurgence in 90s
• Types of pens
• Passive (same as using a finger)
• Active (pen provides some signal)

7
Example Pen Technology

• Passive
• Touchscreen (e.g., PDA, some tablets)
• Contact closure
• Vision techniques
• Active
• Pen emits signal(s)
• e.g. IR ultrasonic
• Where is sensing? Surface or pen

8
Questions about Pens

• What operations detectable
• Contact up/down
• Drawing/Writing
• Hover?
• Modifiers? (like mouse buttons)
• Which pen used?
• Eraser?
• Difference between pen and mouse.

9
Example Expansys Chatpen

• Reads dot pattern on paper
• Transmits via Bluetooth
• http//www.expansys.com/product.asp?codeERIC_CHAT
  PEN

10
Example mimio

• Active pens
• IR ultrasonic
• Portable sensor
• Converts any surfaceto input surface
• We have chained theseto create big surface
• http//www.mimio.com

11
Pen input

• Free-form ink
• Soft keyboards
• Recognition systems
• generalize to gesture-based systems

12
Toolkits for Pen-Based Interfaces

• UC Berkeley SATIN toolkit (UIST 2000)
• http//guir.berkeley.edu/projects/satin
• MS TabletPC
• Descendant of MS Windows for Pen Computing
• MS Pocket PC, CE.net
• Apple Newton OS
• GO PenPoint
• Palm Developer environments
• GDT (Long, Berkeley) Java-based trainable
  unistroke gesture recognizer
• OOPS (Mankoff, GT) error correction based on
  subArctic

13
SATIN (UIST 2000)

• Pen input for informal input
• Sketching (others have investigated this)
• Common toolkit story
• Gee, X sure is a neat class of apps!
• Golly, making X apps is tough!
• Heres a toolkit to build X things easily!

14
The SATIN Toolkit

• The application space
• Informal ink apps
• Beyond just recognition
• Pen look-and-feel
• Abstractions
• Recognizers
• Interpreters
• multi-interpreters

15
Audio

• Natural appeal of sound
• Off-the desktop
• hands-free
• display not necessary
• effective for a short distance

16
How sound is created

• Sound is created when air is disturbed, usually
  by vibrating object(s) causing ripples of varying
  air pressure propagated by the collision of air
  molecules.

17
How sound is perceived

• Characteristics of sound
• Volume
• Pitch
• Features of sound wave
• Amplitude
• Volume amplitude of a sound wave
• Frequency
• Pitch frequency of a single sound wave

18
Example
19
When multiple sounds occur
20
Sampling sound

• Note sampling rate affects accurate
  representation of the sound wave

21
Acquiring audio
22
Generating audio
23
Features in speech
24
Audio features

• Pauses
• Analogy to mouse up/down/drag
• Who is speaking
• Turn-taking
• How is someone speaking
• Prosody, affect
• What is being said
• Recognition of words

25
Challenges with Audio

• Provide support for handling audio in manner
  similar to text
• Simplify acquisition, segmenting, storing,
  playback
• Open air audio (in/out) results in messy audio
  from ambient noise and feedback
• Support noise reduction, enhance audio quality,
  echo cancellation

26
Noise reduction

• Random noise is hard to predict

Noise Filter
f (t)
f(t)
27
Echo cancellation

• Software hardware exist, but are hard for
  developers to easily add to application
• Random noise is hard to predict, but echoes are
  not so random

Echo Canceller
f(t)
f (t)
28
Other interesting audio work

• Auditory icons or earcons
• sound that identifies objects actions
• Automatic transformation of GUI interfaces to
  auditory interfaces
• Mercator (Mynatt, 1995)
• Sound localization sound cones

29
Creating UIs with audio

• Audio
• Java Media Framework
• http//java.sun.com/products/java-media/jmf/
• Speech
• UC Berkeley SUEDE toolkit
• http//guir.berkeley.edu/projects/suede/
• CSLU toolkit
• http//cslu.cse.ogi.edu/toolkit/