Ink, Audio and Video - PowerPoint PPT Presentation

1 / 29
About This Presentation
Title:

Ink, Audio and Video

Description:

As we move off the desktop, means of communication mimic 'natural' human forms of communication ... Hover? Modifiers? ( like mouse buttons) Which pen used? Eraser? ... – PowerPoint PPT presentation

Number of Views:28
Avg rating:3.0/5.0
Slides: 30
Provided by: georgep3
Category:
Tags: audio | hover | ink | satin | video

less

Transcript and Presenter's Notes

Title: Ink, Audio and Video


1
Working with Natural Data Types
  • Ink, Audio and Video

2
Agenda
  • Questions
  • Project selection
  • Natural data types
  • Pen, Audio, Video
  • As data vs. Recognition

3
Project Selection
4
Natural Data Types
  • As we move off the desktop, means of
    communication mimic natural human forms of
    communication
  • Writing, Speaking, Seeing
  • Ink, Audio, Video
  • Understanding these data types leads to
    interesting applications
  • e.g., capture applications

5
Uses of Natural Data Types
  • Recognition
  • Translate into other information (words or
    description of scene)
  • Dealing with errors
  • Raw data
  • as data use for human consumption (to read,
    hear or see)
  • Often linked with other context (time) in capture
    applications

6
Pen Computing
  • Use of pens has been around a long time
  • Light pen was used by Sutherland before Engelbart
    introduced the mouse
  • Resurgence in 90s
  • Types of pens
  • Passive (same as using a finger)
  • Active (pen provides some signal)

7
Example Pen Technology
  • Passive
  • Touchscreen (e.g., PDA, some tablets)
  • Contact closure
  • Vision techniques
  • Active
  • Pen emits signal(s)
  • e.g. IR ultrasonic
  • Where is sensing? Surface or pen

8
Questions about Pens
  • What operations detectable
  • Contact up/down
  • Drawing/Writing
  • Hover?
  • Modifiers? (like mouse buttons)
  • Which pen used?
  • Eraser?
  • Difference between pen and mouse.

9
Example Expansys Chatpen
  • Reads dot pattern on paper
  • Transmits via Bluetooth
  • http//www.expansys.com/product.asp?codeERIC_CHAT
    PEN

10
Example mimio
  • Active pens
  • IR ultrasonic
  • Portable sensor
  • Converts any surfaceto input surface
  • We have chained theseto create big surface
  • http//www.mimio.com

11
Pen input
  • Free-form ink
  • Soft keyboards
  • Recognition systems
  • generalize to gesture-based systems

12
Toolkits for Pen-Based Interfaces
  • UC Berkeley SATIN toolkit (UIST 2000)
  • http//guir.berkeley.edu/projects/satin
  • MS TabletPC
  • Descendant of MS Windows for Pen Computing
  • MS Pocket PC, CE.net
  • Apple Newton OS
  • GO PenPoint
  • Palm Developer environments
  • GDT (Long, Berkeley) Java-based trainable
    unistroke gesture recognizer
  • OOPS (Mankoff, GT) error correction based on
    subArctic

13
SATIN (UIST 2000)
  • Pen input for informal input
  • Sketching (others have investigated this)
  • Common toolkit story
  • Gee, X sure is a neat class of apps!
  • Golly, making X apps is tough!
  • Heres a toolkit to build X things easily!

14
The SATIN Toolkit
  • The application space
  • Informal ink apps
  • Beyond just recognition
  • Pen look-and-feel
  • Abstractions
  • Recognizers
  • Interpreters
  • multi-interpreters

15
Audio
  • Natural appeal of sound
  • Off-the desktop
  • hands-free
  • display not necessary
  • effective for a short distance

16
How sound is created
  • Sound is created when air is disturbed, usually
    by vibrating object(s) causing ripples of varying
    air pressure propagated by the collision of air
    molecules.

17
How sound is perceived
  • Characteristics of sound
  • Volume
  • Pitch
  • Features of sound wave
  • Amplitude
  • Volume amplitude of a sound wave
  • Frequency
  • Pitch frequency of a single sound wave

18
Example
19
When multiple sounds occur
20
Sampling sound
  • Note sampling rate affects accurate
    representation of the sound wave

21
Acquiring audio
22
Generating audio
23
Features in speech
24
Audio features
  • Pauses
  • Analogy to mouse up/down/drag
  • Who is speaking
  • Turn-taking
  • How is someone speaking
  • Prosody, affect
  • What is being said
  • Recognition of words

25
Challenges with Audio
  • Provide support for handling audio in manner
    similar to text
  • Simplify acquisition, segmenting, storing,
    playback
  • Open air audio (in/out) results in messy audio
    from ambient noise and feedback
  • Support noise reduction, enhance audio quality,
    echo cancellation

26
Noise reduction
  • Random noise is hard to predict

Noise Filter
f (t)
f(t)
27
Echo cancellation
  • Software hardware exist, but are hard for
    developers to easily add to application
  • Random noise is hard to predict, but echoes are
    not so random

Echo Canceller
f(t)
f (t)
28
Other interesting audio work
  • Auditory icons or earcons
  • sound that identifies objects actions
  • Automatic transformation of GUI interfaces to
    auditory interfaces
  • Mercator (Mynatt, 1995)
  • Sound localization sound cones

29
Creating UIs with audio
  • Audio
  • Java Media Framework
  • http//java.sun.com/products/java-media/jmf/
  • Speech
  • UC Berkeley SUEDE toolkit
  • http//guir.berkeley.edu/projects/suede/
  • CSLU toolkit
  • http//cslu.cse.ogi.edu/toolkit/
Write a Comment
User Comments (0)
About PowerShow.com