Lecture 10: Advanced Input

1
Lecture 10Advanced Input
2
GUI Input Modalities

• How can a user provide input to the GUI?
• keyboard
• mouse, touch pad
• joystick
• touch screen
• pen / stylus
• speech
• other

3
Pen input

• Pen-based interfaces in mobile computing

4
Pen input

• Handwriting
• very general, well-developed human skill
• thus, make use of what users can already do!
• but hard to recognize (for people machines)
• Gestures
• gesture alphabets
• Palm Pilot graffiti
• editing gestures
• easier to recognize

5
Speech input

• Limited speech recognition
• only allow small sets of words/phrases
• e.g., one nine for phone menus
• Full speech recognition
• again, general, well-developed skill
• full standard vocabulary (1000-10000 words)
• American English 600,000 words.
• specialized vocabularies (research, medical, )
• editing vocabularies (back, delete, )

6
Handwriting Speech

• Common issues
• vocabulary size
• individual variability
• speaker dependent, adaptive, independent
• signal segmentation
• isolated words, continuous
• Lets look at handwriting as an example,but
  almost all concepts apply to speech too
• not to mention other inputs, such as eye
  movements!

7
Off-line vs. on-line recognition

• Off-line recognition
• examine static output of handwriting,i.e., the
  end result of the writing
• On-line recognition
• examine dynamic movement of handwriting,i.e.,
  the strokes, pen up/downs involved
• Which is more informed? more useful?

8
Recognition techniques

• Neural networks
• neurally-inspired computational models
• input bitmap, or vectorized strokes
• output probably characters
• best for off-line recognition
• Hidden Markov models (HMMs)
• powerful probabilistic models
• input vectorized strokes
• output full recognition of chars, words, etc.
• best for on-line recognition
• Consider HMM-based on-line recognition

9
On-line feature extraction

• On-line strokes ? feature vectors
• basic features pen up/down, direction, velocity
• useful features curvature, reversal, ...

10
On-line recognition

• Hidden Markov models (HMMs)
• probabilistic models for dynamic behavior
• Set of N states with
• a(i,j) probability of state transition i?j
• b(o,i) probability of seeing o in state i
• can be discrete or continuous prob. distributions

11
Hidden Markov models

• Lets say we have
• M HMM representing predicted behavior
• O observation vector sequence O
• Three problems
• evaluation find Pr(OM)
• decoding find the state sequence Q
  thatmaximizes Pr(OM,Q)
• training adjust parameters of M toincrease
  Pr(OM)

12
Hidden Markov models

• HMM evaluation
• find Pr(OM)
• evaluate O lt x x y x gt
• can we do this efficiently?

.4
1
.8
.6
.2
13
Hidden Markov models

• HMM decoding (Viterbi algorithm)
• find best state sequence through HMM,maximizing
  the probability of the sequence
• Given the word Hello
• Maximize the possible paths through the model by
  calculating their probability of being actual
  words.
• Its less probable that I meant Hallo or
  Hejjo
• It is more proabable that I meant
• Case 1, word matchingJello
• Case 2, letter matching (vector strokes) HeIIo

14
Hidden Markov models

• HMM training (Baum-Welch / EM algorithm)
• re-adjust a(i,j), b(o,i) to increase Pr(OM)
• iterative procedure
• allows for fine-tuning of HMM parameters for
  particular observation sets
• Every time I write, HeIIo I really mean
  Hello.
• (Increase in the Vector Stroke probability)
• susceptible to just my behaviour. The machine
  learns the probability of my Hello, not your
  Hello.

15
Hidden Markov models

• Composing HMMs
• we can add sub- HMMs into larger HMMs,creating
  a model hierarchy at different levels
• For instance, we can create three levels
• strokes
• letters
• words

16
On-line recognition

• Stroke HMMs with states
• up-down loop
• s1 up, curvature, hi velocity
• s2 down, curvature, hi velocity
• up-down cusp
• s1 up, curvature, , hi velocity
• s2 0 velocity
• s3 down, curvature, hi velocity
• up-down ramphoid
• s1 up, curvature, hi velocity
• s2 0 velocity
• s3 down, curvature, hi velocity

17
On-line recognition

• Letter HMMs based on stroke HMMs

18
On-line recognition

• Word HMMs based on letter HMMs
• basic idea is straightforward
• but its deceptively tricky why??

19
On-line recognition

• Putting it all together
• compacting states
• taking word frequencies into account
• where do frequencies come from? -)

20
Handwriting problems

• Speed-accuracy tradeoff
• as people speed up, their handwriting
  degrades(uh, no duh!)
• Printed vs. handwritten?
• often some combination of the two!!
• Dotting is, crossing ts, for on-line
  recogn(minding your ps and qs?)
• Mixing language with graphics gestures

21
Speech recognition

• Same basic ideas for recognition
• convert to recognizable signal (transforms)
• recognize using hybrid methods and a hierarchy of
  phonemes, words, etc.
• Many similar / analogous problems
• individual variability (esp. female/male voices)
• mixing real input with command input
• speed-accuracy tradeoff (?)

22
Eye-movement recognition (?!)

• Yet again, same idea translate noisy signal to
  what people actually intended
• Example Eye-typing system

23
Discussion

• Have you used handwriting/speech systems?What
  are the benefits of these systems?
• What is handwriting/speech good for?When is it
  easier to use standard input?