Byron Hood version 0.4

1
Sign Language Recognition using Webcams
2
Overview

• Average persons typing speed
• Composing 19 words per minute
• Transcribing 33 words per minute
• Sign speaker
• Full sign language 200 words per minute
• Spelling out estimate 50 words per minute
• Up to 3x faster

3
Purpose and Scope

• Native signers can input faster
• Benefits
• Hearing speaking disabled
• Sign interpreters
• Just letters numbers for now
• Additional complexity too much to handle
• Would require smaller distinctions

4
Research

• Related projects
• Using mechanical gloves, colored gloves
• Tracking body parts
• Neural network-based application
• Still images 92 accuracy
• Motion less than 50 accuracy
• Feature vector-based application
• Also about 90 accuracy on stills
• No motion tests

5
More Research

• Image techniques
• Edge detection (Roberts Cross)?
• Line detection (Hough transform)?
• Line interpretation methods
• Chaining groups of lines
• Macro-scale templates
• Residual math
• Memory management

6
Testing Model

• Human interaction necessary
• General testing model
• bhood_at_testing /syslab-tech \
• ./main images/hand.png
• DEBUG Edge detect time 29 ms
• Errors 0 Warnings 0

7
Program Architecture
SERVER PROCESS

• Webcam capture
• Edge detection
• Line detection
• Interpretation
• Attribute matching

IMAGE
FEATURE OUTLINE
LINE LIST
FINGER POSITONS
8
Edge Detection Results

• Results
• Outlines the important edges and not much besides
• Roberts Cross balances detection of major and
  minor lines

Original image (800 x 703)?
Final image (800 x 703)?
9
Cropping Results

• Remove useless rows columns with no features
• Better contrast
• Very large optimization
• Memory savings
• Area difference means order n2

Original (800 x 703)?
Result (633 x 645)?
10
Line detection

• Finished!
• Recently finished tweaking sensitivities
• Still a few potential memory issues

11
Line Grouping

• Part of line detection
• Large optimization
• Iterate over an order of magnitude fewer items
• Easier to handle, more pronounced trends

Examples of line groups, called chains
12
Line Interpretation

• Chaining groups of lines
• Templates
• Generation
• Template-based comparison
• Line residuals
• Use point coordinate averages
• Calculate average offset from average
• Easy to find height of finger

13
Sample Output

• After a typical run
• 10 days, 61219 until graduation!!
• bhood_at_testing /syslab-tech/src ./main hand.png
• Edge detection took 0.04 sec
• Image cropping took 0.00 sec
• Line detection took 0.17 sec (detected 1424
  lines)?
• Line chaining took 0.25 sec (detected 130
  chains)?
• Getting orientation took -0.08 src (1
  ORIENTATION_FORWARD)?
• Getting pinky pos. took 0.00 sec (2
  FINGER_BENT)?
• Getting ring pos. took 0.00 sec (2
  FINGER_BENT)?
• Getting middle pos. took 0.01 sec (2
  FINGER_BENT)?
• Getting index pos. took -0.00 sec (4
  FINGER_TUCKED)?
• Overall process took 0.47 sec
• TOTALCOUNT allocated 10718901, freed
  10364880 leaked 354021.

14
Timing

• Timing data from runs
• To nearest hundredth of a second
• Edge detection 0.04 sec
• Image cropping 0.00 sec
• Line detection 0.17 sec
• Line chaining 0.25 sec
• orientation 0.08 sec
• Pinky finger 0.00 sec
• Ring finger 0.00 sec
• Middle finger 0.01 sec
• Index finger 0.00 sec
• Overall process 0.47 sec
• A little slow considering goal of real-time

15
The Mysterious Future

• Perfect line interpretation
• Work on memory management
• Am leaking large quantities (50K) of memory
• Aggressive profiling needed
• Finish camera-computer interaction
• Device control must be precise, picky

16
The End!

• Code will be available to future years
• Contact me for a copy
• byron_at_phareware.com