A Method for Temporal Hand Gesture Recognition

1
A Method for Temporal Hand Gesture Recognition

• Joshua R. New
• Knowledge Systems Laboratory
• Jacksonville State University

2
Outline

• Terminology
• Motivation
• Current Research and Applications
• System Overview
• Implementation Approach
• Demonstration

3
Terminology
Image Processing - Computer manipulation of
images. Some of the many algorithms used in image
processing include convolution (on which many
others are based), edge detection, and contrast
enhancement. Computer Vision - A branch of
artificial intelligence and image processing
concerned with computer processing of images from
the real world. Computer vision typically
requires a combination of low level image
processing to enhance the image quality (e.g.
remove noise, increase contrast) and higher level
pattern recognition and image understanding to
recognize features present in the image.
4
Motivation
5
Motivation

• Gesturing is a natural form of communication
• Gesture naturally while talking
• Babies gesture before they can talk
• Interaction problems with the mouse
• Have to locate cursor
• Hard for some to control (Parkinsons or people on
  a train)
• Limited forms of input from the mouse

6
Motivation (2)

• Interaction Problems with the Virtual Reality
  Glove
• Reliability
• Always connected
• Encumbrance
• Sanitation

7
System Overview
8
System Overview
User
Rendering
Update Object
User Interface Display
Hand Movement
Image Capture
Image Input
Standard Web Camera
9
System Overview (2)

• System
• OpenCV and IPL libraries (from Intel)
• Input
• 640x480 video image
• Hand calibration measure
• Output
• Rough estimate of centroid
• Refined estimate of centroid
• Number of fingers being held up
• Manipulation of 3D skull in QT interface in
  response to gesturing

10
System Overview (3)

• Hand Calibration Measure
• Max hand size in x and y orientations in of
  pixels

11
System Overview (4)
Saturation Channel Extraction (HSL space)
Original Image
Hue
Lightness
Saturation
12
Proposed Approach
13
Proposed Approach
14
Proposed Approach (2)
15
Proposed Approach (3)

• The finger-finding function sweeps out a circle
  around the rCoM, counting the number of white and
  black pixels as it progresses
• A finger is defined to be any 10 white pixels
  separated by 17 black pixels (salt/pepper
  tolerance)
• Total fingers is number of fingers minus 1 for
  the hand itself

16
Proposed Approach (4)

• Temporal Recognition System for hand movement
  patterns
• Input feature-vector creation based on
  calculations from 12 centroid locations (hand
  movement during 3 seconds average length of
  temporal gesture)
• Learning system training and recognition SFAM
  classification used for ability to interactively
  add new, user-defined gestures

17
Proposed Approach (5)

• Temporal Gesture Recognition System

18
Proposed Approach (6)

• Input Feature-Vector Creation
• Twelve centroids collected (x,y)
• Find top-left centroid so that gestures are not
  start-point dependent (a square is a square
  whether you start at the top left or the bottom
  right)
• Compute centroid differences to recognize
  movement, not position (a square whether the hand
  is at the precise 12 points or in between those
  points)
• Normalize using the perimeter to recognize
  percentage of total movement since users are
  inaccurate in repeating a gesture (a square
  whether large or small)
• Note System still sensitive to clockwise vs.
  counter-clockwise square (undo-like feature)

19
Proposed Approach (7)

• Learning System Training and Classification
• Feature vector is formatted for use by SFAM
• Movements of the hand were recorded and assembled
  into one file for training the SFAM system
• 25 examples each of circle, square, left/right,
  up/down
• System classification of hand gesture every 3
  seconds
• Train New Gesture button provided, stores gesture
  under the label entered in the box (5 is the
  default since 1-4 are already taken)

20
Proposed Approach
21
Demonstration
System Configuration
System GUI Layout
22
Demonstration (2)
Gesture to Interaction Mapping
Number of Fingers 2 Roll Left 3 Roll
Right 4 Zoom In 5 Zoom Out