Hand Gesture Recognition for Multimedia Applications

1
Hand Gesture Recognition for Multimedia
Applications

• Moaath Al-Rajab

2
Motivation

• Recent development in interactive systems
• Nintendo Wii 2005 console, Microsoft surface.
• Games
• Interactive games are welcomed by parents and
  users.
• Elderly applications
• Helps to stay fit.

3
Research aim

• Create a framework for user-interface
• Develop a simple game controlled by a set of
  designed gestures.
• Design and collect interface commands
• Through selecting the most distinctive and
  appropriate gestures.
• Choose and develop the right techniques
• Segmentation, tracking, feature extraction,
  recognition and detection.
• Evaluate the chosen techniques
• Through experiments.

4
Research focus
5
Datasets
6
Datasets

• Capturing data in uniform background and lighting
  (UBL)
• Hand only in the scene (OEH)
• Whole body in the scene (OEW)
• 3D models and virtual performance of hand
  gestures

7
The application
It is a simple game
Play game /Update/ ..
User
Rendering
User Interface Display
Hand Movement
Image Capture
Image Input
Standard Web Camera
8
The application
Applications block diagram
9
Segmentation, tracking and feature extraction

• Segmentation and tracking
• Segmentation is a process of grouping regions and
  features of the image that have similar
  characteristics together, with the aim of
  detecting semantically important objects.
• - For UBL and virtually created gestures
  datasets
• Optimal thresholding for segmentation
• CAMShift for tracking
• For OEH and OEW datasets
• Skin colour was modelled by collecting skin
  colour samples
• Skin colours typically occupy a relatively
  compact area of HSV-space
• CAMShift tracker

10
Feature extraction
11
Posture and gesture recognition
Key-frames recognition
8 postures 150 instances per posture per dataset
12
Posture and gesture recognition
Key-frames recognition
13
Posture and gesture recognition
Gesture recognition
Dataset Settings
Using 8 Gestures, How do you evaluation ZVM?
14
Posture and gesture recognition
Gesture recognition
What impact the similarity between gestures has
on results? - 80 instances for each of the 5
gestures.
15
Posture and gesture recognition
Gesture recognition
Evaluating 5 distinctive gestures ZVM is tested
on these 5 gestures using 80 instances for
each In user-dependent experiment only 12
instances of OEH are used
Recognition rate
Experiment
84.5
ZVMCvR
88
ZVMCvR user-dependent
94.5
ZMHMM
98.3
ZMHMM with CoM
99.1
ZMHMM CoM user-dependent
16
Gesture detection
Techniques explored

• Sliding window
• A buffer is created which stores a certain number
  of frames
• Each scan is evaluated against HMMs
• The highest likelihood is selected

• Single HMM with Viterbi
• Trained HMMs are used to create one HMM
• The Viterbi algorithm is deployed to find the
  best path cycling through the HMM over multiple
  gestures.

17
Gesture detection
Techniques explored!
Sliding window?
http//www.comp.leeds.ac.uk/moaath/gHand/results.h
tm
18
Gesture detection
Techniques explored!
Single HMM with Viterbi
19
Gesture detection
Techniques explored!
Single HMM with Viterbi
Accuracy rate 76
20
Viewpoint invariance
Capturing models
Six 3D hand models were captured using the
Polhemus FastSCAN

• To smooth multi-layers to one layer, the RBF
  (Radial Basis Functions) interpolation function
  is used.
• The number of facets in each of the models is
  5000.

21
Viewpoint invariance
Capturing models

• 30 different viewpoints
• 8 GMM HMM structures
• Left-to-right topology
• 10 folds cross-validation
• The length of each gesture is set to be 90 frames

22
Viewpoint invariance
Table 6-2 The confusion matrix using 8 virtually
created hand gestures. ZM descriptor is used with
8 HMMs. The number of training and testing
instances is 240 instances, each gesture of 30
with 10 fold cross-validation. The mean accuracy
rate is 43.75 where the base recognition rate is
100/8 12.5.
23
Findings

• Four new datasets have been designed.
• A prototype has been implemented.
• An Evaluation on the newly developed datasets on
  CAMShift.
• An Evaluation of ZVM on the hand gesture
  datasets.
• The ZVM is compared with a standard method.
• Two techniques for hand gesture detection in
  addition to the use of garbage states.
• Virtual hand gesture performances have been
  proposed for dealing with viewpoint variation,
  tested and evaluated.

24
Thank you
25
Datasets (UBL)

• Capturing data in uniform background and lighting
  (UBL)
• Hand only in the scene (OEH)
• Whole body in the scene (OEW)
• 3D models and virtual performance of hand
  gestures

Go Back ltltlt
26
Datasets (OEH)

• Capturing data in uniform background and lighting
  (UBL)
• Hand only in the scene (OEH)
• Whole body in the scene (OEW)
• 3D models and virtual performance of hand
  gestures

Go Back ltltlt
27
Datasets (OEW)

• Capturing data in uniform background and lighting
  (UBL)
• Hand only in the scene (OEH)
• Whole body in the scene (OEW)
• 3D models and virtual performance of hand
  gestures

Go Back ltltlt
28
Datasets (3D)

• Capturing data in uniform background and lighting
  (UBL)
• Hand only in the scene (OEH)
• Whole body in the scene (OEW)
• 3D models and virtual performance of hand
  gestures

Go Back ltltlt
29
Posture and gesture recognition (dataset)
Gesture recognition
Dataset Settings
Using 8 Gestures, How do you evaluation ZVM?
Go Back ltltlt
30
Posture and gesture recognition (settings)
Gesture recognition
Dataset Settings
Using 8 Gestures, How do you evaluation ZVM?
Go Back ltltlt