Facilitating User Interaction with Complex Systems via Hand Gesture Recognition

1
Facilitating User Interaction with Complex
Systems viaHand Gesture Recognition
Joshua R. New, Erion Hasanbelliu, and Mario
Aguilar

• MCIS Department
• Knowledge Systems Laboratory
• Jacksonville State University

2
Outline

• Motivation
• System Architecture
• Implementation Overview
• Proposed Approach
• Demonstration
• Future Directions

3
Motivation

• Gesturing is a natural form of communication
• Interaction problems with the mouse
• Have to locate cursor
• Hard for some to control (Parkinsons or people on
  a train)
• Limited forms of input from the mouse

4
Motivation (2)

• Interaction Problems with the Virtual Reality
  Glove
• Reliability
• Always connected
• Encumbrance

5
System Architecture
User
Rendering
Update Object
User Interface Display
Hand Movement
Image Capture
Image Input
Standard Web Camera
6
Implementation Overview

• System
• 1.6 Ghz AMD Athlon
• OpenCV and IPL libraries (from Intel)
• Input
• 640x480 video image
• Hand calibration measure
• Output
• Rough estimate of centroid
• Refined estimate of centroid
• Number of fingers being held up
• Manipulation of 3D skull in QT interface in
  response to gesturing

7
Implementation Overview (2)

• Hand Calibration Measure
• Max hand size in x and y orientations in of
  pixels

8
Implementation Overview (3)
Saturation Channel Extraction (HSL space)
Original Image
Hue
Lightness
Saturation
9
Proposed Approach
10
Proposed Approach (2)
11
Proposed Approach (3)

• The finger-finding function sweeps out a circle
  around the rCoM, counting the number of white and
  black pixels as it progresses
• A finger is defined to be any 10 white pixels
  separated by 17 black pixels (salt/pepper
  tolerance)
• Total fingers is number of fingers minus 1 for
  the hand itself

12
Proposed Approach (4)
Process Steps Time (ms)
Process Steps Athlon XP 1900 (1.6 Ghz)
1) Extract Sat Channel 9
2) Threshold 3
3) Connected Contour Fill 14
4) Centroid 2
5) Segment Hand From Arm 9
6) Refined Centroid 4
7) Count Number of Fingers 0
Total Time 41

• System Runtime
• Current time 41 ms for one image from camera
• Processing Capability on 1.6 Ghz Athlon
• 24 fps

13
Demonstration
System Configuration
System GUI Layout
14
Demonstration (2)
Gesture to Interaction Mapping
Number of Fingers 2 Roll Left 3 Roll
Right 4 Zoom In 5 Zoom Out
15
Demonstration (3)
16
Demonstration (4)
17
Future Directions

• Optimization
• Calibration Phase
• Defining Hand Orientation
• Learning System
• Interface Extensions

For additional information, please visit
http//ksl.jsu.edu.