Title: Facilitating User Interaction with Complex Systems via Hand Gesture Recognition
1Facilitating User Interaction with Complex
Systems viaHand Gesture Recognition
Joshua R. New, Erion Hasanbelliu, and Mario
Aguilar
- MCIS Department
- Knowledge Systems Laboratory
- Jacksonville State University
2Outline
- Motivation
- System Architecture
- Implementation Overview
- Proposed Approach
- Demonstration
- Future Directions
3Motivation
- Gesturing is a natural form of communication
- Interaction problems with the mouse
- Have to locate cursor
- Hard for some to control (Parkinsons or people on
a train) - Limited forms of input from the mouse
4Motivation (2)
- Interaction Problems with the Virtual Reality
Glove - Reliability
- Always connected
- Encumbrance
5System Architecture
User
Rendering
Update Object
User Interface Display
Hand Movement
Image Capture
Image Input
Standard Web Camera
6Implementation Overview
- System
- 1.6 Ghz AMD Athlon
- OpenCV and IPL libraries (from Intel)
- Input
- 640x480 video image
- Hand calibration measure
- Output
- Rough estimate of centroid
- Refined estimate of centroid
- Number of fingers being held up
- Manipulation of 3D skull in QT interface in
response to gesturing
7Implementation Overview (2)
- Hand Calibration Measure
- Max hand size in x and y orientations in of
pixels
8Implementation Overview (3)
Saturation Channel Extraction (HSL space)
Original Image
Hue
Lightness
Saturation
9Proposed Approach
10Proposed Approach (2)
11Proposed Approach (3)
- The finger-finding function sweeps out a circle
around the rCoM, counting the number of white and
black pixels as it progresses - A finger is defined to be any 10 white pixels
separated by 17 black pixels (salt/pepper
tolerance) - Total fingers is number of fingers minus 1 for
the hand itself
12Proposed Approach (4)
Process Steps Time (ms)
Process Steps Athlon XP 1900 (1.6 Ghz)
1) Extract Sat Channel 9
2) Threshold 3
3) Connected Contour Fill 14
4) Centroid 2
5) Segment Hand From Arm 9
6) Refined Centroid 4
7) Count Number of Fingers 0
Total Time 41
- System Runtime
- Current time 41 ms for one image from camera
- Processing Capability on 1.6 Ghz Athlon
- 24 fps
13Demonstration
System Configuration
System GUI Layout
14Demonstration (2)
Gesture to Interaction Mapping
Number of Fingers 2 Roll Left 3 Roll
Right 4 Zoom In 5 Zoom Out
15Demonstration (3)
16Demonstration (4)
17Future Directions
- Optimization
- Calibration Phase
- Defining Hand Orientation
- Learning System
- Interface Extensions
For additional information, please visit
http//ksl.jsu.edu.