PAKISTAN SIGN LANGUAGE RECOGNITION

1
PAKISTAN SIGN LANGUAGE RECOGNITION
Boltay Haath -
http//www.boltayhaath.cjb.net
Members M. Yousuf Bin Azhar Suleman Mumtaz
Mehmood Usman Ali Muzzaffar
Mentor Aleem Khalid Alvi Assistant Professor
2
Agenda

• WHAT IS BOLTAY HAATH
• Introduction , Motivation, Objectives.
• BENEFITS TO SOCIETY.
• Usefulness in daily life
• WHAT IS PAKISTAN SIGN LANGUAGE (PSL)
• SYSTEM OVERVIEW
• INNOVATIONS AND NOVEL IDEAS.
• SYSTEM IMPLEMENTATION.
• STATISTICAL TEMPLATE MATCHING
• ARTIFICIAL NEURAL NETWORK (ANN)
• Motion DETECTION
• PERFORMANCE MEASURES
• PERFORMANCE AND RESULTS
• SYSTEM DESIGN METHODOLOGY.
• DESIGN MODEL
• TEAM WORK
• BUSINESS PLANS AND MARKETTING STRATEGIES.
• Conclusion
• Future Improvements

3
What is Boltay Haath?

• A computerized sign language recognition system
  for the vocally disabled.

4
Motivation
Communication Gap
Vocally Disabled
Ordinary Person
The reasonable man adapts himself to the world
the unreasonable one persists in trying to adapt
the world to himself. Therefore all progress
depends on the unreasonable man. - George Bernard
Shaw
5
How System Works?
6
Objective
7
Benefits to Society
8
Pakistan Sign Language(PSL)

• A visual gestural language having its own
  vocabulary and syntax used in Pakistan.
• Influenced by spoken languages.
• No proper grammatical structure.

9
PSL Signs
10
INNOVATIONS AND NOVEL IDEAS

• The first system to recognize PSL.
• Accelerometer for continuous real time gesture
  recognition.
• Adaptable other sign languages.

11
SYSTEM IMPLEMENTATION

• STATISTICAL TEMPLATE MATCHING
• ARTIFICIAL NEURAL NETWORK (ANN)

12
STATISTICAL TEMPLATE MATCHING

• How STM works
• Template Specification (Training)
• Interval Matching (Recognition)
• Removal of Ambiguity (LMS)

13
Statistical Training
Start Training
Get Training Samples
Calculate µ for each second
Calculate s for each second
Update Gesture Database
Training complete
14
Statistical Recognition Flow
Sensor Values
DB
SGR
If Any Result
No
Return empty string
Yes
Multiple Results ?
No
Return Symbol
Yes
Use LMS

• LMS
• Get the Symbols from DB,
• Distances.
• -Select Least Distance symbol in
• memory from dataset.
• -Return Result (Symbol)

DB
15
Removal of AmbiguityLEAST MEAN SQUARE (LMS)

• To cater to problem of multiple outputs.
• LMS for each candidate gesture is calculated
• And the gesture with least LMS value is selected
  as the final output.

16
Artificial Neural Network

• Single Neuron Operation

17
ANN Library

• For Static Network.
• Multilayer perceptron network with back
  propagation learning.
• Object Oriented design.

18
Neural Network Classification with Committee
system

• Architecture of Expert 581
• Activation function for hidden nodes Sigmoid
  Logistic
• Activation function for output nodes Hyperbolic
• Input was scaled from the range of 0 to 255 to-
  1.28 to 1.27.

19
Motion Detection (Hand in motion)
Window size

8
151
152
156
154
153
159
161
165
168
150
172
178
182
185
186
190
2
4
3
2
3
4
6
Threshold

1
.
8
A

2

3

2

4

3

4

6


24


3
.
43
7 7
A
gt
Threshold therefore hand is in motion
20
Motion Detection (Hand is stationary)
Window size

8
151
152
155
154
153
155
156
157
158
150
158
159
160
161
161
161
1
1
1
1
0
0
1
Threshold

1
.
8
A

1

0

1

1

1

0

1


5


0
.
714
7 7
A
lt
Threshold therefore hand is stationary
21
PERFORMANCE MEASURES

• Recognition time A gesture should take
  approximately 0.25 to 0.5 second in the
  recognition process in order to respond in real
  time.
• Synchronized speech synthesis. The speech output
  corresponding to a gesture should not lag behind
  the gesture output by more than 0.25 seconds.
• Continuous and automatic recognition To be more
  natural the system must be capable of recognizing
  the gestures continuously without any manual
  indication or help for demarcating the
  consecutive gestures.
• Recognition Accuracy The system must recognize
  the gestures accurately between 80 to 90 percent.

22
TESTING RESULTS
23

24

25
SYSTEM DESIGN METHODOLOGY

• If any thing can go wrong, it will go wrong.
• Murphys Law

26
Team work
Research
ANN
STM
Yousuf Suleman
Mehmood Ali
All Team Members
Integration
Testing
Training
27
BUSINESS PLANS AND MARKETTING STRATEGIES
Vision Capture Pakistani market and then move
on to regional markets.

• Market
• Customers are limited
• Low buying power of targeted market segment

Competition Most projects are in research
phase.No Competition Very little changes in the
industry
Marketing Word of mouth deaf associations
Internet special media Trade shows deaf
seminars
28
Conclusion

• System scalability
• Domain (Different fields/languages)
• Vocabulary
• Future
• More sensors
• Polhemus for location
• Abduction sensors for fingers
• Two gloves
• Can be modified for use on hand held devices
  using Microsoft Compact .Net Framework.