Anthony McCloskey, T'M' Mc Ginnity and Liam Maguire

1
HIERARCHICAL LEARNING, PREDICTION AND TRANSLATION
OF SIGN LANGUAGE MESSAGE STRUCTURES

• Anthony McCloskey, T.M. Mc Ginnity and Liam
  Maguire
• University of Ulster
• Intelligent Systems Engineering Laboratory,
  Faculty of Engineering, Northland Road, Derry
  BT48 7JL, Northern Ireland, United Kingdom
• ta.mccloskey_at_ulster.ac.uk 

2
Motivation and Objective

• Motivation
• Electronic communication in the deaf community
  is largely based on spoken language systems,
  which are both slow and unnatural forms of
  communication for the deaf community. The 3rd
  Generation (3G) phones will enable real time
  video streaming, enabling sign language
  communication. This however does not remove the
  need for communication in a written form of sign
  language. As historically illustrated with the
  use of SMS, written communication is a
  significant part of the mobile network. It is
  therefore envisaged that sign messaging will be
  significant part of mobile sign communication.
• Objective
• The main objective of the system is to provide a
  fast, effective and accurate communication system
  but also to leave compositional control with the
  user. The system aims to provide an an
  electronic system that will enable the deaf and
  non-speaking to communicate in a electronic
  written form of sign language the deaf
  communities primary language. Achieving natural
  communication and a translation system so
  communication can be achieved between different
  languages.

3
System Overview

• The prediction system uses contextual information
  around the message and the current sign
  selections of the user to adaptively predict
  required next sign.
• Sign tags are passed between conversational
  participants.
• Personalised representation of a sign tag
  rreceived are looked up on Systems Database,
  while a common sign tag is passed throughout all
  systems.
• The common sign tag passed through out the system
  enables users to seamlessly communicate in
  different language.

• The foundation of the proposed system is a
  database, which contains user specific signs, and
  records of all users historical utterances and
  messages.
• The user can add or modify signs on the system to
  represent specific language, dialect and personal
  sign representation.
• The user is continuously presented with a system
  prediction of the possible next sign at each
  stage of message composition.

4
Sample of Sign Writing

• Six Hundred Individual sign movement are coded
  into representative symbols.
• The symbols represent all the various hand
  shapes, movements, locations and necessary
  details to represent a sign.
• The collective arrangement of these symbols can
  pictorially represent a signing person.
• The symbols code physical movements therefore
  are universal and not limited to a single sign
  language.
• The selection of different sequences of symbols
  define the sign language and the meaning.

5
Hierarchical Reduction of Prediction Space

• A participant specific approach is
• employed. This dictates that only
• messages relating to a particular
• participant are primarily
• considered.
• All utterance types
• that make up the messages
• are then systematically selected

• Then within the utterance type selected a
  specific utterances structure is
• identified. The sequence of
• sign categories that defines
  the utterance structure
• are then systematically
• selected according to users
  position in the utterance
• composition.

• Finally, within a sign
• category nine signs are
• selected for display to the user
• six of which are exploitative,
• and three are explorative.

6
Hierarchical Prediction Approach

• Architecture based on a hierarchical system.
• Adopts a top down prediction approach, from
  message structure down to individual next sign.
• Each tier has its own sub-goal, which forms an
  input to the lower tier.
• Information is fed back up the tiers confirming
  correct predictions or defining at which stage(s)
  of the process the prediction was incorrect.

• As a user selects a sign, new knowledge is gained
  through success or failure and this knowledge is
  immediately applied to effect subsequent
  predictions.
• The system adapts future predictions to
  incorporate new knowledge whether positive or
  negative.

7
Results and Conclusion
Computation Accuracy
Lexicon Utterance Construction Cost
Size and
Recognition Ohki 4 High 97.1 620
signs Not Attempted Vamplew
9 High 85-94 52 signs 13 Utter 4
signs Kadous 3 Low 85 95 signs
Not Attempted Holden 2 High 95 21
signs Not Attempted Starner 6 High 91.3
40 signs 500 Utter 4 signs Sujan
7 High 89 24 signs Not
Attempted McCloskey Low 100 1000 signs
Unlimited Utter Sign Utterances Unlimite
d This system can handle any combination of the
1000 signs entered on the system

• In the table above the purposed system has been
  compared to other systems which attempt to
  capture and communicate sign language. These
  systems use wearable devices and image processing
  techniques but fail to achieve comparable results
  to the propose system.
• This paper has highlighted the difficulties of
  communication within the deaf and no speech
  community and attempted to apply appropriate
  technology to provide a user based solution.
• The system exploits the historical message and
  utterance structures of the user together with
  the temporal sequences of symbols to provide
  information for the prediction system.
  Intelligent adaptation of the systems prediction
  system using reinforcement learning allows
  erroronus and successful predictions to adapt
  next stage predictions.