MakMessenger and FingerChat, Communications Technologies to Assist in the Teaching of Signed Languag

1
Mak-Messenger and Finger-Chat, Communications
Technologies to Assist in the Teaching of Signed
Languages to the Deaf and Hearing

• James Ohene-Djan
• Andrew Mould
• Robert Zimmer
• James Bassett-Cross
• Ben Cosh

Assistive technologies group at
Goldsmiths College, University of London New Cr
oss, London SE14 6NW Tel [phone], Fax
44 2079197853 Email j.djan_at_gold.ac.uk www.ass
ist-tech.gold.ac.uk
2
Mak-Messenger and Finger-Chat Presentation
Overview

• Introduction
• Aims of Research
• Background
• Mak-Messenger System Session
• Finger-Chat System Session
• Conclusions and Future Work

3
Mak-Messenger and Finger-Chat Introduction

• Communications-based learning technology is an
  area of educational research that seeks to use
  advanced communication systems to enhance
  traditional learning environments with tools that
  allow students and teachers to formulate and
  exchange ideas.
• Technologies such as instant messaging, and chat
  systems are being used for teaching.
  
• We present two novel communications-based
  learning systems,
  
• Mak-Messenger and Finger-Chat, specifically
  designed to assist in the teaching of
  Fingerspelling and signed languages to deaf and
  hearing students.
• By providing these examples of learning
  technologies specifically tailored to the deaf
  community, we hope to illustrate how advanced
  learning technologies may be used to address
  issues relating to the educational needs of the
  deaf.

4
Introduction

• An instant messaging or chat system designed
  specifically to teach fingerspelling to children
  and young adults
  
• A real-time learning technology enabling users to
  communicate using multiple versions of pictorial
  representations of fingerspelling and written
  English
• Finger-Chat illustrates the capabilities of the
  Internet as a platform for delivering sign
  language educational content
  
• We aim to stimulate greater interest into digital
  representations of non-speech notations and
  specialised computer systems for hearing-impaired
  people

5
Background - Fingerspelling

• The use of hand positions to represent the
  letters of the alphabet. It is the oldest
  recognised approach to formalised communication
  for deaf people
• Fingerspelling languages are widely used as
  methods of teaching communication to
  hearing-impaired children and others
  
• The positions of the hand are used to resemble
  the printed letters of the alphabet, thereby
  enabling a direct translation between the two
  languages.
• In educational contexts, fingerspelling
  vocabularies are translated into pictorial
  representations

6
Background - Fingerspelling

• Pictorial representations
• The British Deaf Associations (BDAs)
  fingerspelling fonts
  
• My name is James
• Deafnet Association Incs American Sign Language
  fonts
  
• My name is James
• Valerie Suttons sign writing system
• My name is James

7
Background Messaging Systems

• Real-time instant messaging or chat systems are
  used for synchronous live communications
  
• Users send and receive messages in real-time via
  computing networks
  
• Representative examples of instant messaging
  systems are ICQ instant messenger, MSN Messenger
  and AOL Messenger
  
• Within educational contexts, these systems enable
  students to formulate and exchange ideas in a
  non-intrusive manner
  
• Little research has been conducted into tailoring
  systems to needs of hearing-impaired people

8
Background Finger-Chat

• Finger-Chat is the first instant messaging system
  specifically designed to support the teaching of
  fingerspelling alphabets
  
• Through a dynamically generated user interface,
  the system offers the user multiple modes with
  which to communicate
  
• Users chat on-line to each other using
  combinations of pictorial representations of
  fingerspelling as well as Standard English
  
• Designed so that both hearing and
  hearing-impaired people can interact with each
  other on an equal basis

9
Motivations

• The learning of fingerspelling could be enhanced
  through a communications technology that allowed
  interaction in the classroom with each other in a
  non-intrusive manner
• Communication could take place between users who
  were not in physical proximity
  
• Allow for communication in the classroom on an
  inclusive basis, whereby those who are
  hearing-impaired could use the same software as
  those who are not
• Accommodate forms of fingerspelling from
  different countries and grouping by enabling the
  entire user interface to tailored to the chosen
  vocabulary

10
Systems Architecture

• The Finger-Chat system has been implemented using
  a client-server architecture
  
• Within this architecture client applications
  communicate with each other via a server
  application
  
• The server application is responsible for
  broadcasting messages from a client in real-time
  to multiple client applications
  
• The server therefore has a one-to-many
  relationship with clients and operates as a
  trusted intermediary
  
• Programmed using the Java programming language
• Runs on Windows And Unix

11
Systems Architecture
Client-server architecture over TCP/IP
To facilitate multiple connections, the server
code is implemented using Javas threading
technology This enables parallel program executi
on
12
Systems Architecture

• Multiple versions of pictorial representations of
  fingerspelling vocabularies from different
  sources implemented as font sets
  
• Each key on the keyboard is mapped via the font
  to its pictorial equivalent, allowing the user to
  use a standard keyboard
  
• When a font set is chosen, the user interface is
  dynamically regenerated to reflect the chosen
  fingerspelling vocabulary
  
• Currently distributed with both the BDA, ASL and
  and Valerie Suttons fingerspelling fonts and
  additional fonts can be added
  
• A chosen fingerspelling font can be
  simultaneously with the Standard English alphabet

13
Class Diagram
14
System Use-Case Diagram
15
Finger-Chat Sessions

• Before a session can commence, it is necessary to
  start the Finger-Chat server
  
• Networked with a unique IP address, the server is
  used by client applications to send messages
  
• Users start Finger-Chat client application, this
  displays the current connection settings and
  gives the user access to program, its settings
  and options
• Users can customise the application to meet their
  specific needs by specifying, via an options
  menu, the fingerspelling fonts to be used and the
  server to connect to

16
Finger-Chat sessions Client Interface
17
Finger-Chat sessions Tailoring Interface

• The main tool bar allows users to connect to the
  server and specify a user log-in name
  
• Users can select whether messages received are
  viewed using fingerspelling, English or a
  combination of both
  
• To engage in an on-line conversation, the user
  must choose to connect to the server by pressing
  the connect button
  
• The client application then tries to initiate
  contact with the server and provides
  corresponding feedback to the user when contact
  is made

18
Finger-Chat sessions Online Conversation using
BDA and English
19
Finger-Chat sessions On-screen Interface Keyboard

• Users are able to type a message using either
  their standard keyboard or the on-screen
  interface keyboard
  
• Enables on-line conversations where the user has
  chosen to display all messages as fingerspelling
  
  
• This feature enables users to make a correlation
  between pictorial representations shown and the
  Standard English language
  
• This design feature has been included to enhance
  possibilities for learning the English alphabet
  in tandem with a particular fingerspelling
  vocabulary

20
Finger-Chat sessions Online Conversation using
BDA only
21
Finger-Chat sessions Multimode Conversations

• During a session, users may request for the
  client application to use different
  fingerspelling vocabularies
  
• This feature ensures that the Finger-Chat
  application is extendable to accommodate
  variations of fingerspelling fonts
  
• This means that one user can be using Valerie
  Suttons sign writing font whilst another is
  using Deafnet Association Incs American Sign
  Language fonts
• This provides scope for interactions between
  users of varying skill-levels, abilities and
  dialects.
  

22
Finger-Chat sessions Screen 1 Sutton
Screen 2 BDA
23
Technical Achievements

• Machine independent software (There is no
  prerequisite that particular fonts are installed
  on the users system).
  
• Ability to be customised to meet different user
  needs (user name, server address, port number,
  character sets, etc.)
  
• Ability to meet internationalisation
  requirements Different fingerspelling symbol
  sets can be used to cater for localised
  variations.
• Ability to adjust the typeface without requiring
  a total program restart (Valerie Suttons
  Signwriter program cannot do this).
  
• Easy to use integrated help system.

24
Conclusions

• A useful educational tool should help to reduce
  the amount of time required to learn
  fingerspelling
  
• Should encourage more people to learn
  fingerspelling, and improve the quality of
  communications between hearing and
  hearing-impaired people
• The Finger-Chat system will encourage
  communications between hearing-impaired users in
  different parts of the world
  
• The application will make a difficult educational
  task more interesting to students and encourage
  active participation
  
• Allow multiple users to communicate with each
  other simultaneously. It would be physically
  difficult to look at and interpret the gestures
  being made by several people at the same time

25
Future Work

• A set of Empirical tests currently being devised
  for classroom investigations
  
• Version 2 of Finger-Chat currently under
  development
  
• Work under progress on an email finger-speller
• www.assist-tech.gold.ac.uk

26
Mak-MessengerBackground Sign Languages

• Sign languages are combinations of hand and body
  movements together with facial expressions used
  to communicate without sound
  
• Such languages enable hearing-impaired people to
  communicate with other people, both
  hearing-impaired and hearing
  
• These languages, which consist of intricate
  vocabularies and syntaxes, are the native
  languages for a large number of profoundly
  hearing-impaired people
• In an educational context both spoken and printed
  information can be translated into non-speech
  notations to enable hearing-impaired people to
  have equal access to it

27
Mak-MessengerBackground Non-speech notations

• Non-speech notations aim to capture the emotions
  and visual subtleties of signed speech and relay
  them in a written form
  
• Many proposals made for non-speech notations
• Stokoe Notation Maher J Sacks O, 1996
• Hamburg Notation System (HamNoSys) Prillwitz S
  et al., 1989
  
• Make no provision for the representation of the
  facial expressions and body movements which often
  accompany signed words
  
• SignWriting (Valerie Sutton) aims to represent
  how words and phrases are signed
  
• It differs from other proposals as it aims to
  capture the visual subtleties of body movements
  
• Symbols used in SignWriting to depict signed
  words are difficult to comprehend but clearly
  represent the hand and facial expressions used
  during physical signing.

28
Mak-Messenger Aims of Research

• Mak-Messenger designed to assist in the teaching
  of the Makaton Vocabulary to children through
  means of interactive communication.
  
• Mak-Messenger is an Instant Messaging service
  that allows users to form a connection with a
  remote server and send real-time messages Makaton
  to all connected users.
• To design a system that enables students to
  communicate from remote/different locations by
  means of a non-speech notation
  
• Show that through system-tailoring it may be
  possible to provide all students with equal
  access to the educational technologies and
  resources
• Make the learning of Makaton fun !!

29
Mak-MessengerBackground - Makaton

• The Makaton sign language writing system, derived
  from BSL, was proposed and developed by Margaret
  Walker, a speech therapist, in the 1970s.
  
• Originally designed as a communication system for
  a group of hearing-impaired adults with severe
  learning difficulties
  
• It has since been expanded for a broader use for
  both children and adults with varying
  difficulties in conventional communication, in
  particular, those whose hearing is impaired.
• The Makaton vocabulary has now gained widespread
  acceptance for use in schools, hospitals and in
  the home.
  
• Most widely used sign writing system within the
  United Kingdom
  

30
Mak-MessengerBackground Makaton Example
31
Mak-MessengerBackground - Makaton

• A set of symbols have also been developed to
  accompany the signs, providing a visual
  representation of the language
  
• These extensions make expressive communication
  easier for people with physical disabilities that
  make the formation of signs difficult
  
• Differences between the Makaton and BSL
• It was devised to be an aid to communication
  rather than a complete language
  
• When developed it was assumed that those who
  would use it may have the ability to use speech
  alongside signing
  
• As with spoken languages, there are regional
  variations of dialect and accent. Similarly there
  are variations in the Makaton signs in UK

32
Mak-MessengerUse Case Diagram
33
Mak-MessengerArchitectural Framework
Figure shows the framework, upon which
Mak-Messenger has been designed and implemented.
The intricate network establishment and
communications between the server and its clients
are handled by the Java networking package
(java.net)
34
Mak-MessengerA Mak-Messenger Session

• Once started the user is presented with two
  options Connect and Exit
  
• In order to participate in a conversation the
  user must select the connect option
  
• This takes the user through the login process
  where they can enter their username and a
  connection is established with the server
  
• Once a successful connection has been established
  the user is presented with the chat terminal from
  where they participate in conversations

35
Mak-MessengerA Mak-Messenger Terminal
A Message Display
B Symbol Gallery
C Composition Area
Terminal from where the user creates and views
messages
36
Mak-MessengerA Mak-Messenger Terminal
Messages are typed by selecting signs from the
button panel. When a message is typed it is disp
layed in the preview pane
37
Mak-MessengerMessage composition

• Sending image files to the server and then back
  to connected clients would be very expensive on
  bandwidth.
  
• To counter this problem messages consist of a
  concatenated string of symbol identifiers 3
  character codes relating to a unique symbol.
  
• Example of a message
• 045176087143

38
Mak-MessengerMessage composition

• On receipt of this string each client processes
  the information contained in the message body and
  displays the correct sequence of Makaton symbols
  
• 045 176 087 143

39
Mak-MessengerA Mak-Messenger Session
Terminal 2 - Tina
Terminal 1 - Tommy
40
Mak-MessengerConclusions and Future Work

• Empirical Studies and class room testing
• Incorporate other sign writing systems into the
  application and develop the translation between
  them.  
  
• Develop Mak-Messenger public Internet chat room
  which would vastly increase its accessibility to
  potential users of the system.
  
•  Version 2 of system dependent on funding
• Version 1 of system available at
• www.assist-tech.gold.ac.uk