ICMI02 Poster

1
The NESPOLE! Multimodal Interface for
Cross-lingual Communication Experience and
Lessons Learned
Loredana Taddei l.taddei_at_aethra.it
Erica Costantini costanti_at_psico.units.it
Alon Lavie alavie_at_cs.cmu.edu
NESPOLE! Project (http//nespole.itc.it)
NESPOLE! is a project which aims at providing a
system capable of supporting communication in the
field of e-commerce and e-service by resorting to
automatic speech-to-speech translation and
multimodal interaction. Multimodality in NESPOLE!
is accomplished by the integration of speech and
pen-based gestures (drawings performed on a
shared Whiteboard).
Showcase Scenario
The System

• The first NESPOLE! Showcase was developed for the
  following tourist scenario
• The customer aims at organizing a trip in
  Trentino.
• She starts by browsing APT web pages to get
  information.
• When the customer wants to know more about a
  particular topic or prefers to have a more direct
  contact, the speech-to-speech translation service
  allows her to interact in her own language with
  an APT agent.
• A videoconferencing session can be opened
  clicking a button and the dialog starts.

• NESPOLE! system allows users speaking different
  languages to interact in a natural way by means
  of
• hearing their original voice and their translated
  speech
• seeing the face and expression of their partner
• sharing visual information (maps and web pages),
  even through pen-based gestures
• monitoring the translation process and correcting
  it if and when something goes wrong.

First Version of The Interface

• NESPOLE! Monitor Window
• Remote speech translation the text of the last
  translated utterance of the remote speaker.
• Recognized hypothesis string the text of the
  last recognized utterance of the local user.
• String converted from IF textual
  representation resulting from the translation of
  the last utterance spoken by the local user back
  into their own language.
• Cancel Translation button generates a red
  flashing message on the monitor of the other
  party indicating that the incoming translated
  message should be ignored.

• AeWhiteboard Window
• image loading
• drawing functions (possibility to choose colors
  and shapes)
• rectangular or elliptical area selecting
• URL opening.
• The user can share all these operations with the
  remote user as they were sitting around a table
  exchanging brochures and illustrative material.

User Studies
Throughout the first two years of the NESPOLE!
project we took advantage of many opportunities
to collect data concerning the system and the
interface usability with actual users most of
the interface improvements leading to the current
version were based on the comments and
suggestions of these users and on our observation
of their behavior.
Monolingual Data Collection(February 2002)
User Group Meeting(September 2001)
Experimental on Multimodality (Summer 2001)
Involved 77 people (65 volunteers in the role of
the customers and 12 APT tourist office
agents). Objective system development a 16
items questionnaire was submitted to evaluate the
usability of the monolingual version of the
system. Result 91 of the volunteers reported
they would like to use the NESPOLE! system
instead of telephone or e-mail to ask for tourist
information and 100 of the tourist agents
reported they would like to use the system at
work.
Involved 8 European and US companies. A
questionnaire was distributed it consisted in 18
items concerning interface and devices usability,
multimedia and multimodality, overall system
effectiveness, possible fields of
application. Result many comments and useful
suggestions that have been implemented in the
successive user interface.
Involved 35 novice users speaking Italian, German
or English. Objective to compare the multi-modal
version of the NESPOLE! system with a speech
only version in which multi-modal resources
(pen-based gestures) were not available. Result
multi-modality helped users to overcome
ambiguities and system recognition and
translation errors.
Current Interface
NETMEETING WINDOW added live video
transmission.

WHITEBOARD WINDOW
Quick map saving the user can save the map with
a progressive number by simply clickinga button
or closing the map.
Image scrolling and zoom
DIALOGUE HISTORY WINDOWit contains all feedback
text strings ordered on a temporal base.