Zurich University of Applied Sciences Winterthur

1
Zurich University of Applied Sciences Winterthur

• Project GENUIN
• GENeric multimodal User INterface
• Prof. Dr. H.-P. Hutter
• Roger Toenz
• Michel Müller
• Beat Tscharner

2
Outline

• Objectives
• Environments
• Design Guidelines
• Sample Application E-Mail-Client
• Generation of MMUIs

3
Objectives

• Develop a concept for a generic multimodal user
  interface for mobile applications
• Supports different environments
• At least two input and two output modalities
• The different modalities cooperate and support
  each other.
• The layout of the different user interface
  modalities adapt to the different interaction
  environments and to the user needs.
• There is a general description of the user
  interface logic, from which the actual user
  interface is derived for a specific environment.

4
Environments Office

• Controlled environment (quiet, stable)
• User pays his full attention to the application
• Broadband, high-quality interaction facilities
  (normal keyboard, large high-resolution displays)
• High-quality multimedia facilities

5
Environments Passive Mobile

• The environment can only partially be controlled
  (noise, movement).
• The user attention is mainly focused on
  application
• User sits or stands, possibly in a public
  transport facility
• Limited varying bandwidth access
• Small low-to-medium-resolution displays with no
  or small keyboard, pen, mircophone, camera.

6
Environments Active Mobile

• Same situation as in the passive mobile
  environment
• But user can only partially devote attention to
  the user interface, i.e., one or both hands are
  busy and/or eyes are busy and/or ears are busy.
• Typical situation when user is driving a car or
  walks around.

7
Major Design Guidelines

• 1. Minimize overall task completion time
• ? Is dependent on user and environment

8
Consequences of Guideline 1

• The more frequent a task is the lower its
  completion time should be
• one click / one short command
• accessible in all views
• Less frequent tasks allow more indirect access
• via menus
• only accessible in some of the views

9
Major Design Guidelines

• 2. The system must be transparent to the user
• at any time, in all environments, about
• what is the current state of the dialogue?
• what ist the system currently doing?
• what tasks are currently allowed?
• what is the state of the different modalities?
• which commands are currently available?
• where he can find help

10
Major Design Guidelines

• The different modalities should support each
  other as much as possible
• What you see is what you can say
• What you hear is how you have to say it
• Your hear what you can see
• Speech input You see and hear what is
  recognized and what is being done

11
Major Design Guidelines

• Modalities should be under full user control
• User can turn off each modality separately
• User can interrupt speech output by barging-in
  or pen input

12
Sample Application E-Mail-Client

• Standard E-Mail-Client-Functionality
• Plus Voice Mails
• Modalities
• Pen/wheel/button- Input
• Graphical output (130x170 Pixel)
• Speech Input
• Synth. Speech Output

13
E-Mail-Client MMUI-Design
speech output indicator
Taskbar
speech input indicator (on, off, recognizing,
garbage)
account disconnected
menubar pop-up
active window
direct help, speech help
context menu
preview window
14
Demo...

• Demonstrator running on a PC
• Developed with JAVA and JNI
• Speech Recognition HTK/HAPI
• Speech Synthesis SVOX

15
Next Steps

• Collect user experience with prototype
• Optimize MMUI of Prototype
• Implement on PDA
• Concept for Generation of MMUI

16
Implementation on PDA

• What is needed?
• robust speech recognition on PDA
• noise
• OOV-detection
• vocabulary independent
• speech synthesis
• running on PDA
• mixed language
• output speech and phonemic transcription

17
Generation of MMUIs

• Starting Points
• Domain Model
• concepts
• attributes
• associations
• Task Model
• Goals What does the user want to do?
• Use Cases, scenarios
• How does he want to do it? How often?
• Which alternatives does he need?

18
Concept of MMUI Generation
19
E-Mail-Client Domain Model
20
Example E-Mail-Client Task Model

• E-Mail-ConceptsAndTasks.xls

21
Questions

• ?