A Speech Interface to Virtual Environment

1
A Speech Interface to Virtual Environment

• Authors
• Scott McGlashan and Tomas Axling
• Swedish Institute of Computer Science

2
Presentation Agenda

• Introduction
• The TALKING AGENT system
• DIVE
• SR/TTS
• Agent Modeling Framework
• Interaction Metaphor
• Reference Resolution
• Future Work
• Conclusion

3
Purposes of this paper

• Analyze the technical and design issues to
  combine a virtual world with a speech interface.
• Describe system architecture of the TALKING
  AGENT system.

4
Problems of Integration

• Speech Recognition Limited vocabulary to gain
  accuracy.
• Language Understanding Limited knowledge to
  maximize the understanding.
• Interaction Metaphor Who does the user talk to?
• (Above questions are discussed in detail in the
  authors last paper Speech Interface to Virtual
  Reality.)

5
Innovation of this System

• Combining intelligent agent and speech interface
  to carry out specialized functions in the VR
  World.
• Functions have been implemented
• Transporting objects
• Fetching objects
• Painting objects
• Increasing the size of objects

6
System Architecture
7
DIVE-Virtual Reality System

• DIVE(Distribute Interactive Virtual Environment)
  is a multi-user virtual environment.
• DIVE allow users and environment interact in
  real-time.
• DIVE contains a database composed of
  hierarchically organized objects .

8
DIME- DIVE Meeting Environment
9
Speech Recognition

• SR with limited pre-defined phrases promises good
  recognition performance.
• Using grammar to set constraint to search space.
• Using commercial SR-engine (Nuance).

10
(No Transcript)
11
Agent Modeling Framework

• High-level languages do not support complex
  symbolic computations.
• Oz is well suited for this purpose.
• Using ODI as interface between Oz and DIVE.
• The parent agent consists basic functions.
• We can define more specific agent by extend
  parent agent.

12
Agent Modeling Framework
13
Interaction Metaphor

• Direct manipulation -Personal Presence.
• Various metaphors for spoken interaction have
  been proposed.
• Proxy
• Divinity
• Telekinesis
• Interface Agent
• This system adopt the Proxy metaphor.

14
The DIVERSE System-Interface Agent
15
Addressing Agent

• Inside the users eye-sight
• Dialogue initiated by clicking on the agent.
• Outside the users eye-sight
• Phone agent-First press the phone agent then
  connect to remote agent

16
Feedback

• Given speech input ,system should give the visual
  feedback to the user.
• If the agent listening or not?
• What is the feedback when talking to agent far
  away?

17
Reference Resolution

• Given some descriptions , the reference
  resolution engine maps them to object which user
  is referring to.
• Considerations
• Object focus.
• Property Perception.
• Discourse Modeling.

18
Robust Interaction

• When errors dont matter
• User can view the results and current them by
  direct manipulation.
• Safety-critical applications
• Confirm user command.
• Clarifying incomplete or ambiguous commands.

19
Future Work

• Agent behavior should related to its previous
  action .
• Add mental components.
• Talking to agent by aura-driven .
• Evaluate this system with realistic scenario.
• Ex virtual travel agency.

20
Conclusions

• Add a speech interface to VR-system.
• Using constraint SR to achieve high accuracy.
• Developing an appropriate metaphor.
• The agents modeled in this system provide
  specific functions in the virtual world.

21
Q A
22
Paper Source

• McGlashan, S Speech Interfaces to Virtual Reality
  in Proceedings of the Second Conference on the
  Military Applications of Synthetic Environments
  and Virtual Reality, Stockholm, Sweden, 1995.