Distributed Multimodal Synchronization Protocol DMSP

1
Distributed Multimodal Synchronization Protocol
(DMSP)

• Chris Cross
• IETF 65
• March 20, 2006

With Contribution from Gerald McCobb and Les
Wilson
2
DMSP Background

• Result of 4 year IBM RD effort
• ID to IETF July 8, 2005 by IBM Motorola
• Reason for contribution
• A standard is needed for mobile device access to
  network based services for executing multimodal
  web applications
• Other protocols may have overlap but do not
  address all multimodal interaction requirements
• Other IETF IDs and RFCs
• Media Server Control Protocol (MSCP)
• LRDP The Lightweight Remote Display Protocol
  (Remote UI BoF)
• Media Resource Control Protocol Version 2
  (MRCPv2)
• Widex

3
DMSP Architecture

• There are 4 DMSP building blocks

• Modalities
• Model-View-Controller (MVC) design pattern
• View Independent Model
• Event-based modality synchronization

4
DMSP Architecture Building Blocks

• Modalities are Views in the MVC Pattern
• GUI, Speech, Pen
• Individual browsers for each modality
• Compound browsers for multiple modalities

Compound Browser
5
DMSP Architecture Building Blocks

• Model-View-Controller (MVC) design pattern

• Multimodal system can be modeled in terms of the
  MVC pattern
• Each modality can be decomposed and implemented
  in its own MVC pattern
• A modality can implement a view independent model
  and controller locally or use one in the network
  (e.g., an IM)

6
DMSP Architecture Building Blocks

• View Independent Model

• Enables a centralized model
• Modality interaction updates view and model
• Local event filters reflect important events to
  view independent model
• A modality listens to view independent model for
  only the events it cares about
• Compound clients, centralized control (IM) as
  well as distributed modalities all enabled with a
  single protocol

7
DMSP Architecture Building Blocks

• Event-based synchronization

• Compound Client All modalities rendered in
  client
• Interactions in one modality reflected in others
  thru event based changes to one or more model
• GUI DOM serves as View Independent model

8
DMSP Architecture Building Blocks

• Event-based synchronization (CONTD)

• Distributed Modality A modality is handled in
  the infrastructure
• Requires the DMSP for distributing modality
• Event based synchronization via View Independent
  Model gives a modality independent distribution
  mechanism
• Enables multiple topographies
• Compound Client w/ Distributed Modality
• IM

9
DMSP Design

• There are 4 abstract interfaces
• Command
• Response
• Event
• Signal
• Each interface defines a set of methods and
  related data structures exchanged between user
  agents
• Specified as a set of messages
• XML and Binary message encodings

10
DMSP Message Types

• Signals
• One-way asynchronous messages used to negotiate
  internal processing states
• Initialization (SIG_INIT)
• VXML Start (SIG_VXML_START)
• Close (SIG_CLOSE)

11
DMSP Message Types

• Command and control messages
• Add and remove event listener (CMD_ADD/REMOVE_EVT_
  LISTENER)
• Can dispatch (CMD_CAN_DISPATCH)
• Dispatch event (CMD_DISPATCH_EVT)
• Load URL (CMD_LOAD_URL)
• Load Source (CMD_LOAD_SRC)
• Get and Set Focus (CMD_GET/SET_FOCUS)
• Get and Set Fields (CMD_GET/SET_FIELDS)
• Cancel (CMD_CANCEL)
• Execute Form (CMD_EXEC_FORM)
• Get and Set Cookies (CMD_GET/SET_COOKIES)

12
DMSP Message Types

• Responses
• Response messages to commands
• OK (RESP_OK)
• Boolean (RESP_BOOL)
• String (RESP_STRING)
• Fields (RESP_FIELDS)
• Contains 1 or more Field data structures
• Error (RESP_ERROR)

13
DMSP Message Types

• Events
• Asynchronous notifications between user agents
  with a common data structure
• Events correlated with event listeners
• DOM events
• DOMActivate, DOMFocusIn, and DOMFocusOut
• HTML 4 events
• Click, Mouse, Key, submit, reset, etc
• Error and abort
• VXML Done (e.g., VoiceXML form complete)

14
DMSP Message Types

• Events (CONTD)
• Form Data
• One or more Field data structures (GUI or Voice)
• Recognition Results
• One or more Result data structures with raw
  utterance, score, and one or more Field data
  structures
• Recognition Results EX
• One or more Result EX data structures with raw
  utterance, score, grammar, and semantics
• Start and stop play back
• Play back of audio or TTS prompts has started or
  stopped
• Start and stop play back mark
• TTS encounters a mark in the play text
• Custom (i.e., application-defined)

15
DMSP Conclusion

• A protocol dedicated to distributed multimodal
  interaction
• Based on the Model-View-Controller design pattern
• Enables both IM and Client based View Independent
  Model topographies
• Asynchronous signals and events
• Command-response messages
• Can be generalized for other modalities besides
  GUI and Voice
• Supports application specific result protocols
  (e.g. EMMA) through extension TBD
• Interested in getting more participation