MediaHub: An Intelligent MultiMedia Distributed Platform Hub

1
MediaHub An Intelligent MultiMedia Distributed
Platform Hub
MediaHub

• Glenn Campbell, Tom Lunney Paul Mc Kevitt
• School of Computing and Intelligent Systems
• Faculty of Engineering
• University of Ulster, Magee Campus
• Derry/Londonderry
• Northern Ireland
• Campbell-g8, TF.Lunney, P.McKevitt
  _at_ulster.ac.uk

2
Outline

• Research objectives
• Related research
• Architecture of MediaHub
• Dataflow
• Semantic representation/storage
• Communication
• Decision-making in MediaHub
• Future development

3
Research Objectives

• Interpret/generate semantic representations of
  multimodal input/output
•  
• Perform fusion and synchronisation of multimodal
  data (decision-making)
•  
• Implement and evaluate a multimodal platform hub
  (MediaHub)

4
Key research problems

• Semantic representation and storage?
• Communication?
• Decision-making?

5
Related Research

• CORBA (Vinoski 1993)
• COLLAGEN (Rich et al. 1997)
• Open Agent Architecture (Cheyer et al. 1998)
• Chameleon (Brøndsted et al. 1998)
• Ymir (Thórisson 1999)
• Interact (Jokinen et al. 2002)
• SmartKom (Wahlster 2003, 2006)
• Psyclone (Thórisson et al. 2005)
• Hugin (Jensen 2001)

6
Architecture of MediaHub
7
Architecture of MediaHub
8
Dataflow in MediaHub
Marked-up MultiModal Input/Output (XML)
Dialogue Manager
MediaHub Whiteboard (EMMA)
Decision-Making Module
Hugin Decision Engine
9
Semantic Representation

• XML used for input/output data
• Well established standard mark-up language
• Allows MediaHub to be integrated into other
  existing multimodal systems
• XML input is validated against a Document Type
  Definition (DTD)
• Using EMMA (Extensible MultiModal Annotation
  mark-up language) for semantic representation
• EMMA is a derivative of XML
• EMMA is suited to representing confidences
  relating to multimodal data (confidence tag)

10
Example XML input file

• lt?xml version"1.0"?gt
• lt!DOCTYPE multimodal SYSTEM "C\Psyclone2\MediaHub
  Input.dtd"gt
• lthypothesesgt
• lthypothesis1gt
• ltlanguagegt
• ltmatchgt
• ltyesgt0.8lt/yesgt
• ltnogt0.2lt/nogt
• lt/matchgt
• ltconfidencegt
• ltyesgt0.9lt/yesgt
• ltnogt0.1lt/nogt
• lt/confidencegt
• lt/languagegt
• ltgesturegt
• lt/gesturegt
• ltreferentObjectgt

11
Semantic Storage

• Blackboard-based method of semantic storage
• Marked-up input in EMMA format stored on central
  whiteboard (MediaHub Whiteboard)
• All input/output messages in MediaHub are stored
  on whiteboard and can be accessed at any stage in
  the decision-making process
• Whiteboard and Dialogue Manager form kernel of
  MediaHub

12
Communication

• MediaHub uses Psyclone for distributed processing
• Psyclone uses OpenAIR specification for
  communication
• Modules of MediaHub communicate by passing
  messages through MediaHub Whiteboard
• Implements a publish-subscribe architecture
• For example, Decision-Making Module registers for
  messages of type input
• All messages relating to input posted on
  whiteboard will automatically be sent to
  Decision-Making Module
• Module registration is done in XML specification
  file, called PsyProbe, run automatically at
  start-up

13
PsySpec Example

• ltexecutable name"DMM" consoleoutput"yes"gt
• ltsys ostype"Win32"gt
• java -cp .JavaOpenAIR.jar DMM
  psyclonehostport namename
• lt/sysgt
• lt/executablegt
• ltspecgt
• lttriggers from"any"
  allowselftriggering"no"gt
• lttrigger type"input"/gt
• lttrigger type"MediaHub.shutdown"/gt
• lt/triggersgt
• ltpostsgt
• ltpost to"MediaHub_Whiteboard"
  type"dmm.register" /gt
• lt/postsgt
• lt/specgt
• lt/modulegt

14
Decision-making

• MediaHub employs Bayesian decision-making over
  multimodal data
• Bayesian networks developed using Hugin software
  tool (Jensen 2001)
• Networks are accessed using Hugin API (Java)
• A unique approach to decision-making in an
  intelligent multimedia distributed platform hub

15
Hugin

• Tool for implementing Bayesian Networks as CPNs
  (Causal Probabilistic Networks)
• Hugin GUI
• Graphical user interface to Hugin decision engine
• Hugin API
• Library implemented in Java
• Allows programs to implement Bayesian Networks
  for decision-making

16
Bayesian Networks

• AKA Bayes nets, Causal Probabilistic Networks
  (CPNs), Bayesian Belief Networks
• Consists of nodes and directed edges between
  nodes
• Node represents a variable
• Influence between nodes represented by edges

               
 
17
MediaHub Example Network

• G1-3 represents the belief that the user is
  referring to Objects 1-3, based on gesture input
• L1-3 represents the belief that the user is
  referring to Objects 1-3, based on language input
• CG1-3 and CL1-3 represent the confidence
  associated with G1-3 and L1-3

18
Bayesian Network Design Process

• Characterise decision-making scenarios
• Design Bayesian networks for decision-making
  scenarios
• Use the Hugin GUI to build Bayesian networks and
  complete conditional probability tables
• Run and test networks, making changes to networks
  and tables as required
• Develop Java code that will open, edit and run
  the Bayesian network using the Hugin API

19
Decisions in MediaHub

• Input
• Determining semantic content of input
• Fusing semantics of input
• Resolving ambiguity at input
• Output
• Synchronising multimodal output
• Best modality for output

20
Input example
Copy all files from the process control folder
of this computer to a new folder called check
data on that computer.
21
Output Example
This is the route from Pauls office to Toms
office.
T
P
22
Conclusion

• An intelligent multimodal distributed platform
  hub called MediaHub is under development
• MediaHub interprets/generates semantic
  representations of multimodal input and output
• MediaHub performs fusion and synchronisation of
  multimodal data
• MediaHub provides a new method of decision-making
  within a distributed platform hub

23
Future development

• Define all necessary decisions for example
  scenarios
• Develop Bayesian decision-making using Hugin API
  (Java)
• Develop a GUI to illustrate the functionality of
  MediaHub
• Test MediaHub on example scenarios
• Compare MediaHub to other systems
• Write thesis

24
Questions?