MediaHub: An Intelligent Multimedia Distributed Hub

1
MediaHub An Intelligent Multimedia Distributed
Hub
IntelliMedia

• Student Glenn Campbell
• Supervisors Dr. Tom Lunney
• Prof. Paul Mc Kevitt
• School of Computing and Intelligent Systems
• Faculty of Engineering
• University of Ulster, Magee

2
Project goals

• The primary objectives of this research are to
• Interpret/generate semantic representations of
  multimodal input/output
•  
• Perform decision-making (fusion and
  synchronisation) over multimodal data
•  
•  Implement MediaHub, a multimodal platform hub
•  

3
Project objectives

• Focus on following research questions
• Will MediaHub use frames for semantic
  representation or XML or one of its derivatives?
• How will MediaHub communicate with various
  elements of a platform?
• Will MediaHub constitute a blackboard or
  non-blackboard model?
• What mechanism will be implemented for
  decision-making within MediaHub?

4
Key research problems

• Semantic Representation
• Represent language and vision
• Frames or XML?
• Semantic Storage
• Blackboard model?
• Non-blackboard model?
• Decision-making
• Fusion and synchronisation
• AI technique

5

Semantic representation

• Frames (CHAMELEON)
• MODULE
• INPUT input
• INTENTION intention-type
• TIME timestamp
•  
• SPEECH-RECOGNISER
• UTTERANCE(Point to Hannes office)
• INTENTION instruction!
• TIME timestamp
•  
• GESTURE
• GESTURE coordinates (3, 2)
• INTENTION pointing
• TIME timestamp

• XML (M3L, SmartKom)
• ltpresentationTaskgt
• ltpresentationGoalgt
• ltinformgt ltinformFocusgt ltRealizationTypegtlist
  lt/RealizationTypegt lt/informFocusgt lt/informgt
• ltabstractPresentationContentgt
• ltdiscourseTopicgt ltgoalgtepg_browselt/goalgt
  lt/discourseTopicgt
• ltinformationSearch id"dim24"gtlttvProgram
  id"dim23"gt
• ltbroadcastgtlttimeDeictic id"dim16"gtnowlt/timeDei
  cticgt
• ltbetweengt2003-03-20T194232
  2003-03-20T220000lt/betweengt
• ltchannelgtltchannel id"dim13"/gt lt/channelgt
• lt/broadcastgtlt/tvProgramgt
• lt/informationSearchgt
• ltresultgt lteventgt
• ltpieceOfInformationgt
• lttvProgram id"ap_3"gt
• ltbroadcastgt ltbeginTimegt2003-03-20T195000lt/beginT
  imegt
• ltendTimegt2003-03-20T195500lt/endTimegt
• ltavMediumgt lttitlegtTodays Stock
  Newslt/titlegtlt/avMediumgt

6
Semantic storage

• Blackboard or Non-blackboard?
• High coupling Blackboard?
• Low coupling - distributed architecture?
• Communication
• Via central blackboard?
• Message passing between modules?

7
Decision-making (fusion synchronisation)

• Rule-based
• Potential for Other AI techniques
• Fuzzy Logic
• Neural Networks
• Genetic Algorithms
• Bayesian Networks (CPNs)

8
Distributed processing

• PVM (Parallel Virtual Machine)
• (Sunderam 1990, Fink et al. 1995)
• ICE (Amtrup 1995)
• DACS (Fink et al. 1995, 1996)
• Open Agent Architecture (OAA)
• (Cheyer et al. 1998, OAA 2004)
• JATLite (Kristensen 2001, Jeon et al. 2000)
• JavaSpaces (Freeman 2004)
• CORBA (Vinoski 1993)

9
Intelligent Multimedia Distributed Platforms

• Blackboard Model
• Ymir (Thórisson 1999)
• CHAMELEON (Brøndsted et al. 1998, 2001)
• Smartkom
• (Bühler et al. 2002, Wahlster et al. 2001,
  SmartKom 2004)
• DARBS (Nolle et al. 2001)
• DARPA Galaxy Communicator (Bayer et al. 2001)
• Psyclone (Psyclone 2004)
• Spoken Image/SONAS
• (Ó Nualláin et al. 1994, Ó Nualláin Smith
  1994,
• Kelleher et al. 2000)

10
Intelligent Multimedia Distributed Platforms

• Non-blackboard Model
• WAXHOLM (Carlson et al. 1996)
• AESOPWORLD (Okada 1996)
• COLLAGEN (Rich et al. 1997)
• INTERACT (Waibel et al. 1996)
• Oxygen (Oxygen 2004)
• EMBASSI (Kirste 2001, EMBASSI 2004)
• MIAMM (MIAMM 2004)

11
CHAMELEON

• Language vision integration system
• consists of ten modules, mostly programmed in C
  and C
• DACS communication system used for communication
• Blackboard stores semantic representations
  produced by other modules
• Communication between modules achieved by
  exchanging semantic representations between
  themselves or blackboard
• Semantic representation in form of input, output
  and integration frames

12
Architecture of CHAMELEON
13
SmartKom

• User adaptive interface for human-computer
  interaction
• Mobile
• Public
• Home/Office
• Facilitates speech, gestures and facial
  expression input
• XML-based mark-up language, M3L, used for
  semantic representation
• Distributed multiple blackboard model

14
Architecture of SmartKom
15
Project proposal

• Dialogue Manager
• Acts as a blackboard module
• Facilitates communication between other modules
• Synchronisation
• Semantic Representation Database
• Provides semantic representation of language and
  vision data
• Decision Making Module
• AI technique for a unique form of decision-making
• Bayesian Networks (CPNs)
• Neural Networks, Genetic Algorithms, Fuzzy Logic

16
Architecture of MediaHub
17
Comparison of Intelligent MultiMedia Platforms
18
Software Analysis

• Main Programming Language
• Java
• C
• Semantic Representation
• XML
• XHTML Voice
• SMIL
• RDF Schema
• MPEG-7
• Decision Making
• HUGIN (Bayesian Networks) (Hugin 2004)
• FuzzyJ Toolkit (Fuzzy Logic) (NRC 2004)

19
Project Schedule
20
Conclusion

• An intelligent multimodal distributed platform
  hub called MediaHub will be developed
• MediaHub will interpret and generate semantic
  representations of multimodal input and output
• MediaHub will perform fusion and synchronisation
  of language and vision data
• Unique contribution of MediaHub is to provide a
  new method of decision making
• MediaHub will be tested within an existing
  multimodal platform (e.g. CONFUCIUS)