Automatic Manifestation of Composite Multimedia Objects

1
Automatic Manifestation of Composite Multimedia
Objects

• Ahmed Gomaa
• Nabil Adam
• Vijay Atluri

2
Outline

• Motivating Example
• Introduction to CMO
• Types of constraints
• Universal Access
• Framework
• CMO specification
• Petri net layer
• MMCTPN
• SMIL layer
• Demo
• User Interaction
• Related work
• Future Work

3
Current Situation

• What to do to select your business location?

Currently available on NJ commerce commission and
partners sites
4
Current Situation

• Easy to get lost between the different links.

5
Composite multimedia Object (CMO)
6
Composite multimedia Object (CMO)

• Comprised of different media component such as
  text, image, video, audio, shape files,...
• Variety of relationships among components.
• different types of constraints that must be
  adhered to when rendering it.
• (e.g., Fidelity gt 640480).
• Associated with each component is a set of
  parameters.
• (e.g., Fidelity 800600).

7
Types of Constraints

• Synchronization constraints
• Map must appears in sync.with Text 1
• Fidelity Constraints
• Map must be displayed at a resolution of at least
  640480
• Spatial Constraints
• Fly by is above Text
• Security Constraints
• Sales representative audio and video are accessed
  by subscribers only.

8
Universal Access

• Need to facilitate access to desired CMO
  according to the various users
• Capabilities ( e.g, devices)
• Characteristics (e.g, expertise)
• Credentials (e.g, subscribers)
• When a subject requests a CMO, he may not be able
  to view the entire CMO.
• For example, due to the limitations of his
  appliances, or due to lack of his credentials to
  satisfy the security requirements.

9
Framework

• CMO layer
• Comprises of the formal specification of the CMO,
  such that specifying each components in the CMO
  along with its associated constraints.
• Petri Net layer
• Formal model that can describe,visualize and
  validate the CMO constraints.
• Conceptual tool .
• SMIL layer
• Implementation and automatic rendering purpose.

10
CMO specification

• CMO (c 1 ...,c n, S )
• Each c i is component of CMO
• S is a set of synchronization,spatial,fidelity,
  and security constraints.
• Each component c i in turn,is a tuple c
  (ml,fd,pd)
• ml represents the modality
• fd represents the fidelity
• pd represents the playback duration

11
Constraints

• Synchronization Constraints
• We only need 3 from the 7 temporal relations
• Meets video meets image
• Sync video sync text
• Before

12
Constraints

• Fidelity Constraints
• A set of fidelity constraints for modality ml .
• FD (ml )is set of fidelity ranges fd
• for example gt 640 480 is to specify that the
  resolution be at least 640 480 to view the
  object.
• Spatial Constraints
• Where to place the components on the screen

13
Constraints

• Security Constraints
• Let CR cr 1, cr 2 ... denote the set of all
  distinct credentials relevant to the multimedia
  object.
• Each subject S possesses set of credentials
  denoted as s CR
• A security constraint can be
• (1)Each component c i of the multimedia object is
  associated with set of credentials,denoted c i CR
  .
• (2)A set of components (c i c j ,...)is
  associated with a set of credentials,denoted (c i
  c j ,...)CR .
• E.g., video2 ( v2) need subscription credentials
  so,
• C i CR V 2 subscription

C i CR V 2 subscription
14
Petri net layer

• Token
• Place
• A marking is an arrangement of tokens in places,
  representing state
• Arc
• An arc connects a place and a transition
• Transition
• Inter-object sync.
• A transition is enabled if there is at least one
  token at each of its input places
• Firing rule
• the activity may take place if transition enabled.

p0
t0
p2
p1
t2
t1
p4
p3
15
Colored Petri Nets (CPN)
Before firing
Initial marking

1. p1 has 4 tokens, 2 as and 2 ds
2. p2 has 6 tokens, lta,bgt, ltb,cgt, ltd,agt
3. p3 and p4 has no tokens.

After firing with substitution ax, by, cz
Execution

1. p1 loses 2 tokens of x
2. p2 loses ltx,ygt and lty,zgt
3. p3 gets ltx, zgt
4. p4 get e (constant).

16
MMCTPN Model for a CMO plan

• MMCTPN (Multimedia color-time Petri-Net) consists
  of
• Color-time Petri-Net
• Tokens Color sets that represent
• Types of Fidelity (e.g.,the resolution of the MM
  component)
• Absence of fidelity capabilities is represented
  as Holes
• Types of credentials (e.g., credentials for
  researchers)
• Absence of credentials of the subject requesting
  the object is represented as Holes
• Types of Modality (e.g., image, video,text,
  audio)
• Absence of modality capabilities is represented
  as Holes
• Places
• Includes the length of time the multimedia
  component is played.
• A color set that is needed to activate/ play the
  component.
• Null place in each subMMCTPN (A part of the net
  between two subsequent transitions)

17
MMCTPN

• Credentials and capabilities are represented as
  tokens and absence of them are represented as
  holes.
• Author rules are represented as doted holes.
• The initial place is marked with these tokens to
  start with.
• Tokens are said to be available if it remains in
  a place for their specified duration.
• Transitions are enabled once any token or hole
  are available in all input places.

t
18
MMCTPN Object Manifestation
19
CMO Plan
20
MMCTPN execution
ltor,gy,brgt dur 40s
ltre,bc,brgt dur 15s
ltye,bc,brgt dur 10s
P1
P3
P6
P7
P4
ltgr,br,aqgt dur 15s
ltallgt dur 40s
ltallgt dur 15s
ltbl,wh,ppgt dur 15s
Ps
Pf
null
null
ltallgt dur 15s
null
P8
ltre,gy,br,aqgt dur 15s
P5
P2
P9
ltbl,wh,ppgt dur 15s
ltbl,gy,brgt dur 40s
ltbl,wh,brgt dur 15s
21
Modified CMO Plan
ltye,bc,brgt dur 10s
P6
P4
ltgr,br,aqgt dur 15s
ltallgt dur 40s
ltallgt dur 15s
ltbl,wh,ppgt dur 15s
Ps
Pf
null
null
ltallgt dur 15s
null
P5
P2
P9
ltbl,wh,ppgt dur 15s
ltbl,gy,brgt dur 40s
ltbl,wh,brgt dur 15s
22
Petri Net To SMIL

• Equals ( SYNC)
• ltpar dur"30s"gt
• ltimg id"foo" src"a.jpg"/gt
• lttext  src"text.html" /gt  
• ltaudio src"audio.au" /gt
• lt/pargt
• ------------ image
• ------------ text
• ------------ Audio
• 30s

• After ( Before)
• ltseqgt
• ltimg src"a.gif" dur"6s" /gt
• ltimg src"b.gif" dur"4s" begin"1s" /gt
• lt/seqgt
• ----------
• 6 sec ---------
• 4sec
• --------------------------
• 11 seconds

23
SMIL layer

• SMIL-based rich multimedia presentations
  integrate several types of media.
• SMIL is XML-based, thus flexible and extensible.
• SMIL is W3C supported, thus enjoys cross industry
  support.
• SMIL is supported by readily available tools and
  parsers.

24
Demo

• http//cimic.rutgers.edu/ahgomaa/ua/presentations
  /demo3/egov.smil
• http//cimic.rutgers.edu/ahgomaa/ua/presentations
  /demo3/egov-adj.smil

25
SMIL 2.0 Implementation

• we have adopted the security model and the
  language (XACL) in specifying the security
  constraints on the multimedia object. We have
  used the XACL visual tool,tool from IBM XML
  Security Suite.

26
Incorporating User interactivity

• Identify and model the different types of user
  interactions.
• Make sure that different constraints are still
  valid.
• Visualize and analyze the effect of the user
  interaction on different constraints.

27
User interaction categories

• Temporal Interaction
• continuous component, ( ex. video )
• ''play'', 'fast forward''
• non-continuous component(ex. image)
• "begin", "end", "pause".
• Spatial interaction
• "move , "Zoom" , maximize, "change font".
• Undo Redo
• The user may want to interact with the CMO as one
  object, e.g., pause , begin,

28
User Interaction example

• If a user wants to stop the fly by video, what
  will be the effect on the following text?
  (rendering)
• Appear immediately after stop.
• Appear after a defined amount of time.
• If a user wants to maximize the map, what will be
  the effect on rendering the following image? (
  conflict)
• Rendered behind the map.
• Rendered after another action.
• The CMO author need to
• Visualize
• Analyze

29
User Interaction Problems

• How to model different types of user interaction
  on the composite multimedia object plan?
• How can the author visualize and analyze the
  impact of potential user interaction on the
  rendering of the CMO?
• How to detect potential conflicts between
  different types of constraints?
• How to resolve different conflicts between
  constraints?

30
Implementation with user/web friendly with
flexibility and interoperability (SMIL 2.0)
PETRI-NET Modeling for validation and
analysis
CMO constraints Uniform Model
Conflict resolution
31
Related Work

• Multimedia object rendering
• Security
• User interaction

32
Related Work

• Multimedia Object Rendering
• Adam and Atluri et al.(2001) Presented the
  Universal Access problem and the Oblet approach.
• Uses Petri Net to present temporal, spatial,
  modality and fidelity constraints.
• Bertino et al.(2000) present a system called MPGS
  - Multimedia Presentation Generator System.
• Enables specification of synchronization and
  spatial constraints
• Capable of analyzing the consistency among these
  two types of constraints.
• None of those papers addressed the security
  constraints in a CMO

33
Related Work

• Security
• Damiani et al.(2000),Bertino et Al.(2000) and
  Kudo et al (2000) addressed access control models
  in XML.
• We adopted Kudo et al. approach in presenting a
  new Petri Net model that incorporate the security
  constraints in the CMO.

34
Related Work

• User interaction
• User interaction research within CMO may be
  divided into three categories
• Formal modeling
• Programming approach
• Statistical techniques
• Formal modeling, using Petri net
• Song et al.(1996) presented Timed Petri Net
  (TPN).
• Guan et al. (1998) presented a distributed object
  composition Petri net (DOCPN).
• Prabhakaran et al.(1993) presented a dynamic
  timed Petri nets (DTPN) model.
• All models are restrictive in types of
  interactions.
• They only deal with the synchronization
  constraints.

35
Related Work

• Programming approach
• Partially support user interactions for online
  CMO.
• LimSee(2002), Grins (2002), Yang (2001).
• All of which based on time line events
• which leads to dead times if the CMO is
  adaptable.
• Statistical techniques
• Predict online user interactions
• Hollfelder et al. (2000) where they model the
  user behavior as a Continuous Time Markov chain
  (CTMC).
• Boll et al.(2001) implementing adaptive streaming
  of MPEG videos for interactive internet
  applications to support jumping to bookmarks
  within the MPEG

36
Future Work

• Enforcing Security Constraints.
• Secure the plan on the client side (
  certificates, relational transducer).
• Allowing User Interaction.
• Conflict Identification and Resolution Strategy.