Video on the Semantic Web Experiences with Media Streams

1
Video on the Semantic WebExperiences with Media
Streams

• CWI Amsterdam
• Joost Geurts
• Jacco van Ossenbruggen
• Lynda Hardman
• UC Berkeley SIMS
• Marc Davis

2
Talk Overview

• Video Generation
• Annotating Video/Media Streams
• Porting Media Streams
• Video generation using semantic web technology
• Conclusion

3
Why video?

• Video is powerful information source but
  complicated on syntax and semantics done by
  mostly domain experts.
• Search/Retrieve is difficult for humans near to
  impossible for machines.
• Media metadata is not DC labels, real need for
  formal semantics
• Media support is lacking/under developed on the
  semantic web

4
Why video generation?

• Practical experience with semantic web technology
  from a media perspective
• Media typically is knowledge intensive
• Ambitious application to trigger problems

5
What can we generate?

• Cinema is like a language which has structure
  which can be manipulated
• Establishing shot gives the impression an event
  is happening at a specific location
• Reaction shot gives the impression an actor is
  reacting to an event.
• Example

estab1.smil
estab2.smil
6
Continuity editing in cinema

• Continuity editing rules ensures consistency
  between shots.
• Mise en scene
• Indoors - Outdoors
• Clothing/costume
• Weather conditions
• Cinematography
• Black/white Color
• 180 degree rule (walking direction)
• Framing
• Jump cut

7
Continuity editing

• Works best with generic objects/characters
• Not so well with recognizable objects/characters
• Theoretic principles in cinema can be formalized
  and used to automatically generate video
  sequences.

8
Trip report scenario

• Student visits Berkeley informs colleagues about
  his trip.
• Scenario uses establishing and reaction shots.
• scenario_iswc2004.smil

9
Scenario wrap up
10
Generating an establishing shot

• User
• defines location
• defines character
• Continuity rules
• angle shot 1 gt angle shot 2
• Both shots outdoors
• Annotation requirements on
• Mise en scene (describing scene)
• Cinematography (material, lens foci)

11
Annotating Video in Media Streams (Davis 1995)

• Retrieve/combine existing material for reuse
• Content driven annotations
• Multiple descriptive dimensions
• Icon based
• 7000 icons in ontology 10000 annotations
• subClass, partOf, looksLike, occursWith

12
Media Streams Timeline
13
Media Streams Icon Space
14
Porting Media Streams to the Semantic Web (syntax)

• Media Streams application
• 5 10 year old Lisp using old libraries
• Programmers moved away
• Source code only partly available
• Ontology and annotations are in binary format
• Not efficiently scalable
• Reverse engineering
• Important working legacy application
• Lessons learned
• Knowledge of the application is required to
  understand the ontology and the annotations.
• Focus on syntax

15
Porting Media Streams to semweb (semantics)

• Ontology
• subClass relationship mapped to rdfssubClassOf
• partOf, looksLike, occursWith relations mapped to
  generic rdfproperty (future work OWL)
• Modularity for streams in ontology

XML ltcidi id"4497"gt ltnamegttelephone
booklt/namegt lthasSuperClassesgt ltcidi
id"4499" name"book" /gt lt/hasSuperClassesgt
ltoccursWithgt ltcidi id"4455" name"telephone"
/gt lt/occursWithgt lt/cidigt
RDF ltrdfsClass rdfID"CIDI_4497"gt ltrdfslabel
xmllang"en"gttelephone booklt/rdfslabelgt
lt!--subclassOf book--gt ltrdfssubClassOf
rdfresource"CIDI_4499"/gt lt!--cidi4455
telephone--gt ltoccursWith rdfresource"CIDI_445
5"/gt lt/rdfsClassgt
16
Human cognition vs. machine cognition.

• Media Streams concepts were represented by icons
  with a textual label.
• In RDF representation there is only the label
• Icons can denote complex actions not
  representable by a single word
• Need for WorldNet to retrieve synonyms (hack!)
• Just RDF is not enough

17
Porting Media Streams to semweb (semantics)
18
Porting Media Streams to semweb (semantics)

• Lessons learned
• Annotations are not instances of the ontology but
  refer to it
• Structure embodies semantics
• Need for annotation template which describes
  structure

19
Video generation

• Need for detailed annotations result in detailed
  queries
• Detailed queries give fewer results
• Need to be able to relax query
• Specification (ask pants, retrieve jeans)
• Generalization (ask jeans, retrieve pants)
• System searches for shots which comply with
  continuity constraints
• Query or Rule?

20
Conclusion for porting knowledge to semweb

• Thoroughly understand the original application
  domain before porting knowledge to the semantic
  web
• First focus on porting knowledge to an accessible
  format such as XML postponing modeling issues.
• Annotations are not necessarily instances of an
  ontology but can refer to it, in which case a
  annotation template defines the structure of the
  annotation.

21
Conclusion for applications on the semantic web

• Need for both, precise queries and, queries which
  allows for relaxing.
• Distinction between queries and rules is small.
• Combining proprietary heterogeneous knowledge
  sources on the semantic web leads to
  inconsistencies which have to be dealt with.

22
Take home message

• Legacy sources are because of their longtime
  existence valuable resources worthwhile porting
  to the Semantic Web. Best practices guidelines
  are needed to facilitate this.
• Ambitious applications test and give requirements
  on technology

23
Video on the Semantic WebExperiences with Media
Steams
24
Scratch
lt!-- Character (joost)--gt ltoccurence
id"character1"gt ltstartframegt1lt/startframegt
ltendframegt139lt/endframegt ltcompound
id"joost"gt ltnamegt ltcidi
id"3990" name"BODY"/gt lt/namegt
ltvaluegt ltcompound id"joost-body"gt
ltslotgt ltnamegtltcidi id"6532"
name"APPARENT-BODY"/gtlt/namegt
ltvaluegt ltcompound
id"joost-apparent-body"gt ltslotgt
ltnamegtltcidi id"6553"
name"SEX/AGE"/gtlt/namegt
ltvaluegtltcidi id"3868" name"adult
male"/gtlt/valuegt lt/slotgt
ltslotgt ltnamegtltcidi
id"6557" name"SKIN-COLOR"/gtlt/namegt
ltvaluegtltcidi id"6570" name"dark peach
skin"/gtlt/valuegt lt/compoundgt lt/compoundgt
lt/occurence

• Video generation requires detailed, multi
  dimension descriptive annotations and is
  therefore a well suited test case for semantic
  web technology read some knowledge intensive
  applications, like video generation are dependent
  on shared knowledge sources provided by the
  semanticweb. The semantic web should support
  these. read Video generation is a real
  application giving insights in practical problems
  with the semantic web read technology should
  subordinate applications read ambitious
  applications show deficiencies in technology

25
Content structure vs. Documents structure (a)

• Different ways of annotating
• Structure based (scene, shot, frame)
• Context important
• Content based ( Character walks from left to
  right)
• Context not important

26
Semantics after Syntax (b)
27
Ontology and Annotation (c)
28
Query requirements (d)
29
Rules (e)