Automatic Video Editing

1
Automatic Video Editing

• Stefano Bocconi
• CWI Amsterdam
• Stefano.Bocconi_at_cwi.nl

2
Talk Outline

• Open Video Documentaries
• Rhetoric for Video Documentaries
• VOX POPULI, a video editing engine
• Future work

3
The General problem

• Affordable means of producing video material
• The Web as an interactive distribution channel
• How to make video material accessible on a
  semantic level?
• How to expose the messages contained in the
  footage?

4
The case for Open Video Documentaries

• There is only one final version what is shown is
  the choice of the author/editor
• If the subject is controversial, the multiplicity
  of viewpoints can be lost
• Users have high-level semantic questions, like
• Is opinion X supported by people? How?
• Show me the argumentation of people that have
  opinion X

5
Video Material

• Interview with America 8 hours of footage with
  interviews and background material
• Subject the opinion of American people after
  9-11
• Shooting from 27-10-2001 to 01-11-2001 in
  Stamford (CT), New York (NY), Boston (MA) and
  Cleveland (OH)
• Online at www.interviewwithamerica.com

6
Example (I)

• Topic War in Afghanistan, Show one pro opinion
  and refute it

7
Example (I)

• Topic War in Afghanistan, Show one pro opinion
  and refute it

• I am never a fan of military action, but I can
  not think of a more effective solution
• War has never solved anything
• Two billions dollar bombs on tents

8
Toulmin's Layout of Argumentation
9
Building Block for automatic editing
Two billions dollar bombs on tents
contradict
I can not think of a more effective solution
Claim
weaken
Concession
I am not a fan of military actions
support
War has never solved anything
10
Building Block for automatic editing
Support / contradict
Support / contradict
11
Processing the Statements

• Statement formally annotated
• war best solution
• Generated contradicting statements
• war worst solution
• war bad solution
• diplomacy best solution
• war not solution

12
Rhetoric applied to Video

• Need to formalize means to generate opinionated
  arguments
• The ability, in each particular case, to see the
  available means of persuasion (Aristotle)
• Logos appeal based on logic or reason
• Ethos appeal based on the character of the
  speaker
• Pathos appeal based on emotions

13
Ethos

• The authority of the speaker
• Rates the interviewee according to a simple user
  model
• Interviewees are annotated with social
  characteristics
• Age
• Education
• Employment
• Race
• Religion
• Sex

14
Pathos Example (II)

• Appeal to the emotions of the viewer

15
Pathos

• Images annotated with concepts
• Based on a user model
• Possible improvement Ortonys emotions theory

concept
Heiders balance
/-
/-
viewer
interviewee
/-
16
Domain commitment

• Requirements
• Domain independent annotations and process can
  be applied to other domain (currently working on
  another domain)
• Different material can be added if annotated
• Relations between concepts (e.g. war diplomacy)
• Hierarchy of concepts
• The resulting ontology can be user dependent

17
The annotations

• Descriptive on visual information
• Filmic
• Interviewee (social)
• Question
• Rhetorical on verbal information
• Rhetorical Statement
• Applied to Media items
• Interview segment
• Statement
• Image

18
VOX POPULI Results

• Provide support to access video material in an
  argumentative manner
• Domain dependency located in the ontology and
  user model, not in the process
• Data-driven using existing material not crafted
  for the application
• User-driven, purposeful composition of (short)
  video documentaries some awareness of the
  created meaning

19
Other Approaches

• Media Streams no awareness of the message
• Evolving Documentary association via keywords,
  no composition of a message
• Terminal Time footage crafted for the task,
  closed system
• Auteur only visual (continuity-driven),
  different rhetoric

20
Steps Ahead based on verbal

• Up till now results on the micro level
• Define narrative strategies based on
• Evolution Exposition, Inciting moment,
  Complication, Resolution (Freytags triangle).
  Toulmins model allows directionality (from data
  to claim)
• Intensity Toulmins model allows intensity (the
  qualifier), intensity also contained in the
  modifiers of the statements
• Parallel narratives, possibly converging to the
  same claim

21
Filmic Goals

• The visual narrative must support the verbal
  narrative, at least continuity
• Not so many constraints in the chosen genre
• use metaphors, like the day, the journey
• The visual should support the rhetoric
• Invert the roles, use visual to construct a
  narrative and verbal to support it
• A visual analogous to Toulmins model?

22
Example (III)
23
Filmic Techniques

• Rhythm
• Counterpoint editing
• Transitions
• Spatial arrangement
• Shaffs cinesthetic structures
• Separation
• Familiar image
• Slow Disclosure

24
Questions?

• Thanks for your attention

25
More Future work

• False implications
• Logical fallacies
• Process
• Rule-Based Tunable
• Self Rating

26
Vision

• Generation of meaningful video sequence
  bottom-up approach to the creation of meaning,
  how much knowledge needed to produce what quality
  of meaning

27
Genesis of the idea

• Video Retrieval
• Capable of retrieving (annotated) video sequences
  by content
• The hammer hitting a nail User-Query
• If the Video not available automatic composition
• Automatic Video Editing
• Engine knows the hammer is hitting the nail

28
Example of Toulmin sentence

• Concession There are solutions
• Claim Nobody is trying to find a solution
• Data there is too much money to be made
• Warrant Solutions other than war means loosing
  money
• Backing Hospital supplies, food, oil