Video Textures

1
Video Textures

• This is a very cool paper.

2
Why video texturing?

• Static images are boring
• Video clips are finite in length
• We need something more interesting

3
What is video texturing?

• The process of extracting an infinitely varying
  sequence of continuous images from a finite
  length video clip
• For the purposes of creating backward compatible
  videos, it is sufficient to generate a single
  video clip with continuity between the final and
  beginning frames

4
The System
5
The System

• Objectives
• Find points in the video between which
  transitions can be made
• Smooth the transition as much as possible
• Rendering
• An infinite random play
• A finite video clip using a looping sequence to
  simulate infinite play

6
Representation

• Probability Matrix
• Each entry indicates the probability of a certain
  frame-to-frame transition being taken
• Good for dense set of possible transitions
• Link Set
• Each entry indicates a link from one frame to
  another and that transition
• Good for sparse set of possible transitions
• More typical

7
Techniques

• Splitting the video into distinct regions

8
Techniques

• Sprites

9
Extraction

• First
• Equalize brightness
• Stabilize video
• Enhance in any other way desired
• Compute the L2 distance to determine similarity
  between frames
• Store them in the matrix
• Dij Li-Lj2

10
Extraction

• Now transitions from frame i to frame j can be
  performed when Di1,j is sufficiently small (i.e.
  similar)
• Store the probability of these transitions in a
  probability matrix (or analogous structure)
• Pij exp(-Di1,j/s)
• Then normalize each transition according to the
  rule
• SjPij 1

11
The Problem

• This creates smooth frame-to-frame transitions,
  but

12
Preserving Dynamics

• Smooth frame-to-frame transitions are not
  sufficient
• Smooth motions must also be conveyed
• In other words
• For a transition from i to j to be good, the
  frames im must be similar to the frames jm

13
Preserving Dynamics

• Simply take the similarity matrix D and check for
  similarities between frames to form the filtered
  matrix D
• Dij Sk(wkDik,jk)
• where -mkm-1
• And m is the number of frames to check on either
  side (1 or 2 typically)
• Now compute probabilities based on D

14
Preserving Dynamics

• Graphically

15
Preserving Dynamics

• Much better, but

16
Avoiding Dead Ends

• Now we have a dead end
• No problem
• Simply calculate the anticipated cost of
  transitioning from one frame to another
• D''ij (D'ij)p aSkP''jkD''jk
• High p favors the fewer good transitions
• Lower p favors the abundant poor transitions
• 0.99 lt a lt 0.999 makes a good alpha
• Alpha controls the affect that future transitions
  should have
• P'' is now calculated with D''

17
Avoiding Dead Ends

• Solving the interdependencies has a simple
  iterative solution
• However, its slow to converge
• So simplify the calculation
• D''ij (D'ij)p amink(P''jkD''jk)
• This assumes sigma approaches 0 (i.e. Pij
  approaches 1) for good transitions only the
  best transition is considered

18
Avoiding Dead Ends

• See paper for details on iterative solution

19
Sequencing Random Play

• Random (infinite) play
• Begin sequence at any transition before the last
  non-zero probability transition
• while (1)
• Determine transition after frame i by finding
  frame j such that Pij is greatest
• Generates video textures that never repeat exactly

20
Sequencing Video Loops

• Primitive loop
• Single transition from i to j
• Must satisfy i j to guarantee loop
• Properties
• Range j,i
• Cost Dij
• Length i-j

21
Sequencing Video Loops

• Compound Loop
• Combination of one or more primitive loops (or
  compound loops)
• Each loop must overlap with at least one other
• Properties
• Range Uk range(Tk)
• Cost Sk cost(Tk)
• Length Sk length(Tk)

22
Sequencing Video Loops

• Selecting Transition Sets
• Need to find the optimal loop for a given
  sequence length
• This is just a simple DP algorithm

23
Still Sequencing Video Loops
24
Sequencing Video Loops
25
Sequencing Video Loops

• Scheduling
• Now we have a set of transitions to use
• All we know, based on the properties of compound
  loops is that the set represents a continuous
  range of frames
• Must schedule the set into a coherent sequence of
  transitions

26
Sequencing Video Loops

• Algorithm for Scheduling Primitive Loops
• 1. Schedule the transition at the end of the
  range first
• 2. Remove transition from set this may leave
  one or more disjoint sets
• 3. Schedule the next transition from the first
  disjoint set that begins after the beginning of
  the removed transition
• 4. Repeat 2-3 until no transitions remain

27
Rendering

• Between transitions
• Cross fade
• Morph or de-ghosting
• Between film regions
• Feathering

28
Questions?

• Is cyclified a word?