Video Mosaics

1
Video Mosaics

• CS 294/283 Final Project
• Steve Martin
• U.C. Berkeley, Fall 2003

2
Overview

• Previous Work
• Method Overview
• Metrics for Comparing Video
• Preliminary Results
• Future Work
• Questions?

3
Previous Work on Mosaics

• Image Mosaics
• 1970s Chuck Close/Salvadore Dali make image
  mosaics by hand
• 1990s Papers at SIGGRAPH and commercial
  programs create image mosaics
• Video Mosaics
• 2002 Video Mosaic program created at Princeton,
  work published at NPAR 2002
• Example. . .
• 2003 Wells Fargo commercial (done by hand)
• Example. . .

4
Method Overview

• Step 1 Get a LOT of source video
• Quality of results highly dependant on quality
  and amount of component clips!
• In our case, problem solved!
• Got stock video companies donate video to the war
  effort
• We have 200GB of high quality video to play with
  (more than 3000 clips)!
• Step 2 Preprocess video
• Create a database of feature values for all
  video.
• Each clip further subdivided into tiles
• Size determined by the user.
• With 20 by 20 pixel tiles, each movie has 864
  tiles.
• Metrics are run on each tile, values recorded
• This is done with a separate program that crawls
  the video repository

5
Method Overview (2)

• Step 3 Create video mosaic
• Load in original video
• Parse the previously generated database
• For each clip in the original video
• Run the same metrics as was used to generate the
  database
• Search the database for the closest tile
• Replace the original tile with one from another
  movie, continue
• Note that we cant just brute force compare video
  or well be computing forever.
• How to come up with numbers that describe tiles?

6
Video Comparison Metric 1 Average Color

• Want replacement clips to have close to or same
  average color as original tile.
• To calculate
• Just add color values of pixels in movie tile and
  divide.
• Actually not terrible by itself. . .
• As tiles get smaller, approach pixel size, eye
  blends them
• However, note that average color can be fooled
• i.e. a clip that is half red and half green
  averages to olive brown
• Need something else. . .

7
Video Comparison Metric 2 Color Histograms

• In addition to average color, want some kind of
  measure of color frequency.
• Helps solve previous problem, although not
  entirely
• To calculate
• For each pixel of each frame of each tile in
  movie
• Convert from RGB to HSB
• Take hue value and histogram
• Create histograms on a per-tile basis
• Number of histogram buckets is changeable
• Usually use around 10.

8
Video Comparison Metric 3 Edge Histograms

• Also would like to have some kind of edge
  comparison between original and replacement tile.
• To calculate
• Foreach frame of each tile
• Convert frame to black and white
• Dont care about colorwant edge data
• Resize the image down to 10 x 10
• We want major edges only
• Use a bicubic resize to blur the image slightly
• For each pixel in the new image,
• Calc intensity gradient in the x and y directions
• Take the arctangent and histogram.
• Do for each frame, create histogram.
• Example of this metric at work

9
Finding the Best Match

• Currently, a linear search through a large
  unordered flat file.
• Parse this into a simple row/column data
  structure, keep it in memory.
• Search not as bad as it sounds takes about a
  second with 259200 entries.
• Coming soon some tricks to make this much
  better.
• For each tile of the source movie, compare with
  each tile entry in the database.
• Get the absolute difference between average
  colors
• Get the distance between color histograms
• Get the distance between edge histograms
• Add difference values to get a value for how good
  a match this tile is for the original.
• Can weight different metric to emphasize
  different features
• Choose entry that minimizes this.

10
Video vs. Photo Mosaics

• Why bother using video?
• Could mosaic each frame individually, and then
  string the frames together.
• Example. . .
• Would like to exhibit some kind of temporal
  consistency
• I.e. each tile is recognizable as a movie instead
  of a compression artifact.
• Neat effect less distracting for the eye.

11
Preliminary Results
12
Preliminary Results
13
Whats Next?

• Need to implement some way of dividing tile clips
  into arbitrary length.
• Approximation subdivide tiles into groups of
  frames.
• Index the rest of the video so I can leverage the
  full library.
• Still working on good ways to preserve temporal
  locality
• i.e. place video tiles in a manner that makes
  sense over time in addition to just per frame
• Include metrics that look at neighboring tiles to
  avoid drastically different edges
• These just end up looking like compression
  artifacts
• Princetons video has this problem
• Different sized tiles
• I think a patchwork effect would look
  interesting, albeit harder to compute.
• Optimize, optimize, optimize!
• Takes forever to make one of these!

14
References

• Klien, Grant, Finkelstein, and Cohen. Video
  Mosaics Proceedings of NPAR 2002.
• Microsoft Video For Windows developers site
• Apple Quicktime SDK developer resources
• James OBrien, Alexander Berg, Charless Fowlkes
• Questions? Comments?