Processing Images and Video for An Impressionist Effect

1
Processing Images and Video for An Impressionist
Effect

• Automatic production of painterly animations
  from video clips.
• Extending existing algorithms for producing
  painterly pictures to create temporally
  coherent animations.
• Reduce user interaction to a bare minimum.

2
Methodology Overview

• Brush strokes are clipped to edges detected in
  the original image sequence and are oriented
  normal to the gradient direction of the original
  image.
• Scattered data interpolation is used when
  gradient is near zero.
• Brush stroke list is maintained and manipulated
  through the use of optical flow fields to enhance
  temporal coherence.

3
Rendering Strokes

• Rendered with an anti-aliased line centered at a
  point, with a given length, radius, and
  orientation.
• Color is determined through bilinear
  interpolation of the color at the point in the
  original image.
• Strokes are spaced apart by a user specified
  distance.
• Strokes are drawn using a random ordering to help
  achieve a hand-drawn look.

4
Randomizing Strokes

• Random amounts are assigned to the length and
  radius of the stroke, within a user specified
  range.
• Colors are perturbed by a random amount usually
  between -15,15 for r,g,b. Colors are then
  scaled by an intensity usually within .85,1.15
  and then clamped to 0,255.
• The orientation angle is randomized by adding a
  random angle amount between -15º,15º.
• All these randomizations are then stored in a
  data structure for each stroke and are not
  regenerated on a frame to frame basis.

5
Clipping and Rendering

• An intensity image is created from the original
  color image using the equation
• (30red 59green 11b)/100
• The intensity image is blurred using a Gaussian
  kernel, or a B-spline approximation to the
  Gaussian.
• The resulting blurred image is Sobel filtered
  where the value of the Sobel filter is
• Sobel(x,y) Magnitude (Gx, Gy)

6
Sobel Filtering

• The Sobel filter consists of two kernels which
  detect horizontal and vertical changes in an
  image.
• The magnitude of an edge
• MSobelxy Ghxy Gvxy
• The direction of an edge
• fSobelxy tan-1(Gvxy / Ghxy)

7
Clipping and Rendering Cont

• Start the line at the center point and grow the
  line in its orientation direction until either an
  edge is reached or maximum length obtained. An
  edge is reached if the Sobel value decreases in
  direction of orientation that the line is
  growing.
• Once we have determined the endpoints, the stroke
  is drawn using the perturbations stored in our
  stroke data structure. A linear falloff is used
  in a 1.0 pixel radius region ( alpha transition
  from 1 to 0 ).

8
Brush Stroke Orientation

• Strokes are drawn with a constant color in the
  gradient-normal direction.
• Brush strokes in a region of constant color
  smoothly interpolate the directions defined at
  the regions boundaries. A thin-plate spline is
  used for the interpolation because it cannot be
  assumed that the data lies on a uniformly spaced
  grid.
• The gradient at the center point is bilinearly
  interpolated and the direction angle is computed
  using
• Arctan(Gy/Gx) 90º ( normal to the gradient ).
• The perturbations are added and the result is
  stored as this strokes direction angle.

9
Animating the Images

• Since there is no a priori information about
  pixel movement, vision techniques are applied to
  guide the brush strokes.
• Optical flow vector fields are utilized to
  determine pixel movement.
• Brush stroke centers are mapped to subpixel
  locations in the new image.

10
Handling Sparse Regions

• After using the vector flow field there may be
  regions which are too sparsely populated.
• Sparse regions dont provide the coverage we wish
  to have. Use Delaunay triangulation to find
  triangles with area greater than maximum
  specified. Add pixels to reduce the area of
  these triangles.
• The Delaunay triangulation has the property that
  the circumcircle (circumsphere) of every triangle
  (tetrahedron) does not contain any points of the
  triangulation.

11
Delaunay Triangulation
12
Handling Dense Regions

• Dense regions will greatly slow down the
  rendering process. The edge list of the
  triangulation is traversed. If the distance
  between points is less than a user-specified
  distance, the stroke that is drawn closer to the
  back is removed.
• Display list of strokes determines stroke order.

13
Potential Problems

• The method of optical flow estimation employed by
  Litwinowicz assumes that lighting is constant and
  that occlusion may be ignored, resulting in
  objects appearing and disappearing.
• Stroke movement is only as good as the motion
  estimate technique.
• Noise in the images will cause noise in the
  animations.

14
Possible Enhancements and Future Work

• Scale the offset of the stroke based on the
  length of the stroke.
• Implement additional rendering styles to produce
  other artistic styles. This would involve
  modifying the stroke drawing algorithm to emulate
  the different styles.
• Apply this technique to 3D models where the
  motion of the objects in the scene would be known
  a priori. This could be accomplished by removing
  most of the vision algorithms from the system and
  replacing them with data structures that would
  hold all the necessary information for all the
  models of the items in the animation.