Video Object Tracking and Replacement for Post TV Production

1
Video Object Tracking and Replacement for Post TV
Production

• LYU0303 Final Year Project
• Fall 2003

2
Outline

• Project Introduction
• Basic parts of the purposed system
• Working principles of individual parts
• Future Work
• QA

3
Introduction

• Post-TV production software changes the content
  of the original video clips.
• Extensively used in video-making industries.
• Why changing the content of a video?
• Reducing video production cost
• Performing dangerous actions
• Producing effects those are impossible in reality

4
Difficulties to be overcome

• Things in video can be treated individually
  called video objects.
• Computers cannot perform object detection
  directly because
• Image is processed byte-by-byte
• Without prior knowledge about the video objects
  to be detected
• Result is definite, no fuzzy logic.
• Though computers cannot perform object detection
  directly, it can be programmed to work indirectly.

5
Basic parts of the purposed system

• Simple bitmap reader/writer
• RGB/HSV converter
• Edge detector with smoother
• Edge equation finder
• Equation processor
• Translation detector
• Texture mapper

6
RGB/HSV converter

• Human eyes are more sensitive to the brightness
  rather than the true color components of an
  object.
• More reasonable to convert the representation of
  colors into HSV (Hue, Saturation and Value
  (brightness)) model.
• After processing, convert back to RGB and save to
  disk.

7
RGB/HSV converter

• HSV to RGB

• RGB to HSV

8
Basic parts of the purposed system

• Simple bitmap reader/writer
• RGB/HSV converter
• Edge detector with smoother
• Edge equation finder
• Equation processor
• Translation detector
• Texture mapper

9
Edge detector

• Usually, a sharp change in hue, saturation or
  brightness means that there exist a boundary line.

HSV (0,0,0)
HSV (0,255,255)
10
Edge detector
Before edge highlighting
After edge highlighting
11
Smoother for Edge detector

• Sometimes noise will affect the edge detection
  result of low resolution images.
• Include an image smoother to remove large noise
  points in the image.
• In some cases performing a smoothing will greatly
  enhance the performance of edge detection due to
  the decrease in fake edge points.

12
Smoother for Edge detector
13
Basic parts of the purposed system

• Simple bitmap reader/writer
• RGB/HSV converter
• Edge detector with smoother
• Edge equation finder
• Equation processor
• Translation detector
• Texture mapper

14
Edge equation finder

• Derives mathematical facts out of the edge
  points.
• Works with voting algorithm of Hough Transform.
• Automatically adjusts tolerance value to minimize
  the effect of noise points.
• This helps when the edge is not completely
  straight or blurred.

15
Edge equation finder
Angle in degree Frequency
0 1
45 3
90 1
135 1
(x1,y1)
Desired linear equation in point-slope form
16
Edge equation finder
17
Basic parts of the purposed system

• Simple bitmap reader/writer
• RGB/HSV converter
• Edge detector with smoother
• Edge equation finder
• Equation processor
• Translation detector
• Texture mapper

18
Equation processor

• After finding out the equation, constraints can
  be applied in order to remove redundant
  equations, get shadows or detect occultation.
• Find out the corner points for translation
  detector and texture mapper.

19
Equation processor
Before edge finding
After edge and equation finding
After extra equation removal
20
Equation processor

• The following criteria are currently adopted in
  the equation processor
• Distance between the equations
• Angle between the equations
• Whether the equation intersects the object or
  not.
• Since equation processor is a potential burden to
  the system, it may be replaced in the future by
  improving other parts.

21
Basic parts of the purposed system

• Simple bitmap reader/writer
• RGB/HSV converter
• Edge detector with smoother
• Edge equation finder
• Equation processor
• Translation detector
• Texture mapper

22
Translation detector

• A simple object motion tracker.
• Collects the data of the first key frame to
  accelerate the process of the remaining video
  frames.
• Can be beneficial if the video segment is long
  and the scene seldom changes.

23
Translation detector

• Records the approximate location of the object in
  the key frame.
• When processing the following video frames, just
  scan within a certain boundary near the recorded
  location.
• Will improve its efficiency later.

24
Basic parts of the purposed system

• Simple bitmap reader/writer
• RGB/HSV converter
• Edge detector
• Edge equation finder
• Equation processor
• Texture mapper

25
Texture Mapper

• A graphics design technique used to wrap a
  surface of a 3-D object with a texture map
• The 3-D object acquires a surface texture similar
  to the texture map.
• Colors, brightness values or altitudes

26
Texture Mapper
Mapping (color)
Texture map
Image
27
Texture Mapper

• Definition of terms
• Image coordinates (r, c) location of pixel in
  the image
• Texture coordinates (u, v) location in texture
  map which contains color information for image
  coordinates
• Mapping function determines how texture
  coordinates are mapped to image coordinates or
  vice versa.
  e.g. linear scan-line interpolation

28
Texture Mapper
Image coordinates
Texture coordinates
(r,c)
(u,v)
Mapping function
29
Texture Mapper

• Definition of terms
• Forward mapping maps from the texture space to
  image space
• Inverse mapping maps from the image space to
  texture space
• Scan-line conversion an area-filling technique
  processing a surface line by line. It can be
  applied with forward/inverse mapping.

30
Texture Mapper
Forward mapping
Inverse mapping
31
Texture Mapper
Scan-line conversion
More important
Texture scanning
Image scanning
32
Scan-line conversion
Scanline yk
scanning order
Scanline yk1

• triangle/parallelogram scanning
• line by line, from top to bottom
• process each pixel on every line

for every line
33
Scan-line conversion
Coordinates differ by ?x and ?y
Scanline yk
Scanline yk1

• For a row scan, maintain a list of scanline /
  polygon intersections.
• Intersection at scanline r1 efficiently computed
  from scanline r.

34
Scan-line conversion
1
2
3

• quadrilateral ? triangles or parallelograms
• scan each sub-polygon
• special case only 2 sub-polygons

35
Scan-line conversion with forward mapping

• Algorithm

for u umin to umax for v vmin to vmix
texture scanning
forward mapping functions
r R(u,v) c C(u,v)

• copy pixel at source (u,v)
• to destination (r,c)

36
Scan-line conversion with inverse mapping
Algorithm
texture scanning
for (r,c) polygon pixel
u U(r,c) v V(r,c)
inverse mapping functions

• copy pixel at source (u,v)
• to destination (r,c)

37
Comparison
Inverse mapping
Forward mapping
image ? texture
texture ? image
Principle
a bit complicated as it involves image scanning
easy if the mapping function is known
Ease of implementation
No
Yes
Calculation of fractional area of pixel coverage

Possibility of aliasing
38
Comparison
Aliasing filtering/resampling techniques can be
applied for inverse mapping
39
Comparison
Inverse mapping
Forward mapping
image ? texture
texture ? image
Principle
a bit complicated as it involves image scanning
easy if the mapping function is known
Ease of implementation
No
Yes
Calculation of fractional area of pixel coverage
Yes, but can be avoided with simple
filtering/resampling
Yes
Possibility of aliasing
We used inverse mapping
40
Mapping functions

• Simple linear transformations
• Translation, scaling, etc
• ? parallelograms only
• very fast
• Linear scan-line interpolation
• Based on proportion
• ? any quadrilaterals

not suitable
feasible
41
Linear scan-line interpolation

• Idea of interpolation

(x2, y2)
1-a
(x1, y1)
(x, y) a(x2, y2) (1-a)(x1, y1)
a
42
Linear scan-line interpolation
For a particular scanline,
(u4, v4)
(u1, v1)
(u3, v3)
(r1, c1)
(r, c)
(u, v)
(u5, v5)
texture map
(r4, c4)
(r5, c5)
image
(u2, v2)
(r3, c3)
(r2, c2)
43
After mapping
a bit strange !
44
Shadow mapping

• Mapping of surface brightness
• Retain the brightness of the original surface
• Method
• Convert the original surface ? HSV
• Get the V value
• Replace the V value of the mapped surface

45
After Shadow mapping
More natural !
46
Future Work

• Anti-aliasing
• Mapping different shapes like cans
• Speed optimization
• Movie manipulation
• Use of 3D markers

47
Q A