Background vs. foreground segmentation of video sequences

1
Background vs. foreground segmentation of video
sequences


2
The Problem

• Separate video into two layers
• stationary background
• moving foreground
• Sequence is very noisy reference image
  (background) is not given

3
Simple approach (1)
background
temporal mean
temporal median
4
Simple approach (2)
5
Simple approach noise can spoil everything
6
Variational approach
Find the background and foregroundsimultaneously
by minimizing energy functional
Bonus remove noise
7
Notations
given
need to find
C(x,t) background mask(1 on background, 0 on
foreground)
N(x,t) original noisy sequence
B(x) background image
8
Energy functional data term
B
N
B - N
C
9
Energy functional data term

• Degeneracy can be trivially minimized by
• C ? 0 (everything is foreground)
• B ? N (take original image as background)

10
Energy functional data term
C
1
11
Energy functional data term
original images should be close to the restored
background image in the background areas
there should be enough of background
12
Energy functional smoothness
For background image B
For background mask C
13
Energy functional
14
Edge-preserving smoothnessRegularization term
Quadratic regularization Tikhonov, Arsenin 1977
ELE
Known to produce very strong isotropic smoothing
15
Edge-preserving smoothnessRegularization term
Change regularization
ELE
16
Edge-preserving smoothnessRegularization term
ELE
17
Edge-preserving smoothnessRegularization term
Change the coordinate system
ELE
across the edge
along the edge
Compare
18
Edge-preserving smoothnessRegularization term
Conditions on ?
Weak edge (s ? 0)
?(s)
Isotropic smoothing ?(s) is quadratic at zero
s
19
Edge-preserving smoothnessRegularization term
Conditions on ?
Strong edge (s ? ?)

• no smoothing across the edge

• more smoothing along the edge

?(s)
Anisotropic smoothing ?(s) does not grow too fast
at infinity
s
20
Edge-preserving smoothnessRegularization term
Conclusion
Using regularization term of the form
we can achieve both isotropic smoothness
in uniform regions and anisotropic
smoothness on edges with one function ?
21
Edge-preserving smoothnessRegularization term
Example of an edge-preserving function
22
Edge-preserving smoothnessSpace of Bounded
Variations
Even if we have an edge-preserving functional
if the space of solutions u contains only
smooth functions, we may not achieve the desired
minimum
23
Edge-preserving smoothnessSpace of Bounded
Variations
which one is better?
24
Bounded Variation ND case
bounded open subset, function
Variation of over
f
where
25
Edge-preserving smoothnessSpace of Bounded
Variations
integrable (absolute value) and with bounded
variation
Functions are not required to have an integrable
derivative
What is the meaning of ?u in the regularization
term?
Intuitively norm of gradient ?u is replaced
with variation Du
26
Total variation
Theorem (informally) if u? BV(?) then
27
Hausdorff measure
area 0
area gt 0
How can we measure zero-measure sets?
28
Hausdorff measure
1) cover with balls of diameter ? ?
2) sum up diameters for optimal cover (do not
waste balls)
3) refine ? ? 0
29
Hausdorff measure
Formally
For A? RN k-dimensional Hausdorff measure of A
up to normalization factor covers are countable

• HN is just the Lebesgue measure

• curve in image its length H1 in R2

30
Total variation
Theorem (more formally) if u? BV(?) then
u(x)
u
u-
x
x0
u, u- - approximate upper and lower limits
Su x?? ugtu- the jump set
31
Energy functional
data term
regularization for background image
regularization for background masks
32
Total variation example
perimeter 4
Divide each side into n parts
33
Edge-preserving smoothnessSpace of Bounded
Variations
Small total variation( sum of perimeters)
Large total variation ( sum of perimeters)
34
Edge-preserving smoothnessSpace of Bounded
Variations
Small total variation
Large total variation
35
Edge-preserving smoothnessSpace of Bounded
Variations
BV informally functions with discontinuities on
curves
Edges are preserved, texture is not preserved
energy minimization in BV
original sequence
temporal median
36
Energy functional
Time-discretized problem
Find minimum of E subject to
37
Existence of solution
Under usual assumptions
?1,2 R? R strictly convex, nondecreasing,
with linear growth at infinity
minimum of E exists in BV(B,C1,,CT)
38
(non-)Uniqueness
is not convex w.r.t. (B,C1,,CT) ! Solution may
not be unique.
39
Uniqueness
But if ?c ? 3?range2(Nt , t1,,T, x? ?), then
the functional is strictly convex, and solution
is unique.
Interpretation if we are allowed to say that
everything is foreground, background image is not
well-defined
40
Finding solution
BV is a difficult space you cannot write
Euler-Lagrange equations, cannot work numerically
with function in BV.

• Strategy
• construct approximating functionals admitting
  solution in a more regular space
• solve minimization problem for these functionals
• find solution as limit of the approximate
  solutions

41
Approximating functionals
Recall ?1,2(s) s2 gives smooth solutions
Idea replace ?i with ?i,? which are quadratic
at s ? 0 and s ? ?
42
Approximating functionals
43
Approximating problems
has unique solution in the space

• convergence of functionals if E?
  ?-converge to E then approximate solutions of
  min E?
• converge to min E

44
More results Sweden
45
More results Highway
46
More results INRIA_1
47
More results INRIA_1Sequence restoration
48
More results INRIA_2Sequence restoration
49
Thank You!