Visual Differences

1
Visual Differences
Take two successive images, convert to gray
scale, subtract the images. The result is now
bipolar. Use red to display positive differences
and blue to display negative differences. Extend
the color range by normalizing to the maximum
difference. Show the result as a color image
2
bipolar_image
function rgb bipolar_image(im) display
bipolar differences as blue -gt red scale r
ones(size(im)) g r b r idx
find(imgt0) g(idx) 1 - im(idx) b(idx)
g(idx) idx find(imlt0) g(idx) 1
im(idx) r(idx) g(idx) rgb
cat(3,r,g,b) imshow(rgb)
3
Movie of Frame Differences
4
Script generate the movie
nframes length(seq) mov avifile('test.avi','f
ps',5) im1 im2double(rgb2gray(seq(1).cdata)) f
or n2nframes im2 im2double(rgb2gray(seq(n).cd
ata)) rgb bipolar_image(im2-im1) im1
im2 fr im2double(rgb/max(rgb())) mov
addframe(mov,fr) end mov close(mov)
5
Mobile Sequence
s(x,y)
sx(x,y)
sy(x,y)
st(x,y)
6
Result of OptFlow
Elapsed time is 103.172000 seconds.
7
Result of HSOptFlow
Elapsed time is 0.312000 seconds.
8
Test Cases
I tried two test cases. The first had a simple
ramp in the horizontal direction that moves
horizontally. The block search algorithm fails
because of the aperture effect. There is no
unique minimum. The second uses a quadratic
circular pattern that moves two pixels to the
left.
These are contour plots of the two images
superimposed on one plot.
blue before red after
9
Matlab Script for Test Case
x linspace(-0.6,0.6,80) im
ones(size(x))'x im1 im.2(im').2 rgb
cat(3,im1,im1,im1) seq(1)im2frame(rgb) dx
x(2)-x(1) im2 (im2dx).2 (im').2 rgb
cat(3,im2,im2,im2) seq(2)im2frame(rgb) clear
rgb
10
OptFlow on Test Case
gtgt us vs OptFlow(seq,1) Elapsed time is
6.235000 seconds.
11
OptFlow on Test Case
gtgt us vs OptFlow(seq,1) Elapsed time is
6.235000 seconds. gtgt idx 812 gtgt
vs(idx,idx) ans -2.0000 -2.0000
-2.0000 -2.0625 -2.0000 -2.0000 -2.0000
-2.0000 -2.0000 -2.0000 -2.0000 -2.0000
-1.9375 -2.1250 -2.0000 -2.0000
-2.0000 -2.0625 -2.0000 -2.0000 -2.0000
-2.0000 -2.0000 -2.0000 -2.0000 gtgt
us(idx,idx) ans 0 0
0 0 0 0 0
0 0 0 0 0
-0.0625 -0.0625 0 -0.0625 0
-0.1250 -0.1250 0 0 0
0 0 0
12
Modified Block Search
I modified the block search routine to work like
Matlab image processing toolbox blkproc
operations.
gtgt vx vy blocks(im1,im2,5 5,5 5) Elapsed
time is 0.469000 seconds.
13
New Block Processer
gtgt idx 812 gtgt vx(idx,idx) ans -2
-2 -2 -2 -2 -2 -2 -2 -2
-2 -2 -2 -2 -2 -2 -2 -2
-2 -2 -2 -2 -2 -2 -2 -2 gtgt
vy(idx,idx) ans 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0
14
HSOptFlow on Test Case
The result is strange until you look at the
contour and realize that HS really is calculating
localized normal flow.
gtgt us vs HSOptFlow(seq,1) dsx1 max 0.0176471
dsx2 max 0.0156863 Elapsed time is 0.203000
seconds.
15
Results of Lucas-Kanade Method
gtgt vx vy LKOptFlow(im1,im2,5 5,0
0) Elapsed time is 0.016000 seconds.
16
LK Method Numerical Results
gtgt idx812 gtgt vx(idx,idx) ans -2.0000
-2.0000 -2.0000 -2.0000 -2.0000 -2.0000
-2.0000 -2.0000 -2.0000 -2.0000 -2.0000
-2.0000 -2.0000 -2.0000 -2.0000
-2.0000 -2.0000 -2.0000 -2.0000 -2.0000
-2.0000 -2.0000 -2.0000 -2.0000
-2.0000 gtgt vy(idx,idx) ans 1.0e-013
-0.0082 -0.0208 0.0729 0.1497 0.0504
0.0095 0.0339 -0.0919 -0.1461 -0.0994
0.0038 -0.0099 -0.1725 0
-0.1731 0.0034 0.0142 -0.0946 -0.4395
-0.2779 -0.0025 0.0054 -0.2432 0
0.2152
17
Block Search (mobile Frames 1,2)
gtgt vx vy blocks(im1,im2,5 5,5 5) Elapsed
time is 5.703000 seconds.
18
Average Motion (Block Search)
0.0006 0.0009 0.0023 0.0018
0.0006 0.0009 0.0035 0.0079 0.0035
0.0021 0.0041 0.0106 0.1268 0.6954
0.0176 0.0041 0.0029 0.0091 0.0211
0.0023 0.0006 0.0012 0.0035 0.0035
0.0012
210-1-2
-2 -1 0 1
2
70 of the image shifted 1 pixel right
19
HSOptFlow (revised)
mobile frames 1 2
Algorithm revised from handout changed alpha2
and iteration count.
20
Distribution of Velocities
Velocity not tightly clustered. The average flow
is to the right less than 0.5 pixel
vxc vx() vyc vy() plot(vxc,vyc,'.')
21
Lucas-Kanade
mobile frames 1 2
22
LK Velocity Map
23
Time Difference after Registration
gtgt diff im2(,2352)-im1(,1351) gtgt dmax
max(abs(diff())) gtgt rgb bipolar_image(diff/dma
x)
24
LK Pan (upper-left quadrant)
mobile frames 1 2
gtgt mean(mean(vx(120,120))) ans 1.1088 gtgt
mean(mean(vy(120,120))) ans 0.1153
25
Register Images (fractional shift)
dx 1.11 dy.11 T maketform('affine',1 0 0
0 1 0 dx dy 1) nr nc size(im1) out
imtransform(im1,T,'XData',1 nc,'YData',1
nr) diff im2 - out rgb bipolar_image(diff)
26
mobile Frames 1 and 5
gtgt vx vy blocks(im1,im2,5 5,5 5) Elapsed
time is 5.781000 seconds. gtgt mean(mean(vx(510,51
0))) ans 4.9722 gtgt mean(mean(vy(510,510)))
ans 0.9722
27
Register Frames
no pan
pan (5,0)
Simple pan is not registering the static
background completely.
pan (5,1)
28
Pan Registration Script
dx 5 dy1 T maketform('affine',1 0 0 0 1
0 dx dy 1) nr nc size(im1) out,xdata,ydat
a imtransform(im1,T,'XData',1 nc,'YData',1
nr) diff im2 - out rgb bipolar_image(diff)
29
Color Velocity
Encode hue as direction of motion and saturation
as magnitude of motion. Let value 1.
function rgb color_velocity(vx,vy) sat
sqrt(vx.2vy.2) value ones(size(sat)) hue
angle(vxjvy)/pi hue hue() idx
find(huelt0) hue(idx) 2hue(idx) hue
hue/2 hue reshape(hue,size(sat)) rgb
hsv2rgb(hue,sat,value) imshow(rgb)
30
Mobile Frames 1 2
color velocity map
Frame 2
31
Pan Compensation
before
after
32
Matlab Script
load mobile_short im1 rgb2gray(im2double(seq(1).
cdata)) im2 rgb2gray(im2double(seq(2).cdata))
vx vy LKOptFlow(im1,im2,5 5,0 0) rgb
color_velocity(vx,vy) imwrite(rgb,'color_velocity
1.jpg') script3 pan compensation script vx2
vy2 LKOptFlow(out,im2,5 5,0 0) rgb2
color_velocity(vx2,vy2) imwrite(rgb2,'color_veloc
ity2.jpg')