High Frame Rate Up Conversion

1
High Frame Rate Up Conversion

• Ana Bertran

2
Problem Statement

• Original Frame
• (30 fps)

Dwnsmpld Frame (5 fps)
Recovered Frame (30 fps)
S H
LI
MCLI
MCwA
Hypothesis if we have acceleration MCwA should
provide better results
Xabsin(at), vxbacos(at),
ax-(ba)2sin(at)
3
Motivation

• Uses of frame rate up conversion
• Converting between standards (PAL to NTSC). Using
  MCwA less critical since frame rates not too far
  from each other.
• Low bit rate compression for video-confrncng,
  video-phone and video games. MCwA becomes more
  critical (from 10 fps to 30 fps)

• Problems with traditional methods
• SH motion looks jerky, not smooth, very choppy
• Linear interpolation without motion image looks
  blurry where motion has occurred, we will see
  ghosts due to avg. btwn. frames.

4
Motivation
Original
Linear Interpolation
Sample Hold
5
MCLI vs. MCwA
LMC
ME
Catalog Occsns
AMC
6
ME

• For forward and backward MVs

• Top Left block use its MV
• 1st row blocks min SAD(L, current or next
  frame)
• 1st colmn blocks min SAD(T, current or next
  frame)
• others min SAD(T, L, current or next frame)

Finer MV Selection
Overlapping BM
16x16 blcks, move by 8
Take out illegal MVs
½ pixel MV zoom in
7
Cataloging Occlusions
For covered pixels we need to use frame 2 as the
predicted image, one whose blocks we serch
for. For uncovered pixels we need to use frame 3
as the predicted image, one whose blocks we
serch for.
Need to track block from 3 to 2 Criteria
compare SADs depending which one minimum classify.
1 2 3 4
8
Cataloging Occlusions Results
blueuncov redcov Greenno occsn
blueuncov redno occsn
9
LMC
For each block along motion trajectory MVx_estw
MVx_3_2 MVy_estwMVy_3_2 Motion will be
non-integer interpolate
No occlusions
Uncovered pixels
Assumed motion in x Depending on ve/-ve x mtn,
unvov/cov
3 cases
Covered pixels
3
2
No occlusions
Occlusions
10
AMC
To solve distancexvxt0.5axt2 you need two
MVs but you can get more accurate motion
trajectories by solving a LMS problem on 3
MVs. For each MV between two frames take the
previous and next MVs to estimate the
trajectory. Need to solve
11
Frame rate up conversion a difficult problem

• Need to use true motion vector fields but block
  matching does poorly with this, specially across
  object boundaries and if the moving object is too
  small. Cant cope with discontinuities in the
  velocity plane.
• Some motion can fall in between pixels
• Dealing with occlusions
• Motion can change dramatically between frames and
  we wont realize it. (Critical vels)
• Not to talk about scene changes! we no longer
  will have enough MVs to track motion trajectory.

12
Results/Demo

• ?
• Will have the demo in the website by the end of
  the week.
• For now you can see the demo for the other types
  of FR up-conversion

13
Current temporary results
14
Future Work

• Need better MV estimation of the true MVs
• Implement ½ pixel resoltn or
• Use hierarchical block matching to find true
  motion MVs or
• Object based interpretation of the video to
  smooth out MVs
• Use 3D Recursive Search Block Matching
• Solve occlusion problem for both x and y movement

15
References

• Using Motion-Compensated Frame-Rate Conversion
  for the Correction of 32 Pulldown Artifacts in
  lman, Video Sequences, Kevin Hilman, Hyuon Wook
  Park, and Yongmin Kim
• True-Motion Estimation with 3-D Recursive Search
  Block MatchingGerard de Haan, Paul W.A.C.
  Biezen, Henk Huijgen and Olukayode A. Ojo
• Digital Video Standards Conversion in the
  presence of accelerated motionAmdrew J. Patti,
  M. Ibrahim Sezan and M.Murat Tekalp
• Framee Rate Up- Conversion using transmitted true
  motion vectorsYen-Kuang Chen, Anthony Vetro,
  Huifang Sun, and S.Y. Kung