Visual Artifact Reduction By PostProcessing

1
Visual Artifact Reduction By Post-Processing
Presented at Carnegie Mellon University Dept.
ECE, November 1, 2001

• Yu Hen Hu (hu_at_engr.wisc.edu)
• University of Wisconsin -- Madison
• Dept. Electrical and Computer Engineering
• in collaboration with Dr. Seungjoon Yang

2
Outline

• Visual artifacts in digital image and video
• Post-processing Method
• Enhancement
• Restoration
• Blocking Artifact Reduction using GenLOT-embedded
  Inverse DCT
• Ringing Artifact Reduction Using Robust ML filters

3
Visual Artifacts in Digital Visual Materials

• Digital Images and Videos
• Obtained by digitizing analog visual materials or
  direct capture.
• Visual quality remain unchanged after copying.
• Visual Artifacts in digital images and videos
• Artifacts due to inadequate acquisition
• Out of focus, smearing, dust, scratch, blotches
• Artifacts due to inadequate processing
• Lossy compression coding artifact
• Watermarking
• Artifacts due to transmission error
• Lost blocks, frames, discoloring

4
Types of Visual Artifacts

• Spatial artifacts
• Content independent
• Blotches
• Scratches
• Additive noise
• Content dependent
• Blocking
• Ringing
• Smearing

• Temporal artifacts
• Frame jittering
• Successive image frames at slightly different
  spatial positions
• Line jittering
• Individual raster line mis-aligned, causing
  vertical straight line jitters
• Rapid, uneven camera motion
• Uneven panning, tracking, etc.
• Discontinued motion of objects
• Due to frame skipping

5
Coding Artifacts

• Depending on the type of the transform

(a) Blocking Artifact
(b) Ringing Artifact
Non-Overlapping Transforms
Overlapping Transforms
6
Visual post-processing

• Image and video processing procedure applied
  after normal processing (including compression,
  decompression), prior to final presentation.
• Purpose to enhance visual quality of images and
  videos by removing unsightly visual artifacts.
• Artifact reduction is accomplished in two phases
• Artifact detection
• Artifact removal

7
Artifact Detection

• A statistical signal detection problem.
• Exploit unique feature of different types of
  artifacts
• Difficult for content dependent artifacts
• Temporal artifact detection must be performed by
  examining consecutive frames (spatial-temporal
  analysis)
• Must exploit human visual system (HVS)
  characteristics
• Visual masking effect

8
Artifact Reduction

• Two steps
• Visual quality enhancement reducing unsightly
  artifact
• Restoring original content by reversing the
  degradation process
• Need to estimate lost information
• Artifact may cover original contents e.g.
  blotches
• Artifact may be due to lost of original content
  e.g. coding artifact

9
Transformed Image Coding Artifact
ReductionProblem Statements
10
Transformed Image Coding
f
F
Fq
Transform
Quantizatoin
Lossless Coding
Bit Allocation
(a) Encoder
Fq
g
Lossless Decoding
Dequantization
Inverse Transform
(b) Decoder
11
Coding Artifacts

• Blocking artifact
• Cause
• Lossy quantization of frequency coefficients
• Symptom
• Spurious edges at block boundary (known position)
• Issues
• preserving true edges that cross or overlap with
  block boundary
• Common approaches
• spatial domain filtering
• DCT coefficients adjustment

• Ringing Artifact
• Cause
• Lossy quantization of frequency coefficients
• Symptom
• Spurious edges along major edges
• Issues
• difficult to separate ringing artifact with true
  edge of texture
• Common approaches
• Restoration while preserving major edges
• Medium filtering

12
Blocking Artifact

• The edge map is enhanced to show spurious edges
  along block boundaries.
• Visual masking effect Blocking artifact is more
  visible at relatively flat region of an image.

13
Blocking Artifact

• With block-based transform coding, pixels
  acrossing the block boundary are encoded with
  different set of basis functions.
• Reconstructed Block (g pixel value, Fq freq.
  Coef., ? basis)
• Discontinuity at Block Boundary
• where
• If the basis function overlaps between adjacent
  blocks, the blocking effect may be alleviated.

14
Ringing Artifact

• Ringing Artifact Oscillation (spurious edges) at
  the Vicinity of major (high contrast, large
  scale) edges
• Visual masking effect prominent in areas with
  relatively smooth background

Ringing artifact Enhanced edge map
15
Cause of Ringing Artifact

• Gibbs phenomenon -- Truncation of frequency
  domain coefficients corresponds to convolving the
  time domain sequence with a sinc function. The
  side lobes manifest themselves as oscillations in
  spatial domain around step discontinuities.
• The cause of the ringing artifact is similar to
  that the of Gibbs phenomenon when long basis
  functions are cut short due to heavy lossy
  quantization

Truncation of DCT coefficients from 128 down to
32 leads to ringing artifact Original is a step
function. With only 32 low frequency coefficients
left, ringing occurs.
16
New Approaches

• Blocking Artifact Reduction using
  GenLOT-embedded IDCT
• Embedding DCT/IDCT in a more general lapped
  orthogonal filter bank transformation.
• Modifying the GenLOT coefficients to reduce
  blocking artifact.
• Performed only at decoding end without altering
  encoding process.

• Maximum Likelihood estimation of ringing
  artifact free image
• Use a flat-surface image model to distinguish
  true edges from ringing edges that have smaller
  magnitudes.
• Use ML method to estimate true image value within
  a sliding window.
• Implemented as a data-adaptive, nonlinear robust
  filter

17
Blocking Artifact Reduction using Embedded IDCT
S. Yang, S. Kittitornkun, Y. H. Hu, T. Q. Nguyen,
and D. L. Tull, Generalized Lapped Biorthogonal
Transform embedded inverse discrete cosine
transform, IEEE Trans. Image Processing, pp.
submitted, 2000.
18
Existing Blocking Artifact Reduction Methods

• MAP based restoration Approach ORourke
  Stevenson, 95
• POCS (Projection onto convex set)
• Constraints can be imposed to ensure the
  quantized bit stream of smoothed image is close
  to that of decoded image. E.g. POCS Yang,
  Galatsanos, Katsaggelos, 95
• Location-specific smoothing (filtering)
• Replacing step edges along block boundary by
  smoothed pixel values. Nonlinear smoothing may be
  used (H.263, annex J)
• Can be applied in spatial as well as frequency
  domain
• Optimization Approach Minami Zakhor, 95

19
MAP Estimation

• Given g, find f such that

Feasible Image Set
Prior Knowledge on Estimate
Improved Image Decompression for Reduced
Transform Coding Artifacts, T.P. O'Rourke and
R.L. Stevenson
20
MAP Estimation

• Gibbs Image Prior Distribution
• With Gibbs Image Prior Distribution

c clique
? potential function
21
MAP Estimation with Line Process

• Take away the option for discontinuity at block
  boundaries.

?
Line Process
At block boundaries
22
POCS

• Find f such that
• Actual Implementation

Projection-Based Spatially Adaptive
Reconstruction of Block-Transform Compressed
Images, Y. Yang and N.P. Galatsanos
and A.K. Katsaggelos
23
Nonlinear Filtering

• Deblocking option in H.263 Annex J

where d1,d2,clip() are designed for appropriate
smoothing
Annex J of H.263,
24
Blocking Artifact Reduction using Overlapped
Transformation

• Blocking artifact is due to block-based transform
  of images.
• At low bit-rate compression, many frequency
  coefficients are quantized to zero
• Overlapped Orthogonal Transform (LOT) compute
  frequency coefficients of the same block size
  using longer bases that overlap adjacent blocks.
  As a result, blocking effect is less prominent
  using LOT in place of DCT
• However, LOT is not standard-compliant!
• Our approach
• Realize blocking artifact reduction in overlapped
  transform coefficient domain
• Maintain standard compliance by embedding
  DCT/IDCT pair within an overlapped transformation.

25
Overlapped and Non-overlapped Transforms

• 1D linear transform projection of 1D signal
  onto basis function of the transform.
• Base on how the basis functions are interlaid,
  can be classified into overlapped and
  non-overlapped transform.
• Let

• Non-Overlapping Transforms (e.g. blocked DCT)

(b) Overlapping Transforms, e.g. LOT
26
Generalized Lapped Biorthogonal Transform (GLBT)
Transformed coefficients from adjacent blocks
T
G0(z)
GK-1(z)
GK-1-1
Forward Transform
Inverse Transform
The Generalized Lapped Biorthogonal Transform,
T.D. Tran and T.Q. Nguyen
27
DCT is Part of GLBT
DCT
GLBTs Frist Stage
T
28
GLBT Embedded IDCT (ge-IDCT)
Standard non-compliant
Forward Transform
Inverse Transform
Tdct-1
G
G-1
Tdct
Standard compliant
DCT
ge-IDCT
29
ge-IDCT based Transform

• ge-IDCT is standarad compliant!

Tdct
Tdct-1
G
G-1
Quantization
Same as standard DCT based Encoder
Ge-IDCT based Decoder
30
Blocking Artifact Reduction in Embedded Lapped
Transform Domain
Perform blocking artifact reduction in GLBT
coefficient domain.
Tdct
Tdct-1
Post-Processing
G-1
Quantization
G
Standard Encoder
Improved Decoder
31
Deblocking

• Blocking Artifact Discontinuity between 2 DCT
  blocks
• Empirically, odd-symm. frequency coefficients
  contribute more to blocking effects.

32
Frequency Weighting

• Solution
• reduce excessive energy in odd-sym. coef.s.
• Method
• After lapped transform G of DCT coefficients,
  apply weights on two lowest odd-symm.
  coefficients
• Features
• Selective Smoothing
• Preservation of Major Structure Robustness

33
New Inverse Transforms
(a) ge-IDCT
(b) le-IDCT
34
PSNR MSDS

• Peak Signal to Noise Ratio
• Mean Square Difference of Slope

An Optimization Approach for Removing Blocking
Effects in Transform Coding, S. Minami and A.
Zakhor
35
Evaluation PSNR MSDS
PSNR improvement
MSDS reduction
(a) Airplane
(b) Lena
(c) Peppers
36
Evaluation Subjective Quality
(a) JPEG
(b) ge-IDCT
At quality factor 15
37
Comparative Study

• Existing Methods
• Maximum a posteriori (MAP) Estimation (MAP)
• Projection onto Convex Set (POCS)
• Nonlinear Filtering (NF)
• We also implement
• MAP Estimation with Line Process (MAP-L)

38
Comparison PSNR MSDS
(a) PSNR
(b) MSDS
39
Comparison Less Over-smoothing
MAP
ge-IDCT
MAP-L
40
Comparison Less Under-smoothing
POCS
le-IDCT
NF
41
Comparison Simplicity

• Unitary transformation in ge-IDCT and le-IDCT can
  be implemented with Lattice Structures and
  CORDIC.

Iterative Algorithms
Non-Iterative Algorithms
MAP, MAP-L, POCS
NF, ge-IDCT, le-IDCT
42
Summary

• New Inverse Transforms
• (DCT, ge-IDCT) replaces (DCT, IDCT)
• Processing in Embedded Lapped Transform Domain
• Effective and Efficient Removal of Blocking
  Artifact

43
Ringing Artifact ReductionUsingMaximum
Likelihood Robust Filtering
S. Yang, Y. H. Hu, D. L. Tull, and T. Q. Nguyen,
Maximum likelihood parameter estimation for
image ringing artifact removal, IEEE Trans.
Circuits and Systems for Video Technology, vol.
11, 2001, (to appear).
44
MAP Estimation

• Wavelet Bit plane Coding Based Codecs.
• MAP Estimation
• Redesending Functions

45
Feasible Image Set

• For Bit Plane Coding
• Feasible Image Set

46
Approximated MAP Estimation

• Enforcement of the Feasible Image Set
• Replacement

Artifact Removal in Low Bit Rate Wavelet
Coding with Robust Nonlinear Filtering,
M. Shen and C.C.J. Kuo
47
Morphological Filter

• Detection Phase
• Removal Phase

Image coding ringing artifact reduction using
morphological post-filtering, part I the
algorithm, S.H.Oguz, Y.H.Hu, and T.Q.Nguyen
48
Flat Surface Model

• Replace ripped surface with flat surface.
• Flat Surface Model Images are montage of flat
  surfaces.
• Each surface consists of a cluster of pixel
  intensity values.

K the number of surfaces
? averaged grayscale value of each surface
z surface information (membership of individual
pixel)
49
Examples
50
ML Parameter Estimation

• Estimate model parameter from the samples.
• Maximum Likelihood Parameter Estimation

A pair of (G,z) is regarded as complete data G
pixel values Z membership of each pixel to a
particular surface (cluster)
51
k-means Algorithm

• K-means clustering is an implementation of the EM
  (expectation-maximization) Algorithm
• k-Means Algorithm

Expectation Maximization
52
Hierarchical K-Cluster Model

• Similarity between Two Clusters
• Insensitivity
• Modification of Vapniks ?-Insensitive
  Function

? variance
? cluster center
53
Hierarchical K-Cluster Model

• Cluster Similarity Measure (CSM)
• Optimal Number of Clusters

54
Hierarchical K-Cluster Model

• Algorithm
• Step1 For LKmax to 2 Run k-means algorithm
  with L clusters. Measure the max similarity
  between cluster pairs. Calculate CSM(L) Merge
  two most similar clusters.
• Step 2 Determine the number of clusters K.
• Step 3 Run k-means algorithm with K clusters.

55
Example
56
Example
57
Three Cluster Model

• Cluster Validation leads to a complex algorithm.
• Simplify to a Three Cluster Model
• Reduce the k-means algorithm to one iteration.

Gc the center pixel of G
58
Robust Filter

• Let C(i,j) be index set of pixels in G centered
  at (i,j)
• Define the set
• Define the function
• ML Estimation with

59
Applications
Lossless Decoding
Dequantization
Inverse Transform
(a) Standard Decoder
Lossless Decoding
Dequantization
Inverse Transform
Post- Processing
(b) Improved Decoder
60
VRM

• Visible Ringing Measure
• Average Local Variance around Major Edges

(a) Image
(b) Edge Map
(c) Filtering Mask
(d) VRM Mask
Image coding ringing artifact reduction using
morphological post-filtering, part I the
algorithm, S.H.Oguz, Y.H.Hu, and T.Q.Nguyen
61
Evaluation PSNR VRM
(a) PSNR
(b) VRM
62
Evaluation Subjective Quality
JPEG2000
K-means
RF
At 0.125 bpp
63
Comparative Study

• Existing Methods
• Approximated MAP Estimation (MAP-A)
• Morphological Filtering (MF)
• We implement
• MAP Estimation (MAP)

64
Comparison PSNR VRM
(a) PSNR
(b) VRM
65
Comparison Miss of MF
(b) Robust Filter
(a) Morphological Filter
66
Comparison Staircase Effect of MAP-A
(a) Approximated MAP
(b) Robust Filter
67
Comparison Simplicity
(a) Memory Usage in byte
(b) Execution Time in sec
RF
MF
MAP-A
With JPEG2000 compressed 2048x2560 size images
at 0.125 bpp
On 333MHz Dual Pentium,512M Ram, WindowsNT, in
sec
68
Summary

• ML Model Parameter Estimation
• Hierarchical Clustering Algorithm
• k-means algorithm
• CSM
• Robust Filter
• Based on three cluster model
• New Modeling of Pgf
• Effective and Efficient Removal of Ringing
  Artifact

69
Conclusion

• Post-processing can effectively reduce image
  coding artifact.
• An embedded LOT based blocking artifact reduction
  algorithm is presented.
• A maximum likelihood based robust filter is
  proposed to reduce ringing artifact
• These coding artifact reduction methods are
  amenable for hardware implementation for embedded
  applications in digital cameras as well as
  digital photo printers.

The ppt file can be downloaded from http//www.ec
e.wisc.edu/hu/postproc.ppt