Perceptual Image Filtering with Connected Operators and Image Inpainting Department of Signal Theory and Communications Technical University of Catalonia (UPC) Barcelona, Spain

1
Perceptual Image Filtering with Connected
Operators and Image Inpainting Department of
Signal Theory and CommunicationsTechnical
University of Catalonia (UPC)Barcelona, Spain

• Mariella Dimiccoli and Philippe Salembier

2
Objectives
Introduction ??

• Objective
• Improvement of filtering based on Connected
  Operators

Connected Operators
3
Objectives
Introduction ??

• Objective
• Improvement of filtering based on Connected
  Operators

Image Inpainting
Connected Operators

through Image Inpainting
4
Objectives
Introduction ??

• Presentation Outline
• Introduction
• Connected Operators
• Image Inpainting
• Proposed Approach
• Experimental Results
• Conclusions

5
Objectives
Connected Operators ????

• Presentation Outline
• Introduction
• Connected Operators
• Image Inpainting
• Proposed Approach
• Experimental Results
• Conclusions

6
Objectives
Connected Operators ????
Introduction

• Definition Flat zones that do not fulfill a
  given simplification criterion are removed and
  merged into a single one
• Property Cannot create new contours nor shift
  existing ones

• Construction strategies
• Reconstruction
• Region-tree pruning

7
Objectives
Connected Operators ????
Connected Operators by Reconstruction

• Relies on a pixel-based representation of the
  image
• is the reference image
• is the marker image
• is a generic reconstruction (leveling)

A connected operator by reconstruction can be
written as
where is a transformation that defines the
simplification effect
8
Objectives
Connected Operators ????
Connected Operators by Region Tree Pruning
Relies on a region-based representation of the
image

• The general approach involves three steps
• Tree-construction
• Tree root represents the image support
• Tree nodes represent image regions
• Tree branches represent inclusion relationships
  between nodes
• Tree-pruning
• Image construction

9
Objectives
Connected Operators ????
Perceptual Analysis

• Drawback Removed regions may be still
  perceptible in the filtered image as single flat
  zones

Size simplification Opening by reconstruction
of erosion
Proposed Solution Use inpainting to estimate
pixel values of flat zones that have been removed
and merged
10
Objectives
Image Inpainting ??

• Presentation Outline
• Introduction
• Connected Operators
• Image Inpainting
• Proposed Approach
• Experimental Results
• Conclusions

11
Objectives
Image Inpainting ??
Introduction
What is Image Inpainting? Restoring tool allowing
the recovery of a given part of an image based on
the surrounding information

• Classification
• Textural (textural exemplares)

• Structural (contour information)

12
Objectives
Image Inpainting ??
A state-of-Art Inpainting Algorithm
Variational Method Join interpolation of
isothopes and gray levels (Ballester et all.,
2001)
Basic Idea For each possible continuation, the
angular change of the normal along the whole
contour in is minimized
?
B
Masked
Inpainted
Original
Study of the convergence For each level set the
stopping criterion is defined the by a
thresholding on the slope of the energy function
13
Objectives
Proposed Approach ??

• Presentation Outline
• Introduction
• Connected Operators
• Image Inpainting
• Proposed Approach
• Experimental Results
• Conclusions

14
Objectives
Proposed Approach ??

• Proposed Solution

• Basic idea Use inpainting to estimate pixel
  values of flat zones
• that have been removed and merged

15
Objectives
Proposed Approach ??

• Proposed Solution

• Basic idea Use inpainting to estimate pixel
  values of flat zones
• that have been removed and merged

Which flat zone need to be estimated by
inpainting? Only the perceptually most important
removed regions
16
Objectives
Proposed Approach ??

• Implementation

17
Objectives
Proposed Approach ??

• Implementation

18
Objectives
Proposed Approach ??

• Implementation

19
Objectives
Proposed Approach ??

• Implementation

20
Objectives
Proposed Approach ??

• Implementation

21
Objectives
Proposed Approach ??

• Implementation

22
Objectives
Proposed Approach ??

• Implementation

23
Objectives
Proposed Approach ??

• Implementation

24
Objectives
Proposed Approach ??

• Implementation

25
Objectives
Proposed Approach ??

• Implementation

26
Objectives
Proposed Approach ??

• Implementation

27
Objectives
Experimental Results ?????

• Presentation Outline
• Introduction
• Connected Operators
• Image Inpainting
• Proposed Approach
• Experimental Results
• Conclusions

28
Objectives
Experimental Results ?????
Size criterion
Size criterion is defined as allows
It allows the removal of those flat zones
that represent a maxima of size inferior to the
size of the structuring element
29
Objectives
Experimental Results ?????
Contrast criterion
Contrast criterion is defined as follows
It allows the removal of those flat zones that
have a contrast inferior to
Original
Filtered
Mask
Result
30
Objectives
Experimental Results ?????
Motion criterion
Motion criterion allows the removal of those flat
zones that do not undergo a given motion
Original
Filtered
Mask
Result
31
Objectives
Experimental Results ?????
Complexity and contrast criteria
Original
Reference
32
Objectives
Experimental Results ?????
Complexity and contrast criteria
Complexity Criterion allows the removal of those
flat zones that have a big ratio between
perimeter and area
Original
33
Objectives
Experimental Results ?????
Complexity and contrast criteria
Complexity Criterion allows the removal of those
flat zones that have a big ratio between
perimeter and area
Original
Contrast Oriented Connected Operator
Complexity Oriented Connected Operator
IA
IB
ID
IE


-
IC

34
Objectives
Experimental Results ?????
Complexity and contrast criteria
Original
35
Objectives
Experimental Results ?????
Complexity and contrast criteria
Original
36
Objectives
Experimental Results ?????
Complexity and contrast criteria
Original
37
Objectives
Experimental Results ?????
Complexity and contrast criteria
Result
Original
38
Objectives
Experimental Results ?????
Complexity and contrast criteria
Original image detail
Original
Filtered
Result image detail
Mask
Result
39
Objectives
Experimental Results ?????

• Quality Assessment

• Given a reference and a distorted image
• The PSNR (Peak Signal to Noise Ratio) quantifies
  objectively the distortion but it is not related
  with the perceptual visual appearance.
• The SSIM (Structural Similarity Image Measure)
  quantifies the perceptual similarity
  independently on local contrast and local
  luminance changes
• The SSIM is equal to 1 for two identical images

Average
Median
Type of filter PSNR SSIM
Average (5x5) 19.8 0.06
Median (5x5) 23.2 0.87
Connected Operator 21.9 0.91
Proposed Approach 39.2 0.99
Connected Operator
Proposed Approach
40
Objectives
Conclusions ?

• Presentation Outline
• Introduction
• Connected Operators
• Image Inpainting
• Proposed Approach
• Experimental Results
• Conclusions

41
Objectives
Conclusions ?

• Summarizing..
• A filtering strategy combining Connected
  operators and Image Inpainting has been
  presented.
• Based on experimental results, the proposed
  approach is
• Efficient Outperforms objectively other
  filtering strategies
• Flexible Any connected operator can be used
  involving also semantic criteria
• Future work will be devoted to improve the
  current method to select the regions to be
  inpainted

42
Perceptual Image Filtering with Connected
Operators and Image Inpainting

• Any Question?

Mariella Dimiccoli and Philippe Salembier
43
Perceptual Image Filtering with Connected
Operators and Image Inpainting

• Thank you!

Mariella Dimiccoli and Philippe Salembier