A Performance Characterization Algorithm for Symbol Localization

1
A Performance Characterization Algorithmfor
Symbol Localization

• Mathieu Delalandre1,2, Jean-Yves Ramel2, Ernest
  Valveny1 and Muhammad Muzzamil Luqman1,2
• 1 CVC, Barcelona city, Spain
• 2 LI Laboratory, Tours city, France
• LaBRI - Partnerships Meeting
• Bordeaux, France
• Thursday 14th of October 2010

2
A Performance Characterization Algorithmfor
Symbol Localization
Symbol localization systems (recognition and
spotting)
Performance characterization
To make the correspondence in term of localization
To compute characterization measures (recall,
precision, recognition rates, etc.)
3
A Performance Characterization Algorithmfor
Symbol Localization
Performance evaluation of image segmentation
Zhang1996
Performance evaluation of object localization
Delalandre2009

• Global discrepancy methods
• Number of missed segmented pixels
• Position of missed segmented pixels
• Local discrepancy methods
• Number of region in the image
• Features values of regions

Single an object in groundtruth matches only
with one detected object. Split two objects in
groundtruth match with one detected
object. Merge an object in groundtruth matches
with two detected objects.
truth results
False alarm a detected object doesn't match
with any object in groundtruth. Miss an object
in groundtruth doesn't match with any detected
object.
Performance evaluation image segmentation vs.
object localization
Performance evaluation Performance evaluation
image segmentation object localization
coverage of results all image part of
precision of localization high importance weak importance
semantic matching weak importance high importance
4
A Performance Characterization Algorithmfor
Symbol Localization
Performance evaluation of object localization
Delalandre2009
False alarm a detected object doesn't match
with any object in groundtruth. Miss an object
in groundtruth doesn't match with any detected
object.
Single an object in groundtruth matches only
with one detected object. Split two objects in
groundtruth match with one detected
object. Merge an object in groundtruth matches
with two detected objects.
truth results
Layout analysis Antonacopoulos1999
Symbol spotting Rusinol2009
Text/graphics separation Liu1997
char and text boxes
isothetic polygons
Convex hulls
5
A Performance Characterization Algorithmfor
Symbol Localization
Open problem with object localization
in a part of segmentation problem, how to make
the difference between segmentation errors of
background with segmentation errors of objects
Ways to solve ... 1. naive To use thresholds
to reject some segmentation results (bad
...) 2. ideal To define directed knowledge
based approaches to model localization/segmentatio
n algorithms (hard ...) 3. intermediate
(proposed) To use fuzzy-based approach,
to characterize the characterization results
according to confidence rate i.e. this is a
positive matching between groundtruth and
systems results with a confidence rate of ?.
6
A Performance Characterization Algorithmfor
Symbol Localization
7
A Performance Characterization Algorithmfor
Symbol Localization
8
A Performance Characterization Algorithmfor
Symbol Localization
null probabilities, equidistant case
highest probabilities, nearest points
maximum probability, equality case
9
A Performance Characterization Algorithmfor
Symbol Localization
How to compute the probability between a
groundtruth point gi and the result point r,
considering the neighboring groundtruth point gj
we define - pi the probability r ?
gi, regarding gj - si is the scaling factor
between gi and r - sj is the scaling factor
between gj and r
r
gi
gj
cases
si 0 sj k sj? ? si k si sj sj 0 si k si? ? sj k
0 0 1 ? ?
1 1 0 0 0
r
r
r
gi
gj
gi
gj
gi
gj
10
A Performance Characterization Algorithmfor
Symbol Localization
Thus, our probability function must respect the
following properties
0 1 ?
1 0 0
Several mathematics functions could be used
(affine, exponential, trigonometric, etc.) we
choose a Gaussian based function as it is good
model of random distribution
11
A Performance Characterization Algorithmfor
Symbol Localization
We extend the computation of probability to a
neighboring composed of n groundtruth points like
this we define - is the set
of groundtruth points - si is the scaling
factor between gi and r - are
the scaling factors between and r
- is the
probability r ? gi, regarding
12
A Performance Characterization Algorithmfor
Symbol Localization
13
A Performance Characterization Algorithmfor
Symbol Localization
floorplans
max
Score 0,11
Ts 0,57
Tf 0,31
Tm 0,20
Drawing level Drawing level Symbol level Symbol level

Setting backgrounds 5 models 16
Dataset images 100 symbols 2521

Setting backgrounds 5 models 17
Dataset images 100 symbols 1340
floorplans
diagrams
diagrams
max
Score 0,20
Ts 0,62
Tf 0,06
Tm 0,37
14
A Performance Characterization Algorithmfor
Symbol Localization
Each result is context dependent, how to compare
them ?
15
A Performance Characterization Algorithmfor
Symbol Localization
We compute the difference between a result and
self-matching of his groundtruth (g), to make the
new results test-independent.
Transform function
q ?i
0 0
n 1
?? ?0
16
A Performance Characterization Algorithmfor
Symbol Localization
electrical diagrams
?i(1) 0.529
?i(1) 0.496
floorplans
?i(e)
Drawing level Drawing level Symbol level Symbol level

Setting backgrounds 5 models 16
Dataset images 100 symbols 2521

Setting backgrounds 5 models 17
Dataset images 100 symbols 1340
floorplans
1,00
score error (e)
diagrams
?i(e)
score error (e)
17
Conclusion and perspectives

• Conclusion
• A new fuzzy way to evaluate object localization
• distribution of matching cases regarding a
  confidence rate
• Experimentation with a real system
• electrical and architectural drawings, 200 test
  images, 3821 symbols
• Perspectives
• Extending experiments
• several systems, to add noise, scalability, real
  datasets