Pattern Recognition and Image Analysis Group (RFAI), Document (Image) Analysis related work

1
Pattern Recognition and Image Analysis Group
(RFAI)Document (Image) Analysis related
workLaboratory of Computer Science
(LI)François Rabelais UniversityTours city,
France
2
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition
• 6.7. Graph matching and embedding

3
Tours city
Paris
Tours
- 137 046 people (2009) - 204 km southwest of
Paris - Region   centre - Indre et loire  -
1h20 from Paris by high speed train - Direct
train connection to Charles de Gaulle -Orly
airport in 2h00
4
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition
• 6.7. Graph matching and embedding

5
François-Rabelais University,les deux lions /
Portalis
François Rabelais University
Faculties Art human sciences, economy, business manage-ment, health, information and technology
Students 21 207 (2 500 foreign students)
Teachers 1 300
Support staff 1000
Laboratories 40
Place 5
François Rabelais i.e. a famous French writer of
XV Century
6
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition
• 6.7. Graph matching and embedding

7
School of Engineering Polytech
- 12 schools in France (Grenoble, Lille,
Marseille, Montpellier, Nantes, Nice-Sophia,
Paris-UPMC, Paris ORSAY, Savoie, Orléans, Tours,
Clermont-Ferrand) - 12 000 students

• 720 students
• - 5 departments (with Labs)

Urban Planning
CITERES
Mechanics
LMR
Electronics
LMP
Computer Science
LI
Embedded computing
8
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition

9
Laboratory of Computer Science
77 people, 5 research groups (2009)
Data Bases and Natural Language Processing
Pattern Recognition and Image Analysis
Visual Data Mining and Biomimetic Algorithms
Handicap and New Technologies
Scheduling and Control
10
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition

11
Pattern Recognition and Image Analysis (RFAI) (1)

• Medical Imaging
• - Image segmentation (ultrasound, MRI)
• - Video analysis, 3D reconstruction
• Document Image Analysis
• - Layout analysis document recognition
• - OCR, word spotting signature verification
• - Symbol recognition spotting
• - Content based Image Retrieval
• - Camera based Recognition
• - Graph matching and embedding
• Machine learning for time series prediction

12
Pattern Recognition and Image Analysis (RFAI) (2)
Professors
Hubert Cardot
Jean-Yves Ramel
Romuald Boné
Thierry Brouard
Alireza Alaei
Sabine Barrat
Muzzamil Luqman
Mathieu Delalandre
Romain Raveaux
PhD
Partha Roy
Nicolas Ragot
Gilles Verley
Julien Olivier
Nicolas Sidere
Pascal Makris
12
13
Pattern Recognition and Image Analysis (RFAI) (3)
Aymen Cherif
Fareed Ahmed
Cyrille Faucheux
Ahmed Ben Salah
PhD Students engineers
The Anh Pham
Frédéric Rayar
Anh Khoi Ngo ho
13
14
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition

15
Projects partners outline (1)
International projects National projects Local government projects Scholarships Partnership contracts
Madonne 2003-2006 ?
Navidomass 2006-2009 ?
EPEIRES 2004-2007 ?
BVH 2004-today ?
ATOS 2005-today ?
PIVOAN 2008-2009 ?
HEC 2005-2011 ?
SNECMA 2008-2011 ?
AAP 2010-2011 ?
VIED 2010-2013 ?
Bnf 2010-2013 ?
Digidoc 2011-2014 ?
Google 2011-2012 ?
ISRC2011 2011-2012 ?
DOD 2011-2015 ?
IndoFrench 2012-2015 ?
SPD 2012-2015 ?
16
Projects partners outline (2)
National projects
People Institutes Length Funding
ACI MADONNE 2003-2006 55 8 2 years 110 k
ANR Navidomass 2006-2009 40 7 3 years 443 k
Technovision ÉPEIRES 2004-2007 30 7 3 years 100 k
ANR Digidoc 2011-2014 18 7 3 years 866 k
Centre de Recherche en Informatique de Paris 5
(Paris)
Institut de Recherche en Informatique et
Systèmes Aléatoires (Rennes)
Centre dEtude Supérieures de la Renaissance
(Tours)
Laboratoire Lorrain de Recherche en Informatique
et ses Applications (Nancy)
Laboratoire d'Informatique de Traitement de
l'Information (Rouen)
Laboratoire d'InfoRmatique en Image et Systèmes
d'information (Lyon)
Laboratoire dinformatique image et interaction
(La Rochelle)
Laboratoire Bordelais de Recherche en
Informatique (Bordeaux)
Laboratoire Informatique (Tours)
17
Projects partners outline (3)
Partnership contracts
Local government projects (i.e. projets region
centre)
People Institutes Length Funding
PIVOAN 3 1 1 year 33 k
AAP 3 1 1 year 38 k
So famous !
Centre des études supérieures de la renaissance
bibliothèque virtuelle humaniste
PhD Scholarships
Maison des Sciences de l'Homme
international high-technology group in aerospace,
defense and security
R.J. Qureshi Higher Education Commission (HEC) of Pakistan
M. Luqman Higher Education Commission (HEC) of Pakistan
T.H. Pham Vietnam International Education Development (VIED)
Bibliothèque Nationale de France - portail Gallica
Bilateral program
Digitalisation company
People Institutes Length Funding
IndoFrench 3 2 3 year 70 k
capturing, automatically processing, and managing
all companys incoming documents
Atos Origin is a leading international IT
services provider for business solutions
18
Projects partners outline (4)
Computer Vision Center Document Analysis
Group Barcelona - Spain J. Llados, E. Valveny
Dept. of Computer Science and IS Osaka Prefecture
University Osaka - Japan K. Kise
Indian Statistical Institute Kolkata - India U.
Pal
Computational Intelligence Laboratory Athens -
Greece B. Gatos
19
Projects partners outline (5)
Layout analysis document recognition OCR, word spotting signature verification Symbol recognition spotting Graph matching embedding CBIR Camera based Recognition
Madonne 2003-2006 ?
Navidomass 2006-2009 ? ?
EPEIRES 2004-2007 ?
BVH 2005-today ? ?
ATOS 2005-2009 ?
PIVOAN 2008-2009 ? ?
HEC 2005-2011 ? ?
SNECMA 2008-2011 ?
AAP 2010-2011 ?
VIED 2010-2013 ? ?
Bnf 2010-2013 ?
Digidoc 2011-2014 ?
Google 2011-2012 ?
ISRC2011 2011-2012 ?
DOD 2011-2015 ? ?
IndoFrench 2012-2015 ?
SPD 2012-2015 ?
20
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition

21
Layout analysis document recognitionAGORA (1)
(1) Text/graphics separation
People Jean-Yves Ramel
Funding CESR partnership, Madonne, PIVOAN
Starting 2005
Ref J.Y. Ramel and al. User-driven Page Layout Analysis of historical printed Books. IJDAR, 2007.
Foreground map adaptive binarization Saul2000
with connected component labeling, text/graphics
separation is done in terms of size of connected
components
(2) Line/word segmentation
1. Background map statistical distribution of
white and black pixel on horizontal and vertical
scanline
2. Fusion word segmentation (i.e. connected
components grouping) is done in terms of
thresholding on the background map.
22
Layout analysis document recognitionAGORA (2)
(3) Interactive system (i.e. user driven analysis)
People Jean-Yves Ramel
Funding CESR partnership, Madonne, PIVOAN
Starting 2005
Ref J.Y. Ramel and al. User-driven Page Layout Analysis of historical printed Books. IJDAR, 2007.
(4) Results, since 2005 300 books (50 000
pages) http//www.bvh.univ-tours.fr/
22
23
Layout analysis document recognitionDocument
image characterization (1)
People Nicholas Journet
Funding CESR partnership Madonne project
Starting 2006
Ref N. Journet and al. Document Image Characterization Using a Multiresolution Analysis of the Texture Application to Old Documents. IJDAR, 2008.
(1) Descriptor based on five features
Directional rose (1) main direction
Directional rose (2) isotropy
Directional rose (3) standard deviation
Spatial (4) ink/paper transition
Spatial (5) white spaces separating the collateral elements
24
Layout analysis document recognitionDocument
image characterization (2)

• (2) Segmentation
• features are extracted at four different
  resolution (4?5 20 features)
• features are then processed with the clustering
  algorithm CLARA (Clustering LARge Applications)
  Kaufman1990 to achieve automatic segmentation
  in text/graphics/background

People Nicholas Journet
Funding CESR partnership Madonne project
Starting 2006
Ref N. Journet and al. Document Image Characterization Using a Multiresolution Analysis of the Texture Application to Old Documents. IJDAR, 2008.
25
Layout analysis document recognitionDocument
image characterization (3)

• (3) Indexing applied on two different problems
• Layout retrieval, distance is based on a
  contingency table Younes2004
• Graphics retrieval, distance based on a
  dissimilarity function

People Nicholas Journet
Funding CESR partnership Madonne project
Starting 2006
Ref N. Journet and al. Document Image Characterization Using a Multiresolution Analysis of the Texture Application to Old Documents. IJDAR, 2008.
Handmade dataset I (400 images)
Handmade dataset II (400 images)
26
Layout analysis document recognition
Cognitive digitalization
Topic Incremental and interactive learning for
document image, application for intelligent
cognitive scanning of old documents. Problematic
- Estimate the scan parameters according
to usage, past experience. - Improve the scan
parameters for a document during the scanning. -
Detect the default settings for a document, a
collection, a work.
People A.K. Ngo ho, N. Ragot, J.Y. Ramel
Funding Digidoc project
Starting 10/2011
Ref Na
27
Layout analysis document recognition Document
classification
Form
Publicity

• Topic Recognition of administrative forms for
  companies
• Problematic
• - high variability 600 to 800 classes
• binary images at 300 dpi
• time constraint ? to 1,5 s per image
• commercial systems cant outperform
• a 60 recognition rate
• Goals
• 1. To gain in robustness (set of adapted
• and robust specialists)
• 2. To gain in flexibility (self
• learning, content adaptation)

People Mathieu Delalandre
Funding DOD project
Starting 12/2011
Ref Na
Free letter
Acknowledge reply
28
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition

29
OCR, word spotting and signature verification
Robust OCR-I (1)
Key idea improving OCR robustness by using
similar technics as those used for handwriting
recognition - Hidden Markov Models without
explicit segmentation - Adapting a polyfont
OCR to specificities of pages (fonts/noise)
People Kamel Ait-Mohand
Funding Navidomass project
Starting 2006
Ref K. Ait-Mohand and al. Structure Adaptation of HMM Applied to OCR. ICPR, 2010.
(1) Feature extraction is based on a sliding
window and HoG features (no word/character
segmentation)
Sliding window

• (2) HMM classification and training
• HMM characters models are learnt on a synthetic
  dataset (numerous fonts, degradations possible,
  no limits in the number of samples per character)
  gt polyfont OCR system
• Each character model can be adapted to a specific
  font/book using only few lines of transcriptions.
  The HMM model is adapted at the structure level
  (number of states) and at the parameter level
  (Gaussian MAP adaptation).

initial model
Structure adaptation
New model
training
30
OCR, word spotting and signature verification
Robust OCR-I (2)
Experiments done using 100 fonts with the
degradation model of Baird
People Kamel Ait-Mohand
Funding Navidomass project
Starting 2006
Ref K. Ait-Mohand and al. Structure Adaptation of HMM Applied to OCR. ICPR, 2010.
blurred
Training Testing
Degradation models Baird Baird
Font size 12 12
Resolution 300 dpi 300 dpi
Fonts 70 30
Image sets (lines) 10000 15000
thresholding
sparse pixels
Commercial OCR Polyfont Adapted
Average (30 fonts) 88.72 91.70 97.37
69.21 98.33 98.68
44.78 85.46 98.09
52.73 39.73 74.89
31
OCR, word spotting and signature verification
Robust OCR-II

• Topic digitalization and indexing of a military
  document database for retired pay
• Problematic
• large amount of data (800 000 applications every
  3 years)
• large heterogeneity
• from XIX century middle to today,
• handwritten and typographic documents,
• different languages,
• no common layout,
• different colors,
• etc.

People J.Y. Ramel, N. Ragot, S. Barrat
Funding SDP
Starting 10/20012
Ref Na
32
OCR, word spotting and signature verification
Performances prediction/control of OCR
People Ahmed Ben Salah, K. Ait-Mohand
Funding BNF
Starting 10/2010
Ref Na

• Problematic control and cost reduction of the
  digitization service to know which
  collection/document/part of document is OCRisable
  and at which quality
• Select only adequate documents to be sent to the
  private service provider in charge of the
  digitization and OCRisation
• Studies of relationships between meta-data
  information (date, format, ) and OCR results gt
  difficult without deep analysis of the pages
• Characterization of image content with SIFTLBP
  regression towards OCR results
• Control of OCR quality assessed by the service
  provider
• Detection of text zones forgotten by OCR using
  correct detection performed (contextual
  information) gt in progress
• Verification of OCR result by matching with image
  (gt in a near futur)

Real OCR Result ? Perfomance expected ? 0-70 70-80 80-85 85-90 90-100
Good ( 88) 0 0 6,1 23,11 77,12
Undecidable (82-88) 5,5 16 53,43 59,11 20,34
Bad (lt82) 94,55 84 40,47 17,78 2,54
33
OCR, word spotting and signature
verificationSemi-automatic transcription (1)
Topic user driven transcription of character in
historical books
People Jean-Yves Ramel, Frédéric Rayar
Funding CESR partnership, PIVOAN, Google
Starting 2008
Ref S. Hocquet and al. Analyse de classes de formes pour la transcription de textes imprimés anciens. CIFED, 2010

• (1) Segmentation process based on Agora?
  Standardized output (e.g. Alto)
• (2) Clustering process
• Finer description of shapes
• Features extraction and selection

(3) Transcription Typography studies
34
OCR, word spotting and signature
verificationSemi-automatic transcription (2)
Experiments, The Vésales book
People Jean-Yves Ramel, Frédéric Rayar
Funding CESR partnership, PIVOAN, Google
Starting 2008
Ref S. Hocquet and al. Analyse de classes de formes pour la transcription de textes imprimés anciens. CIFED, 2010
Pages connected components Custers (i.e. Classes) Custers (i.e. Classes) Custers (i.e. Classes)
150 1 062 081 40 000 40 000 40 000
150 1 062 081 10 10 90
150 1 062 081 gt 10 occurrences gt 10 occurrences lt 10 occurrences
150 1 062 081 93 of the text 93 of the text 7 of the text
150 1 062 081 0.5 (top 200) 89.5 7 of the text
150 1 062 081 85 of text 8 of text 7 of the text
Reasons is noise
spot
touching characters
character on verso
split character
35
OCR, word spotting and signature
verificationWord Spotting (1)
Topic Word Retrieval in Historical Documents
People P.P. Roy, J.Y. Ramel, F. Rayar
Funding AAP , Renon
Starting 10/2010
Ref P. P. Roy and al. An efficient coarse-to-fine indexing technique for fast text retrieval in historical documents", DAS 2012.
36
OCR, word spotting and signature
verificationWord Spotting (2)
Topic Word Retrieval in Historical Documents
People P.P. Roy, J.Y. Ramel, F. Rayar
Funding AAP , Renon
Starting 10/2010
Ref P. P. Roy, F.Rayar and J.Y.Ramel. An efficient coarse-to-fine indexing technique for fast text retrieval in historical documents", DAS 2012.
The codebook is created using a clustering
algorithm by template matching of similarity
Overcoming of segmentation problems are solved by
the Water reservoir method.
Query word is thus converted into a string of
primitives. Approximate string matching algorithm
is used for string matching

• Tests done from
• - 24 pages
• corresponding to 57324 primitives
• clustered in 183 representative primitives
• P/R computed with 20 query word images

37
OCR, word spotting and signature verification
Multlingual Word Spotting

• Topic Robust multilingual word spotting
• Problematic
• Query by text/image
• Partial matching allowed (for occlusion, special
  characters)
• Matching in two steps global (shape context) /
  local (HMM)

People N. Ragot, J. Y. Ramel, U. Pal
Funding IFCPAR
Starting 04/2012
Ref Na
38
OCR, word spotting and signature verification
Online signature verification (1)
People Nicolas Ragot
Funding ATOS project
Starting 2005
Ref N. Ragot and al. Study of Temporal Variability in On-Line Signature Verification. ICPR, 2008
Problematic to evaluate impact of temporality
(i.e. time evolution) on signature, for
performance evaluation of signature verification
algorithms.
(1) Database acquisition
signers 18 18 18 18
sessions 12 12 12 12
signatures/ session 10 mean time interval 2 weeks
total signatures 2160 total duration 25 months
39
OCR, word spotting and signature verification
Online signature verification (2)
variation correlation correlation
Speed variation yes
Length variation yes no
Duration stable no
People Nicolas Ragot
Funding ATOS project
Starting 2005
Ref N. Ragot and al. Study of Temporal Variability in On-Line Signature Verification. ICPR, 2008
(2) Statistical analysis
(2.1) Global i.e. without temporal variability
(2.1) With temporal variability
Total duration per signer/ session
Total length per signer/ session
(3) Performance evaluation Authentication (i.e.
recognition) algorithm based on a Coarse to fine
approach - Coarse step on basic features
(length, duration) - Fine step based on DTW
Proposed dataset
Dataset without temporality
40
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition

41
Symbol recognition spottingVectorization and
GbR (1)
(1) Contour detection, chaining and
polygonalisation Wall1984 (2) Quadrilateral
building
People Jean-Yves Ramel
Funding Na
Starting Na
Ref J.Y. Ramel. A Structural Representation for Understanding Line-Drawing Images. IJDAR, 2000.
(2.1.) Matching
(2.2.) Sorting
(2.3.) Merging
41
42
Symbol recognition spottingVectorization and
GbR (2)
(3) Graph based representation
People Jean-Yves Ramel
Funding Na
Starting Na
Ref J.Y. Ramel. A Structural Representation for Understanding Line-Drawing Images. IJDAR, 2000.
(3) Pros and cons
Cons - lost of connectivity
Pro - better representation of filled crossed
areas
Cons - parasite quadrilaterals
43
Symbol recognition spottingGeneration of
synthetic documents (1)
People Mathieu Delalandre
Funding Na
Starting Na
Ref M. Delalandre and. Generation of Synthetic Documents for Performance Evaluation of Symbol Recognition Spotting Systems. IJDAR, 2010.
Key idea
Graphical documents are composed of two layers
(1) Constraint model
43
44
Symbol recognition spottingGeneration of
synthetic documents (2)
People Mathieu Delalandre
Funding Na
Starting Na
Ref M. Delalandre and. Generation of Synthetic Documents for Performance Evaluation of Symbol Recognition Spotting Systems. IJDAR, 2010.
(2) Building engine and user interaction
44
45
Symbol recognition spottingGeneration of
synthetic documents (3)
People Mathieu Delalandre
Funding Na
Starting Na
Ref M. Delalandre and. Generation of Synthetic Documents for Performance Evaluation of Symbol Recognition Spotting Systems. IJDAR, 2010.
(3) Datasets
Mean localization results
(4) Performance evaluation - Goal is to
evaluate variability impact of produced
datasets on spotting system(s) - Experiments
have been done from the spotting system of
R. Qureshi
Background sets
46
Symbol recognition spottingGraph scoring for
symbol spotting (1)
People Rashid Qureshi
Funding HEC scholarship
Starting 2005
Ref R. Qureshi and al. Spotting Symbols in Line Drawing Images Using Graph Representations. GREC, 2008.
(1) Graph based representation based on the
Jean-Yves Ramels work
(2) Seeds detection in graph a set of scoring
functions is computed from all nodes and edges
Scoring functions Scoring functions Scoring functions
Edges PE1 parallel segments
Edges PE2 junctions
Edges PE3 comparable length segments
Nodes PN2 2-3 connection
Nodes PN3 short length segments
(3) Score propagation based on a shortest path
algorithm, a global score is normalized from
individual score of edge/node
46
47
Symbol recognition spotting Graph scoring for
symbol spotting (2)
People Rashid Qureshi
Funding HEC scholarship
Starting 2005
Ref R. Qureshi and al. Spotting Symbols in Line Drawing Images Using Graph Representations. GREC, 2008.
(4) Results performance evaluation
47
48
Symbol recognition spottingBayesian based
system for symbol spotting (1)
(1) Representation phase used the graph based
representation of Jean-Yves Ramel
People Muzzamil Luqman
Funding HEC scholarship
Starting 2008
Ref M.M. Luqman. A Content Spotting System for Line Drawing Graphic Document Images. ICPR, 2010.
(2) Description phase approach based on
attributes (of nodes and edges)
(3) Learning and classification phases base on
Bayesian network
(3.1.) Discretization step based on the Akaike
Information Criterion
(3.2.) Learning step - network topology is
done from a genetic algorithm - parameters
conditional probabilities is done from a
maximum likelihood estimation
(3.3.) Classification step
48
49
Symbol recognition spottingBayesian based
system for symbol spotting (2)
(4) Performance evaluation at recognition level
People Muzzamil Luqman
Funding HEC scholarship
Starting 2008
Ref M.M. Luqman. A Content Spotting System for Line Drawing Graphic Document Images. ICPR, 2010.
ISRC 2003 dataset
Rotation Scaling Scalability Scalability Scalability Scalability
Clean Clean yes Yes 100 100 100 100
Hand drawn level1 no no 99 96 93 92
Hand drawn level2 no no 98 94 92 91
Hand drawn level3 no no 91 77 71 69
Binary degrade Binary degrade no no 98 95 93 92
clean
Hand drawn
Binary degrade
(5) Improvements of Rashid Qureshis results
49
50
Symbol recognition spottingGraph Embedding
Topic Topological Graph Embedding
People N. Sidere, J.Y. Ramel
Funding Navidomass
Starting 2007
Ref N. Sidère et al. Vector Representation of Graphs Application to the Classification of Symbols and Letters. ICDAR 2009.
(1) A lexicon is generated from the network of
non-isomorphic graphs
(2) The embedding is based on occurrences of the
patterns
51
Symbol recognition spottingInternational
Symbol Recognition Contest 2011 (1)
Workshop GREC 2011
Contest starting March 2011
Training datasets 6th of April 2011
Call of participation 2sd of May 2011
Final datasets 25th of July
Contest slot 25th July - 1st August 2011
People M. Delalandre, R. Raveaux
Funding Support of TC10
Starting 2011
Ref E. Valveny and al. Report on the Symbol Recognition and Spotting Contest. GRECLNCS 2012.
http//iapr-tc10.univ-lr.fr/index.php/symbol-conte
st-2011
id domain models symbols distortion

Training 1-7 Technical 36-150 16650 Rotation, Scaling, Kanungo, Context
Final 1-4 Technical 36-150 16800 Rotation, Scaling, Kanungo, Context
Recognition Tests
id domain models images symbols distortion

Training 8-15 Electrical Architectural 16-21 40 835 None, Kanungo
Final 5-12 Electrical Architectural 16-21 120 3463 None, Kanungo
Localization Tests
52
Symbol recognition spottingInternational
Symbol Recognition Contest 2011 (2)
The participant Nayef, N. Breuel, T. On the
Use of Geometric Matching for Both Isolated
Symbol Recognition and Symbol Spotting Workshop
on Graphics Recognition (GREC), 2011
People M. Delalandre, R. Raveaux
Funding Support of TC10
Starting 2011
Ref E. Valveny and al. Report on the Symbol Recognition and Spotting Contest. GRECLNCS 2012.
Connected components filtering
Contour detection and sampling
Geometric matching
set name models noise recognition rate

Final 1 50 Kanungo 94.76
Final 3 150 Kanungo 85.88
Final 4 36 Context 96.22
groundtruth
results
Recognition Tests
set name domain models noise precision / recall

Final 5 Architectural 16 None 0.62 / 0.99
Final 6 Architectural 16 Kanungo 0.64 / 0.98
Final 9 Electrical 21 None 0.37 / 0.56
Final 10 Electrical 21 Kanungo 0.44 / 0.63
intersection
Localization Tests
52
53
Symbol recognition spottingPerformance
characterization of symbol localization (1)
People Mathieu Delalandre
Funding Na
Starting 2008
Ref M. Delalandre and al. A Performance Characterization Algorithm for Symbol Localization. GREC, 2010.
Open problem
how to make the difference between segmentation
errors of background with segmentation errors of
objects
The characterization method must do some
rejection, ways to solve are... 1. To define and
apply manual threshold (bad ...) 2. To propose a
method for adaptative tthresholding, how to do ?
Key idea, characterization method based on context
53
54
Symbol recognition spottingPerformance
characterization of symbol localization (2)
Groundtruth
Results
People Mathieu Delalandre
Funding Na
Starting 2008
Ref M. Delalandre and al. A Performance Characterization Algorithm for Symbol Localization. GREC, 2010.
(1) The method
(1.1) Localization comparison moves results from
Euclidean to a scale space, to deal with the
scale invariance
3.1 Localization comparison
(1.2) Probability scores are computed from a
groundtruth point gi and the result point r,
considering the neighboring groundtruth point gj.
Final result is computed considering all the
groundtruth using a probability score function
3.2 Probability scores
(1.3) Matching algorithm looks for statistical
distribution of single, miss, merge and split
cases, in an decreasing order of precision using
a bipartite list.
3.3 Matching algorithm
(1.4) Transform function make results context
independent, making difference with self-matching
of groundtruth, to achieve coherent comparison of
methods on different datasets
3.4 Transform function
54
55
Symbol recognition spottingPerformance
characterization of symbol localization (3)
electrical diagrams
?i(1) 0.529
People Mathieu Delalandre
Funding Na
Starting 2008
Ref M. Delalandre and al. A Performance Characterization Algorithm for Symbol Localization. GREC, 2010.
(2) Results obtained from the Rashid Qureshis
system
?i(1) 0.496
floorplans
?i(e)
SESYD dataset
1,00
Drawing level Drawing level Symbol level Symbol level

Setting backgrounds 5 models 16
Dataset images 100 symbols 2521

Setting backgrounds 5 models 17
Dataset images 100 symbols 1340
score error (e)
floorplans
diagrams
?i(e)
score error (e)
56
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition

57
Content based Image RetrievalRobust key points
detection for document image retrieval
People The Anh Pham
Funding VEID scholarship
Starting 10/2010
Ref Na
The work focuses on robustness-keypoint
detection, robustness in DIA including 2D and
illumination, noises, artifacts,
blurred. Keypoints detection takes part of the
general features extraction, some typical
features used as key points and their
characteristics are
So, some ideas of new robust keypoint detection
may be - Detecting robust features first,
then extracting salient points, or - Using
robust methods (i.e Machine learning, Model-
based, Parametric-based) to extract salient
points, or - Combination of above methods
57
58
Content based Image RetrievalLogo recognition
and spotting

• Problematic
• document classification can be supported by logo
  recognition
• a meta engine will manage the decision rules
• spotting will depend of a previous stage of page
  segmentation
• binary images at 300 dpi
• time constraint ? to 1,5 s per image

People Mathieu Delalandre
Funding DOD project
Starting 04/2012
Ref Na
Some logo recognition case looks like OCR only
Logo are rich graphical parts, 300 dpi and
binarization could result in a high level of
degradation
High variability and scalability of logos
59
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition
• 6.7. Graph matching and embedding

60
Camera based recognitionRobust OCR for video
text recognition
People Thierry brouard
Funding SNECMA project
Starting 2008
Ref International patent
Problematic -Automatic routing of input
letters by digitization and OCR of input
documents received from customers -Response
time lt 3 s, Recognition rate gt 80, Precision
Equal to 100, Java environment on mutualized
servers Approach based on cognitive vision and
knowledge based systems (blackboard
mathematical theory of evidence), to achieve
robust segmentation OCR
61
Camera based recognitionReal time logos
recognition in urban environment
People Mathieu Delalandre
Funding Na (JSPS grant)
Starting 2010
Ref Na
Problematic - Logo detection from video
capture using some handled interactions,
to display context based information
(tourist check points, bus stop, meal,
etc.). - Hard points are the real time
constraints and the complexity of the
recognition task.
First goal of the project is to support the real
time recognition. We start from the hypothesis
than logo appear in a static way in video. We
propose to achieve an automatic control/selection
of image capture to reduce the amount of data to
process.
62
Talk workplan

• Tours city
• François-Rabelais University, les deux lions /
  Portalis
• School of Engineering PolytechTours
• 4. Laboratory of Computer Science
• 5. RFAI group
• 6. DIA related work
• 6.1. Projects partners outline
• 6.2. Layout analysis and document recognition
• 6.3. OCR, word spotting and signature
  verification
• 6.4. Symbol recognition spotting
• 6.5. Content Based Image Retrieval
• 6.6. Camera based recognition

63
Analysis and recognition of image contents Graph
Matching
Application to document images with a new
similarity measure between graphs Qureshi2006
Shape comparison
Length comparison
Connexion type
Angle