Pavel Praks

1
Intelligent Image Retrieval using Numerical
Linear Algebra and Applications

• Pavel Praks
• Dept. of Math. and Descriptive Geometry
• Dept. of Applied Math.
• Faculty of Electrical Engineering and Computer
  Science,
• VB Technical University of Ostrava, Czech
  Rep.
• http//praks.am.vsb.cz/
• KEG VE Praha 23.3.06

mdg.vsb.cz
am.vsb.cz
www.fei.vsb.cz
2
Outline

• Latent Semantic Indexing (LSI) of Images
• Image Coding
• Document Matrix Coding
• The Comparison of LSI in Matlab tm
• Information Retrieval Results

3
Image Retrieval Results Using Latent Semantic
Indexing
The River Query Image
Fig02 0.9740
Fig06 0.9011
Fig05
4
Image Retrieval Results Using Latent Semantic
Indexing
The Presents Query Image
Fig08 0.9769
Fig09 0.9165
Fig07
5
Image Retrieval Results Using Latent Semantic
Indexing
The Art Query Image
Fig25 0.4792
Fig29 0.4421
Fig30
6
Intelligent Information Retrieval Motivation

• The rapid development of information technologies
  provides users with a simple and easy access to a
  very large amount of data, for instance text
  documents, voice, and images.
• Automated information retrieval using computers
  is not still straightforward. Wide popular
  techniques, which are based on keyword matching,
  are not very efficient for real text databases
  because of polysemy (words having multiple
  meanings), synonymy (multiple words having the
  same meaning) and omnipresent typing errors.
• Moreover, real digital images contain reflex and
  noise, measured data are always weighted by
  measurement errors.

7
Intelligent Information Retrieval and Numerical
Linear Algebra

• The numerical linear algebra is used as a basis
  for the information retrieval in the retrieval
  strategy called Latent Semantic Indexing. LSI can
  be viewed as a variant of a vector space model,
  where the database is represented by the document
  matrix, and a user's query of the database is
  represented by a vector. LSI also contains a
  low-rank approximation of the original document
  matrix via the Singular Value Decomposition (SVD)
  or the other numerical methods. The SVD is used
  as an automatic tool for identification and
  removing redundant information and noise from
  data.
• The next step of LSI involves the computation of
  the similarity coefficients between the filtered
  user's query and filtered document matrix. The
  well-known cosine similarity can be used for a
  similarity modelling.

8
Image Coding

• An image a sequence of pixels
• 3 x 2 pixels image example

reshape()
Fig2
v2
9
Document Matrix Coding

• The document matrix A a sequence of coded images

Fig2
10
An example of similarity modelling using the
cosine similarity
An example of a similarity measure is the cosine
similarity. Here A, Q, B represents the vectors.
Symbols fA and fB denotes the angle between the
vectors A, Q and B, Q, respectively. The vector A
is more similar to Q than vector B, because fA lt
fB. A small angle is equivalent to a large
similarity.
11
Latent Semantic Indexing (LSI)

• D. A. Grossman, O. Frieder, Information
  retrieval Algorithms and heuristics, Kluwer
  Academic Publishers, Second edition (2000)

• 1. Compute the co-ordinates of documents in an
  k-dim space by the partial SVD of A
• U,S,V svds(A,k)
• 2. Compute the co-ordinate of the query vector q
• qc q' U pinv(S) S is diagonal
• 3. Compute the similarity coefficients between
  the query vector and documents
• for i 1n Loop over all documents
• sim(i) (qc V(i,)) / (qc
  V(i,))

12
SVD-free Latent Semantic Indexing

• The Singular Value Decomposition
• A U S VT
• The SVD of real documents is expensive.
• A V U S
• A V S-1 U
• V, S2 eigs(ATA, k)
• partial symmetric eigenproblem solved using a
  Lanczos-based iterative method

13
The comparison of LSI in Matlab tm

• Data-collection SKUDCI from Agricultural Research
  Institute Kromerí, Ltd., Czech Rep. Documents
  (columns) 211
• Keywords (rows) 21 766

Singular values k80
Kromeriz gardens, http//guide.travel.cz/
Pentium III, 512 MB, Debian Linux, Matlab tm
6.1, ARPACK tm
14
Semantic web data collection (Svátek, Labský,
University of Economics, Prague)

• SVD-free LSI Properties of the document
  matrix
• Number of keywords 6300 (Resolution 100 x 63)
• Number of documents 394
• Size in memory 18.938 MB
• LSI processing parameters
• Dimension of the original space 394
• Dimension of the user defined space k 20
• Time for A'A operation (in secs) 9.84 (Compaq
  AP400)
• Results of the Matlab (tm) sparse partial
  eigen-solver eigs() (in secs) 2.36
• The total time for the SVD-free LSI processing
  (in secs) 12.20

15
Cycle Query 1
praks.am.vsb.cz
16
Cycle Query 2
praks.am.vsb.cz
17
Cycle Query 3
praks.am.vsb.cz
18
Pattern Identification
praks.am.vsb.cz
19
Advertisement Query 1
praks.am.vsb.cz
20
Advertisment Query 2
praks.am.vsb.cz
21
Banner Detection 1
praks.am.vsb.cz
22
Banner Detection 2
praks.am.vsb.cz
23
Bike Accessories 1
praks.am.vsb.cz
24
Bike Accessories 2
praks.am.vsb.cz
25
Reference

• Svátek V., Labský M., Praks, P., váb O.
  Information extraction from HTML product
  catalogues coupling quantitative and
  knowledge-based approaches. In Dagstuhl Seminar
  on Machine Learning for the Semantic Web. Ed. N.
  Kushmer-ick, F. Ciravegna, A. Doan, C. Knoblock
  and S. Staab, Wadern, Germany, Feb. 1318 2005,
  pg. 1-5. Also available at http//www.smi.ucd.ie/D
  agstuhl-MLSW/proceedings/labsky-svatek-praks-svab.
  pdf

26
Human Expert Modelling Using Linear Algebra a
Heavy Industry Case Study

• Images are from the coking plant Mittal Steel
  Ostrava, Czech Republic. The query image is
  situated in the left up corner and it is related
  to the state Coke is pushing out the coking
  furnaces.
• All of the 6 most similar images (except one) are
  related to the same topic.

27
Summary LSI image retrieval results from the
coking plant Mittal Steel Ostrava
LSI processing parameters Dimension of the
original space 71 Dimension of the user defined
space k 8 Time for A'A operation (in secs)
2.388 Results of the Matlab (tm) sparse partial
eigen-solver eigs() (in secs) 0.069 The total
time for the SVD-free LSI processing (in secs)
2.457

• Properties of the document matrix
• Number of keywords 640 x 480 307 200
• Number of documents 71
• Size in memory
• 166.4 MB

28

• The Feasibility of Using
• Special Quantitative Methods
• for Prediction of Currency Crises
• P. Dvorák1, M. Striík2, P. Praks3, P. Pudil1,
  M. umpíková1, O. Leetický1

1 University of Economics Prague, Faculty of
Management, Czech Republic (dvora-pa_at_fm.vse.cz) 2
VB Technical University of Ostrava, Faculty of
Safety Engineering, Czech Republic
(michal.strizik_at_vsb.cz) 3 VB Technical
University of Ostrava, Faculty of Electrical
Engineering and Computer Science, Czech Republic
(pavel.praks_at_vsb.cz)
29
Hypothesis
By means of the LSI method or the statistical
pattern recognition (using an automatic decisive
rule based on the learning process), applied on
macroeconomic data characterizing the economics
of pre-crisis and tranquil period, we can perform
safe prediction of the financial crises.
Can we distinguish pre-crisis and tranquil
economic period?
30
Data issues (1)

• Currency crisis definition (Kent Osband and C.
  van Rijckeghem)
• Monthly depreciation
• exceeds 10
• exceeds the monthly average depreciation 314
  months prior to the crisis 2xSD of the rate of
  depreciation over the preceding 2 years
• Samples (IMF Kent Osband and Caroline van
  Rijckeghem)
• Industrial markets 54 tranquil periods
• 11 pre-crisis periods
  
• Emerging markets 47 tranquil
  periods
• 13 pre-crisis periods
• Macro-aggregates
• Gross reserves / M2Y Budget balance / GDP
  Current account / GDP
• Growth of exports of goods and services Growth
  in real GDP
• Foreign direct investment and 3 others from
  13th-24th month before the crisis

31
Data issues (2)
32
Input Data for Latent Semantic Indexing

• Total 125 countries
• The each economy (pattern) is represented by 9
  macro-aggregates
• The each macro-aggregate is represented by 12
  numbers (time-series for 1 year)
• -gt Each economy is represented by an vector in
  108 dimensional space (9 x 12 108)
• Data are represented by an 108 x 125 matrix

33
Latent semantic indexing
Query (query vector) Vector consists of time
trends referring to 1) pre-crisis period 2)
tranquil period for countries which belong to
the group of a) emerging markets b)
industrial markets.
Term document matrix
Economies
Monthly records of macro-aggregates used
34
Latent semantic indexing
Economic situations the most and the least
relevant to the query (the query  economy of
Argentina in the period from May 1992 to April
1993) querytranquil period of emerging markets
35
Latent semantic indexing
Economic situations the most and the least
relevant to the query (the query  economy of
Sweden in the period from September 1979 to
August 1980) querytranquil period of industrial
markets
36
Latent semantic indexing

• Conclusion The LSI method (k10) searched the
  economies mainly according their pertinence to
  the market group regardless the fact if it was a
  pattern of tranquil or pre-crisis period.
• The latent semantic indexing method can be used
  for differentiation of emerging markets economics
  from the industrial markets economics and to
  offer supporting information for the
  vulnerability estimation of specific economy in
  the context of the financial crises. Based on the
  macro-aggregates investigated, it was impossible
  to distinguish the pre-crisis economic periods
  sufficiently from the tranquil ones.

37
Singular Value Decomposition

• From Wikipedia, the free encyclopedia
• In linear algebra singular value decomposition is
  an important factorization of a rectangular real
  or complex matrix, with several applications in
  signal processing and statistics.

38
Singular Value Decomposition

• Let A is a mn real matrix, mgtn
• A USV T ,
• where U is an mn matrix and V is an nn square
  matrix both of which have orthogonal columns so
  that
• UTU VTVI.
• The columns of matrices U and V are called the
  left singular vectors and the right singular
  vectors respectively.
• S is an nn diagonal matrix with nonnegative
  diagonal elements called the singular values. The
  decomposition can be computed so that the
  singular values are sorted in non-increasing
  order.

39
SVD and statistics a Matlab example (1)

• A -3 -2 -1  0  0  1  2  3            0 
  0  0 -1  1  0  0  0
• plot(A(1,),A(2,),'o')
• grid on axis(-6 6 -6 6)
• print('-dpng','-r200','data1.png')
• Direction of maximum
• variance (-1, 0)

40
SVD and statistics a Matlab example (2)
A USV T

• Let us compute stddev for first row
• std1rowstd(A(1,),1)1.8708
• And for second row
• std2rowstd(A(2,),1)0.5000
• Let us assume the data transformation
• BA/sqrt(size(A,2))
• What are the singular values of SVD?
• U,S,Vsvds(B,2)
• S     1.8708         0         0    0.5000

41
SVD and statistics a relationship between left
singular vectors and direction of variance
A USV T

• And what about U and VT of SVD?
• U,S,Vsvds(B,2)
• U    -1.0000   0.0000   0.0000    1.0000
• plot(V(,1)',V(,2)','o')
• print('-dpng','-r200','V1.png')

The direction of maximum variation is in the
first column of U. The maximum singular value
measures the variation in this direction i.e. it
is the standard deviation in this direction.
42
Iris Recognition ProblemUsing Latent Semantic
Indexing

• Collection of Irides of two persons
• Number of keywords 768 x 576 442 368
• Dimension of the user defined space k 4
• Time for ATA operation 7.32 secs
• Results of eigen-solver ARPACKtm 3.45 secs 
• Dobe M, Machala L. The database of Iris Images.
  Palacký Univ., Olomouc, Czech Republic 2004,
• http//phoenix.inf.upol.cz/iris

43
Iris Recognition Results (1)
44
Iris Recognition Results (2)
45
The process of the iris recognition for the
large-scale iris recognition example
46
The large-scale iris recognition (1)

• The collection of 384 digital images of irides is
  connected with 64 persons
• Resolution 768 x 576, 24 bits, 488 MB of data
• Very simple preprocessing of images
• Capturing of relevant parts of images

47
The large-scale iris recognition (2)

• LSI processing parameters
• Dimension of the original space 384
• Dimension of the user defined space k 10
• Time for A'A operation (in secs) 13.906
• Results of the Matlab (tm) sparse partial
  eigen-solver eigs() (in secs) 4.843
• The total time for the SVD-free LSI processing
  (in secs) 18.749

• Properties of the document matrix
• Number of keywords 52488
• Number of documents 384
• Size in memory
• 157 464 Kbytes
• Dobe M, Machala L. The database of Iris Images.
  Palacký Univ., Olomouc, Czech Republic 2004
• http//phoenix.inf.upol.cz/iris

48
The large-scale iris recognition (3)

• The histogram of the SVD-Free LSI similarity
  coeff.
• (Total 384 unique images)

A) For the same irides The total of comparing
384 x 2 768
B) For the other irides The total of
comparing 146 304
49
The large-scale iris recognition (4)

• Results of the Large-Scale Database Query

• The quality of the recognition is usually
  quantified by two types of errors
• The False Reject Rate (FRR) represents the
  percentage of the authentic objects that were
  rejected.
• The False Accept Rate (FAR) represents the
  percentage of the impostors that were accepted.
• FRR and FAR generally depend on the given
  similarity coefficient threshold ?. The lower the
  threshold ? the higher is the probability of
  accepting the impostor.

50
The large-scale iris recognition (5)

• Similarly, the higher the threshold ? the higher
  is the probability of rejecting the authentic
  object. Such dependences can be expressed
  graphically by the DET curve.
• Aim ?optmin(FAR(?)FRR(?))
• FAR1.12, FRR 2.34
• The Total Error of Identification 3.46
• ?opt 0.766

51
Conclusion
reshape()

• An raster image is represented as a sequence of
  pixels
• Information retrieval is powered by the SVD-Free
  Latent Semantic Indexing (LSI)
• Simple preprocessing of images, no a priori
  information.
• Partial symmetric eigenproblem is computed using
  a Lanczos-based iterative method (ARPACK tm)
• The threshold optimization was powered by
  DET-Curve Plotting software for use with MATLAB,
  http//www.nist.gov/speech/tools National
  Institute of Standards and Technology
• Special thanks to Zlatko Drmac (Croatia) and
  William Ferng (U.S.A.) for proposing the scaling
  of the document matrix

52
Selected references (1)

• 1 Muller N., Magaia L., Herbst B. M. Singular
  Value Decomposition, Eigenfaces, and 3D
  Reconstructions. SIAM REVIEW Vol. 46, No. 3,
  (2004), pp. 518545
• 2 P.Praks, J.Dvorský, V.Snáel Latent Semantic
  Indexing for Image Retrieval Systems. SIAM
  Conference on Applied Linear Algebra (LA03). The
  College of William and Mary, Williamsburg, U.S.A
  (2003).
• http//www.siam.org/meetings/la03/proceedings/
  
• 3 P.Praks, J.Dvorský, V.Snáel, J. Cernohorský
  On SVD-free Latent Semantic Indexing for Image
  Retrieval for application in a hard industrial
  environment. IEEE ICIT'03, Maribor, Slovenia.
• 4 Praus P., Praks P. Information retrieval in
  hydrochemical data using the latent semantic
  indexing approach. Journal of Hydroinformatics.
  2006. IWA Publishing, London, UK, ISSN
  1464-7141. In print.

53
Selected references (2)

• 4 Praks P., Machala L., Snáel V. Iris
  Recognition Using the SVD-Free Latent Semantic
  Indexing. MDM/KDD2004 - 5th International
  Workshop on Multimedia Data Mining "Mining
  Integrated Media and Complex Data" in conjunction
  with KDD'2004 - The 10th ACM SIGKDD International
  Conference on Knowledge Discovery Data Mining
  pg. 67-71, August 22, 2004, Seattle, WA, USA.
• 5 Labský M., Svátek V., váb O., Praks P.,
  Krátký M., Snáel V. Information Extraction from
  HTML Product Catalogues from Source Code and
  Images to RDF. The 2005 IEEE/WIC/ACM
  International Conference on Web Intelligence
  Campiegne Univ. of Technology, France, Sep. 19-22
  2005 pp. 401-404, ISBN 0-7695-2415-X. Published
  by IEEE Computer Society Washington, DC, USA
  Also at http//rainbow.vse.cz/wi05fi.pdf
• 6 Praks P., Machala L., Snáel V. On SVD-free
  Latent Semantic Indexing for iris recognition of
  large databases. In V. A. Petrushin and L. Khan
  (Eds.) Multimedia Data mining and Knowledge
  Discovery (Part V, Chapter 24). Springer Verlag.
  In print (2006) invited contribution

54
Selected citations

• Praks P., Dvorský, J., Snáel, V. Latent
  semantic indexing for image retrieval systems.
  In Proceedings of the SIAM Conference on Applied
  Linear Algebra (LA03), Williamsburg, USA, The
  College of William and Mary (2003)
• Kosinov S., Marchand-Maillet S. Overview of
  Approaches to Semantic Augmentation of Multimedia
  Databases for Efficient Access and Content
  Retrieval. Revised Selected And Invited Papers by
  Nurnberger and M. Detyniecki (Eds.) AMR 2003,
  Hamburg, Germany, September 15-16, 2003, LNCS
  vol. 3094, pp. 19-36, Springer-Verlag 2004, ISBN
  3540221638
• Kosinov S., Marchand-Maillet S. Hierarchical
  Ensemble Learning For Multimedia Categorization
  And Autoannotation. In Proceedings of the 2004
  IEEE Signal Processing Society Workshop (MLSP
  2004), pp. 645-654, Sao Luís, Brazil, 2004.
• Srovnal V., Bernatík R., Horák B., Snáel V.
  Strategy Extraction for Mobile Embedded Control
  Systems Apply the Multi-Agent Technology. In ICCS
  2004 proceedings of Computational Science, LNCS
  3038. Springer-Verlag Berlin Heidelberg 2004,BRD.
  pp. 631-637, ISBN -540-22116-6
• Praus P.SVD-based principal component analysis
  of geochemical data. Central European Journal of
  Chemistry, 3 (2005) 731-741
• Praus P., Praks P. Information retrieval in
  hydrochemical data using the latent semantic
  indexing approach. Journal of Hydroinformatics.
  2006. IWA Publishing, London, UK, ISSN
  1464-7141. In print.
• Elghazel H., Idrissi K., Ben Amar C., Baskurt A.
  Approches textuelles pour la recherche d'images.
  In Proceedings of the 2005 IEEE SETIT 2005,
  Sousse, Tunisia, March 2005