Survey of Algorithms to Query Image Databases

1
Survey of Algorithms to Query Image Databases
Image from Kodaks PhotoQuilt
lt-

• COMP 290-72Computational Geometry
• Benjamin Lok 11/2/98

2
Outline of Talk

• Overview of the problem
• Three methods
• Color based
• Shape based
• Vision based
• Conclusions

Image from Microsoft Clip Gallery
3
Problem

• Query an image database
• What does a match mean?
• Application dependent
• Notion of subjectivity
• Sensitivity to noise

4
Problem

• Semantic similarity is still not possible
• ex. All images with cats
• To determine similarity,
• we need to a new
• Metric
• Space

Images from Microsoft Clip Gallery and website
5
Ambiguity
Images from Shrine to Long Haired Men and
Videos of Women Getting Their Heads Shaved
websites
Girl
Guy
Guy
6
The Earth Movers Distance, Multi-Dimensional
Scaling, and Color-Based Image Retrieval
Image from Microsoft Clip Gallery

• Yossi Rubner, Leonidas Guibas, Carlos Tomasi
  (1997)
• Stanford Vision Laboratory

7
Color Signatures

• Utilize the CIE-LAB color space
• Based on human perception of color
• Map each pixel to a point in color space
• Common color values increase weight of point
• Group clusters into points (8-12 per image)

Rubner, Guibas, and Tomasi
8
Earth Movers Distance

• To compare two images, compute the work needed
  to move the cluster points from one image to the
  other

Rubner, Guibas, and Tomasi
9
Earth Movers Distance (cont)

• Solving a linear programming problem
• Given two signatures
• p (p1,wp1),,(pm,wpm) and
• q (q1,wq1),,(qn,wqn)
• Find C where Cij is the amount of weight pi
  matched qj

10
Applications

• Visualize Databases (Queries and Results)
• Scale the multiple dimensions into 2D using MDS
  and minimize STRESS

Rubner, Guibas, and Tomasi
11
Database Visualization
Rubner, Guibas, and Tomasi
12
Algorithm Recap
Image from YenPen Stationary Website

• Map pixels to 3D color space points
• Locate and compress clusters of points
• 8 to 12 points determine the color signature
• Calculate the Earth Movers Distance to determine
  distance between two images

13
Advantages
Disadvantages

• Based on human perception of color
• Some invariance to small change in viewpoint and
  lighting
• Meaningful metric
• Relatively fast
• Can embed multiple metrics

• Depending on application, query format might be
  not be intuitive
• Not much use for non-color images
• False positives a real possibility depending on
  working domain

14
Image from Microsoft Clip Gallery

• Shaped-based Image Retrieval Using Geometric
  Hashing
• Scott D. Cohen and Leonidas J. Guibas
• 1997
• Stanford Vision Laboratory

15
Overview

• Implementation
• Search through 500
• Chinese characters
• Goals
• Provide invariance to scale, rotation, and
  translation
• speed and accuracy

Cohen and Guibas
16
Generating a Illustration

• Illustration - set of curves that summarize an
  image
• Edgel detection
• Medial Axis determination

Cohen and Guibas
17
Approximating with Polylines

• Convert medial axis representation to polylines
• Tradeoff between speed and accuracy

Cohen and Guibas
18
Geometric Hashing
Cohen and Guibas

• Geometric Hashing - method used to compare two
  point sets under some transformation group
• Take each point and use it as the origin of a
  coordinate system

19
Geometric Hashing (cont)
Cohen and Guibas

• If translating P by qj - pi produces a good match
  Ii(P) and Ij(Q) will match.
• This property can be generalized to other
  transformation groups.
• Each line segment is a basis of a coordinate
  system
• Translation, Rotation, and Orientation defined
• I(P) transform all other segments into new CS

20
Notes on GH
Cohen and Guibas

• Each segment will be transformed to 2m Coordinate
  systems
• I(P) stores O(m2) segments
• Can be done as preprocessing step
• Expressing the different possible transformations
  using each segment as a basis

21
Querying the Database

• Query image undergoes the
• feature extraction process
• For each query feature, a nearest-neighbor query
  is applied and the k closest or within some j
• Similarity score increases if database image has
  a feature that is close to the query feature

Cohen and Guibas
22
Closeness

• How do you describe the closeness of two lines?
• Transform to a 4D space made of (l,?,a,b)
• With two (l,?,a,b) descriptions for lines, can
  compute distance
• Divide by standard deviation
• over sample of database
• features

23
Details

• Closeness is relative to database contents
• Nearest-neighbor algorithm by Arya, Mount, et. al
  (1994). Query time for k nearest features is O(k
  log n)

Cohen and Guibas
24
Advantages
Disadvantages

• Fast
• Queries database of 500 characters in 1 second on
  SGI Indy
• Queries based on important features

• Working domain currently limited
• Could get too expensive as complexity in images
  increases

Cohen and Guibas
25

• A Multi-Resolution Technique for Comparing Images
  Using the Hausdorff Distance
• Daniel Huttenlocher and William Rucklidge
• 1992 Cornell University

Huttenlocher and Rucklidge
26
Directed Hausdorff Distance

• Given Aa1, , am and Bb1, , bm
• Identifies the point in A farthest from any point
  in B
• Measures the degree of mismatch between between
  two sets.

27
Properties of Hausdorff Distances

• Not symmetric h(A,B) ! h(B,A)
• Compute kth maximum
• Notion of rank
• Reduces sensitivity
• Fraction of A within
• h(A,B) of B
• Obscured portions
• h(A,B) hypothesis
• h(B,A) test

28
Transformations t( )

• Given A is an image, B is the model
• Without Orientation, if A is in B then A
  undergoes transformation t.
• fB(t)H(t(B),A) t(tx,ty,sx,sy) forward
• fA(t)H(A,t(B)) reverse

29
Bidirectional Hausdorff Distance

• Solve for which values of t,
• the following holds
• Results in searching a four
• dimensional space

30
Restricting Search Space

• Slope of f(t)HLK(A,t(B))
• is linear
• Divide space into cells
• Calculate HLK(A,tc(B))
• Determine a maximum delta per cell
• Based on limit in scale and translation
• Allows for quick rejection and acceptance
• Label cells as interesting or disregard

31
Restricting Search Space

• Create smaller cells from interesting cells
• Bounds based on
• transformations
• Quickly narrow
• down to areas that
• could possibly be
• within ? of A

32
Subtleties

• Discretization useful if working in computer
  vision domain (integers)
• Can compare partially obscured images
• Optimizations
• Early rejection/acceptance
• Pretty slow
• 200 to 250 seconds

Website on submarines
33
Huttenlocher and Rucklidge
34
Advantages
Disadvantages

• Accurate
• Geared towards image processing and vision
• Partially obscured images
• Searches similar to humans

• Slow
• No Orientation
• Database must be specialized
• Potential problems in generating queries

35
Recap

• Three Algorithms
• Color Based
• Color Signatures
• Earth Movers Distance
• Shaped Based
• Polylines
• Transform Invariant Sets
• Vision Based
• Hausdorff Distance
• Subdivision of Transformation Space

www.sportsmanscaps.com
36
Final Thoughts
www.jerryspringer.com

• Algorithms work well in various domains
• Query construction not formalized
• Other methods
• wavelet-based
• texture-based
• object-based
• Took 5 minutes to find Shrine to Men with Long
  Hair and Videos of Women Getting Their Head
  Shaved

All other images generated by author using Paint
Shop Pro