You Look Like Risu: Two Approaches to an Image Similarity Function

1
You Look Like Risu Two Approaches to an Image
Similarity Function

• Brendan Elliott Casey Kretschman
• bxe7_at_cwru.edu cjk21_at_cwru.edu

2
Image Similarity

• Human Perception is final arbiter
• Query by image content
• Methods based on color, shape, texture and
  metadata
• Query by keyword
• Keywords assigned by human input
• Information represented in high dimensionality

3
The M-tree

• Dynamic distanced based index structure much like
  Btree
• Balanced
• Bottom Up construction
• Supports insert and delete
• Creates nodes of data like GNAT
• Nodes may overlap
• Stores distance of parents
• Supports range and kNN queries

4
The M-tree

• Split
• Nodes overflow in dynamic structures
• Split data between overflow and new node
• Pass pointer up tree
• Ideally splits would have min overlap and min
  volume
• Execution is expensive
• Must also select a routing object to promote
• Routing objects are pointed to by the parent and
  are the center of the node

5
The M-tree

• Promotion heuristics
• m_RAD minimum RADii
• Calculate all possible pairs and covering radii
• Promotes pair with minimum combined covering
  radii
• Optimal, but most expensive
• mM_RAD
• Similar, but promotes pair with minimum maximum
  radii

6
The M-tree

• Promotion heuristics
• M_LB_DIST Maximum Lower Bound on Distance
• Only uses stored distances to find farthest entry
  from nodes routing object
• Calculates by triangle inequality
• RANDOM selects randomly
• Used as reference to compare other policies

7
The M-tree

• Promotion heuristics
• SAMPLING set of RANDOM results
• Select the best RANDOM result
• Analysis
• Low node capacity favors Random and M_LB_DIST
• Higher node capacity favor more complex heuristics

8
The M-tree

• Split heuristics
• Generalized Hyperplane Same as weve seen in
  GNAT
• Balanced Each node contains the same number of
  objects
• Routing objects take turns selecting their
  nearest neighbors
• Select based on data distribution

9
Implementation Environment

• RisuPicWeb 3.0 (http//risukun.com/Node.aspx)
• Provide User with k-NN searching
• C/ASP.NET front end
• SQL2000 back end with 26,000 images
• M-tree written in C, wrapped to be used in .NET
  environment
• Involved a few modifications to meet our needs
  and for more robust error handling

10
RGBAvg Metric Definition

• Based off davg by Faloutsos et al.
• Averages RGB of all pixels in an image
• Maps these to 3 dimensional space
• No information based on pixel position

11
RGBAvg Metric Definition

• RGB does not map to human perception
• Easily verified by human perception
• Selected as a bad method to compare CSH metric
  to

12
RGBAvg L value

• Distance function used in paper is standard L2
  metric distance
• Experiment on L value
• Use L 1, 2, 3, 10
• Hope to find different list of results, and then
  select the one closest to human perception
• Also hope to evaluate performance

13
RGBAvg Data Distribution
L 2
14
RGBAvg Construction
15
RGBAvg Query Efficiency
16
RGBAvg Results

• See Handout

17
RGBAvg Analysis

• L2 is the most efficient
• 2nd best construction, best query
• Query results all are about equal

18
CSH Metrics Overview

• Based off Color-Shape by Stehling et al.
• Contains information about color, as well as
  color position
• Overcomes RGB to human perception and average
  color drawbacks of RGBAvg

19
CSH Image Ratio Metric Definition

• Create histograms for each image
• Histograms cover RGB color-space
• Partition images into cells
• Partitions equidistance and the same number
  vertically and horizontally
• Stehling only required non-overlapping cells
• Create 2d matrix holding the ratio of pixels of
  that color in cell to pixels of that color in the
  image
• Percentage of that bucket represented in that
  cell

20
CSH Image Ratio Distance

• Stehlings distance function
• Our distance function
• where
• She, hd) distance between hq, hd, i number of
  cells per image, j number of colors available,
  hij value of the jth cell of the ith color
  of an image, aqi adi sum of the areas of
  the regions described by each histogram wi
  min(aqi, adi)

21
CSH Color Correlation Matrix

• CCM idea presented by Faloutsos et al.
• Applied to davg for dhist
• Increased complexity distance function
• davg presented as filter for dhist metric

22
CSH Color Correlation Matrix

• Correlates colors that humans would consider
  close to each other
• CMxy Euclidian distance between colors x
  and y in RGB color-space

23
CSH CCM Distance Function

• The same image preprocessing can be used as CSH
  Image Ratio, but a new index tree must be
  generated
• Could break symmetry
• Obj1xj CMix Obj2ij !
  Obj1ij Obj2xj CMix
• Use the average of the two terms to keep symmetry

24
CSH CCM Distance Function
25
CSH Noise Reduction

• During preprocessing, many histograms with a
  single cell with a 1 where noticed
• Many where due to color buckets represented by a
  small number of pixels in 1 cell
• Human perception would not put much weight to
  these few pixels, so neither should a good metric

26
CSH Impact of Noise

• Compare images that are identical except for
  noise pixels
• Impact for 1 color 1 / cells
• Impact for an image 1 / ( cells colors)
• Impact on results unknown at time, but worth
  investigating

27
CSH Noise Reduction Method 1

• If the number of pixels of a color is low, it
  could be noise
• Lower than the amount of a cell it represents
• Use min( percentage of color in the cell,
  percentage of the cell that is the color)
• If there is a color that is represented in many
  cells, but only represents a small portion of the
  cell, this information will be lost

28
CSH Noise Reduction Method 2

• Set noise thresholds
• If the original value is high, and the ratio of
  color pixels in a cell to total pixels in a cell
  is low, we can identify this as noise
• Set to a predetermined noise value

29
CSH Noise Reduction Method 2

• Not every color histograms value will add to one
• Setting by trial and error is most logical, but
  time prohibitive
• Would require new preprocessing for each trial

30
CSH Noise Reduction Method 3

• Change the way value is computed
• Was percentage of the images color in a cell
• Change to percentage of the cell color
• Each color histogram will still not add up to 1
• But each cell across the histograms will add to 1

31
CSH Cell Ratio Metric

• We select Method 3 as our new metric and name it
  CSH Cell Ratio
• The same distance function as CSH Image Ratio may
  be used
• Different image and tree preprocessing needs to
  be done, but only once
• As opposed to trial and error

32
CSH Cell Ratio with Color Correlation

• CSH Cell Ratio can incorporate the Color
  Correlation Matrix
• Uses the same image preprocess as CSH Cell Ratio,
  but new tree preprocessing
• Uses the same distance function as CSH Image
  Ratio with Color Correlation

33
CSH Data Distributions
34
CSH Construction Costs
Due to Bulk Loading errors, CSH trees where
constructed with less efficient individual
loading, and can not be compared to the RGB
trees The exact information for this was lost
during tree construction
35
CSH Query Efficiency
36
CSH Results

• See handout

37
CSH Image Ratio Dark Returns

• Very few colors match everything, because 0 0
  0

38
CSM Improving Performance

• Increased Fanout
• Tight data distribution leads to a lot of overlap
• Bulk loaded M-tree
• Bulk loaded have been proven to allow for more
  efficient querying than individual loading
• Reason for failure in bulk loading our CSH trees
  unknown

39
Comparison of Metrics

• RGBAvg
• Efficient
• Results are bad
• CSH
• CSH Node Ratio returns better results than CSH
  Image Ratio
• Query is very inefficient
• CSH shows more promise for improvement

40
Further Research Image Subset

• Searching for a section of an image
• Treating the query object as a single cell, and
  all search objects as the target cells
• Obstacles
• Query object spans cells in search object
• Query object present in search object, but larger
  than a single cell

41
Further Research RGBAvg in CSH Metrics

• Crosshatch or old tri-color comic book art is
  represented with small marks that human
  perception takes the average of
• Applying RGBAvg to the cell by cell comparisons
  of CSH may return good results

Custom Crosshatch by Mikael Noguchi
http//www.katode.org/noguchi/web/p_art_crosshatch
.htm
42
Further Research Special Case CCM

• Many objects come in a range of colors
• Leaves Green, Brown, Red, Orange
• Sky Blue, Grey, Black, White
• Leather Black, Brown, Red
• Creating a specialized CCM that ties these colors
  together may allow for better searching by content

43
Further Research Cell Count in CSH

• Will query images with larger, simple shapes
  benefit from fewer cells per image
• Will query images with small, complex shapes
  benefit from fewer cells per image

44
Questions?