Selected Advanced Topics

1
Selected Advanced Topics
Storing and Retrieving Images Content-based
Image/Video Indexing and Retrieval
2
Problem
Find all images contain horses ..
3
Text-based technology

• Annotation Each image is indexed with a set of
  relevant text phrases, e.g.,
• Retrieval based on text search technology

Appropriate phrases to describe the content of
this image include Mother, Child, Vegetable,
Yellow, Green, Purple .
4
Text-based technology - Drawbacks

• Annotation - subjective
• different people may use different phrases to
  describe the same or very similar image/content

5
Text-based technology - Drawbacks

• Annotation - Laborious
• It will take a lot of man-hours to label large
  image/video databases with 1m items

6
Content-based Technology
Using Visual Examples
Image/Video Database
7
Content-based Technology
Using Visual Features
Image/Video Database
8
Content-based Technology

• Content-based image indexing and retrieval
  (CBIR), is an image database management
  technique, which indexes the data items (images,
  or video clips) using visual features (e.g.,
  color, shape, and texture) of the images or video
  clips.
• A CBIR system lets users find pictorial
  information in large image and video databases
  based on visual cues, such as colour, shape,
  texture, and sketches.

9
Content-based Technology

• The visual features, computed using image
  processing and computer vision techniques are
  used to represent the image contents numerically.
• Image Content - a high level concept, e.g., this
  image is a sunset scene, a landscape scene, etc.
• Numerical Content Representations - Low level
  numbers, often the same set of numbers can come
  from very different images, making the task very
  hard!

10
Content-based Technology

• Techniques for Computing Visual
  Features/Representing Image Contents
• some are very sophisticated, and many are still
  not matured
• hence the computational processes
• in some cases are automatic
• but in other cases are semi-automatic
• in the most difficult cases, it may have to be
  done manually

11
Content-based Technology

• Comparing Image Content/Retrieving Images based
  on Content
• Simple approaches - compute the metric distance
  between low level numerical representations
• Advanced Approaches - using sophisticated pattern
  recognition, artificial intelligence, neural
  networks, and interactive (relevant feed-back)
  techniques to compare the visual content (low
  level numerical features)

12
Content-based Technology - IBM QBIC System

• The IBMs QBIC (Query by Image and Video Content)
  system is one of the early examples of CBIR
  system developed in 1990s.
• The system lets users find pictorial information
  in large image and video databases based on
  color, shape, texture, and sketches.

13
Content-based Technology - IBM QBIC System

• The User Interfaces Module
• Let user specify visual query by drawing,
  selecting from a color wheel, selecting a sample
  image
• Display results as an ordered set of images
• The Database Population and Database Query
  Modules
• Database population - process images and video to
  extract features describing their content -
  colors, textures, shapes and camera and object
  motion, and store the features in a database
• Database Query - let user compose a query
  graphically, extract features from the graphical
  query, input to a matching engine that finds
  images or video clips with similar features

14
Content-based Technology - IBM QBIC System

• The Data Model
• Still image, or scene - full image
• Objects contained in the full image - subsets of
  an image
• Videos - broken into clips called shots - sets of
  contiguous frames
• Representative frames, the r-frames, are
  generated for each shot
• R-frames are treated as still image - from which
  features are extracted and stored in the
  database.
• Further processing of shots generates motion
  objects - e.g., a car moving across the screen.

15
Content-based Technology - IBM QBIC System

• Queries are allowed on
• Objects - e.g., Find images with a red round
  object
• Scenes - e.g., Find images that have
  approximately 30 red and 15 blue colors
• Shots - e.g., Find all shots panning from left to
  right
• A combination of above - e.g., Find images that
  have 30 red and contain a blue textured objects

16
Content-based Technology - IBM QBIC System

• Similarity Measures
• Similarity queries are done against the database
  of pre-computed features using distance functions
  between the features
• Examples include, Euclidean distance, City-block
  distance, .
• These distance functions are intended to mimic
  human perception to approximate a perceptual
  ordering of the database
• But, it is often the case that a distance metric
  in a feature space will bear little relevance to
  perceptual similarity.

17
Content-based Technology - Basic Architecture
Similarity Measures
Imagery
Meta data
Query
Image Database
18
Colour - An effective Visual Cue
Colors can be a more powerful visual cue than you
initially thought!
19
Colour - An effective Visual Cue
In many cases, color can be very
effective. Here is an example
Results of content-based image retrieval using
4096-bin color histograms
20
Colour Spaces
Colour Models RGB Model This colour model uses
the three NTSC primary colours to describe a
colour within a colour image.
Sometimes in Computer Vision, it is convenient to
use rg chromaticity space r R/(RGB) g
G/(RGB)
21
Colour Spaces
YIQ Model The YIQ models is used in commercial
colour TV broadcasting, which is a re-coding of
RGB for transmission efficiency and for
maintaining compatibility with monochrome TV
standard.
In YIQ, the luminance (Y) and colour information
(I and Q) are de-coupled.
YCbCr Model Y 0.299R 0.587G 0.114B Cb
-0.169R - 0.331G 0.500B Cr 0.500R -
0.419G - 0.081B
22
Perceived Color Differences

• One problem with the RGB colour system is that
  colorimetric distances between the individual
  colours don't correspond to perceived colour
  differences.
• For example, in the chromaticity diagram, a
  difference between green and greenish-yellow is
  relatively large, whereas the distance
  distinguishing blue and red is quite small.

r R/(RGB) g G/(RGB)
23
CIELAB

• CIE (Commission Internationale de l'Eclairage)
  solved this problem in 1976 with the development
  of the Lab colour space. A three-dimensional
  color space was the result. In this model, the
  color differences which you perceive correspond
  to distances when measured colorimetrically. The
  a axis extends from green (-a) to red (a) and
  the b axis from blue (-b) to yellow (b). The
  brightness (L) increases from the bottom to the
  top of the three-dimensional model.
• With CIELAB what you see is what you get (in
  theory at least).

24
Colour Histogram

• Given a discrete colour space defined by some
  colour axes (e.g., red, green, blue), the colour
  histogram is obtained by discretizing the image
  colours and counting the number of times each
  discrete colour occurs in the image.
• The image colours that are transformed to a
  common discrete colour are usefully thought of as
  being in the same 3D histogram bin centered at
  that colour.

25
Colour Histogram Construction

• Step 1
• Colour quantization (discretizing the image
  colours)
• Step 2
• Count the number of times each discrete colour
  occurs in the image.

26
Colour Quantization

• A true colour, 24-bit/pixel image (8 bit - R, 8
  bit - G, 8 bit -B), will have 224 16777216 bins
  !
• That is, each image will have to be represented
  by over 16 million numbers
• computationally impossible
• in practice not necessary
• Colour quantization - reduce the number of
  (colours) bins

27
Simple Colour Quantization

• Simple Colour Quantization (Non-adaptive)
• Divide each colour axis into equal length
  sections (different axis can be divided
  differently).
• Map (quantize) each colour into its corresponding
  bin

28
Simple Colour Quantization
Example In RGB space, quantize each image colour
into one of 8x8x8 512 colour bins
Colour Bin Colour Bin (123,23,45) (3, 0, 1
) (122, 28, 46) (3, 0, 2) (132, 29,50) (4,
0, 1) (122, 172, 27) (3, 5, 0) (121,26,48)
(x, x, x) (142, 28, 46) (x, x, x)
29
Advanced Colour Quantization

• Adaptive Colour quantization (Not required)
• Vector Quantization
• K-means clustering
• K representative colours
• The colour histogram consists of K bins, each
  corresponding to one of the representative
  colours.
• A pixels is classified as belonging to the nth
  bin if the nth representative colour is the one
  (amongst all the representative colours) that is
  closest to the pixel.

A pixel is a point in the 3D colour space
B
G
R
Representative colours
30
Colour Histogram Construction - An Example

• A 3 x 3, 24-bit/pixel image has following RGB
  planes
• Construct an 8-bin colour histogram (using simple
  colour quantization, treating each axis as
  equally important).

Green 213 24 77 11 232 239 22 12 12
Blue 23 24 77 12 24 69 22 123 123
Red 23 24 77 11 24 69 22 12 12
Bin (0,0,0) Bin (0,0,1) Bin (0,1,0) Bin
(0,1,1) Bin (1,0,0) Bin (1,0,1) Bin
(1,1,0) Bin (1,1,1)
31
Colour Histogram Construction - An Example

• Quantized Colour Planes
• Count the number of times each discrete colour
  occurs in the image.

Green 1 0 0 0 1 1 0 0 0
Blue 0 0 0 0 0 0 0 0 0
Red 0 0 0 0 0 0 0 0 0
Bin (0,0,0) 6 Bin (0,0,1) 0 Bin (0,1,0)
3 Bin (0,1,1) 0 Bin (1,0,0) 0 Bin (1,0,1)
0 Bin (1,1,0) 0 Bin (1,1,1) 0
32
Colour Based Image Indexing
The histogram of colours in an image defines the
image colour distribution
33
Colour based Image Retrieval
Images are similar if their histograms are
similar!
Colour Distribution

(10,0,0,0,100,10,30,0,0)
Dissimilar
Similar!
Colour Distribution

(0,40,0,0,0,0,0,0,110,0)
34
Formalizing Similarity
1
2
Similarity(Image 1, Image 2) D (H1, H2) where
D( ) is a distance measure between vectors
(histograms) H1 and H2
35
Metric Distances
A distance measure D( ) is a good measure if it
is a metric!
D(a,b) is a metric if
D(a,a) 0 (the distance from a to itself is 0
D(a,b) D(b,a) (the distance from a to b
distance from b to a)
D(a,c) lt D(a,b) D(b,c) ( triangle inequality
the straight line distance is always the
least! )
D(a,b) D(b,c) should be no smaller than D(a,c)
36
Common Metric Distance measures
Histogram Intersection, HI
H1 (10, 0, 0, 0, 100, 10, 30, 0, 0)
H2 ( 0, 40, 0, 0, 0, 6, 0, 110, 0)
Similarity HI(H1, H2) 0 0 0 0 0 6
0 0 6
37
Common Metric Distance measures
Euclidean or straight-line distance or L2-norm,
D2
H1 (10, 0, 0)
H2 ( 0, 40, 0)
Similarity D2(H1, H2) sqrt(100 1600 0)
41.23
38
Common Metric Distance measures
Manhattan or city-block or L1-norm, D1
H1 (10, 0, 0)
H2 ( 0, 40, 0)
Similarity D1(H1, H2) (10 40 0) 50
39
Histogram Intersection vs City Block Distance
Theorem if H1 and H2 are colour histograms and
the total count in each is N (there are N-pixels
in an image) then
(Histogram Intersection inversely proportional to
a metric distance!)
Proof
(by definition)
(1)
(2)
40
Histogram Intersection vs City Block Distance
(3)
Substituting (2) and (3) in (1)
(4)
(5)
41
Colour Histogram Database
42
How well does Color histogram intersection work ?
66 test histograms in the database
Swain Original Test
31 query images
Recognition rate almost 100
Indeed, because color indexing worked so well it
is at he heart of almost all image database
systems
43
Google Image Search
44
Google Image Search
After clicking this colour patch
45
Problems with color histogram matching
1. Color Quantization problem

Colour Distribution
(0,40,0,0,0,0,0,110,0)
Because, the two images have slightly different
color distributions their histograms
have nothing in common!
0 intersection!

Colour Distribution
(0,0,40,0,0,0,0,0,110)
Sources of quantization error noise,
illumination, camera
46
Problems with color histogram matching
2. The resolution of a color histogram

Colour Distribution
(0,40,0,0,0 ,0,0,110,0)
For the best results, Swain quantized
colour space into 4096 distinct colours gt Each
colour distribution is a 4096-dimensional vector.
gt Histogram intersection costs O(4096)
operations (some constant 4096)
4096 comparisons per database histogram gt
histogram intersection will be very slow for
large databases
Many newer methods work well using 8 - 64 D
features
47
Problems with color histogram matching
3. The colour of the light
Under a yellowish light all image colours
are more yellow than they ought to be
48
Problems with color histogram matching
4. The structure of colour distribution

• All four images have the same color distribution
  - need to take into account spatial
  relationships!

49
Problem solution gt Use statistical moments
1st order statistics
2nd order statistics
50
Statistical similarity
Colour Distribution

(50,50,50)
Compare mean RGBs (In general compare all
statistical measures)

Colour Distribution
(20,70,40)
Statistical similarity
(Euclidean distance between corresponding
statistical measures)
51
Histogram vs Statistical Similarity
Completeness of representation
Sensitivity to Quantization error
params/ Match speed
histogram
Many/slow
complete
sensitive
Low order stats
Few/fast
incomplete
insensitive
Low and high order stats
complete (or over complete)
Many/moderate
sensitive
52
Advanced Topics

• Fast Indexing
• Interactive/Relevant Feedback
• Reducing the Semantic Gap
• Visualization, Navigation, Browsing
• Internet scale image/video retrieval
• Flickr billions of photos
• Youtube billions of videos