Content-Based Image Retrieval

1
Content-Based Image Retrieval
2
What is Content-based Image Retrieval (CBIR)?

• Image Search Systems that search for images by
  image content
• lt-gt Keyword-based Image/Video Retrieval (ex.
  Google Image Search, YouTube)

3
How does CBIR work ?

• Extract Features from Images
• Let the user do Query
• Query by Sketch
• Query by Keywords
• Query by Example
• Refine the result by Relevance Feedback
• Give feedback to the previous result

4
Query by Example

• Pick example images, then ask the system to
  retrieve similar images.

What does similar mean?
5
Relevance Feedback

• User gives a feedback to the query results
• System recalculates feature weights

Initial sample
6
Two Classes of CBIRNarrow vs. Broad Domain

• Narrow
• Medical Imagery Retrieval
• Finger Print Retrieval
• Satellite Imagery Retrieval
• Broad
• Photo Collections
• Internet

7
The Architecture of a typical CBIR System
8
The Retrieval Process of a typical CBIR System
Feature Database
Image Database
Query Image
9
Basic Components of CBIR

• Feature Extraction
• Data indexing
• Query and feedback processing

10
How Images are represented
11
Image Features

• Representing the Images
• Segmentation
• Low Level Features
• Color
• Texture
• Shape

12
Image Features

• Information about color or texture or shape which
  are extracted from an image are known as image
  features
• Also a low-level features
• Red, sandy
• As opposed to high level features or concepts
• Beaches, mountains, happy

13
Global features

• Averages across whole image
• Tends to loose distinction between foreground and
  background
• Poorly reflects human understanding of images
• Computationally simple
• A number of successful systems have been built
  using global image features

14
Local Features

• Segment images into parts
• Two sorts
• Tile Based
• Region based

15
Regioning and Tiling Schemes
Tiles
Regions
16
Tiling

• Break image down into simple geometric shapes
• Similar Problems to Global
• Plus dangers of breaking up significant objects
• Computational Simple
• Some Schemes seem to work well in practice

17
Regioning

• Break Image down into visually coherent areas
• Can identify meaningful areas and objects
• Computationally intensive
• Unreliable

18
Color

• Produce a color signature for region/whole image
• Typically done using color correllograms or color
  histograms

19
Color Features

• Color Histograms
• Color Space Selection
• Color Space Quantization
• Color Histogram Calculation
• Feature Indexing
• Similarity Measures
• Color Layout
• Histograms based on spatial distribution of
  single color
• Histograms based on spatial distribution of color
  pair
• Histograms based on spatial distribution of color
  triple
• Other Color Features
• Color Moments
• Color Sets

20
Color Space Selection

• Which Color Space?
• RGB, CMY, YCrCb, CIE, YIQ, HLS,
• HSV?
• Designed to be similar to human perception

21
HSV Color Space

• H (Hue)
• Dominant color (spectral)
• S (Saturation)
• Amount of white
• V (Value)
• Brightness

How to Use This?
22
Content Based Image Retrieval

• CBIR
• utilizes unique features (shape, color, texture)
  of images
• Users prefer
• To retrieve relevant image by semantic categories
• But, CBIR can not capture high-level semantics in
  users mind

23
Relevance Feedback

• Relevance Feedback
• Learns the associations between high-level
  semantics and low-level features
• Relevance Feedback Phase
• User identifies relevant images within the
  returned set
• System utilizes user feedback in the next round
• To modify the query (to retrieve better results)
• This process repeats ? until user is satisfied

24
(No Transcript)
25
Now, We have many features (too many?)

• How to express visual similarity with these
  features?

26
Visual Similarity ?

• Similarity is Subjective and Context-dependent.
• Similarity is High-level Concept.
• Cars, Flowers,
• But, our features are Low-level features.
• Semantic Gap!

27
Which features are most important?

• Not all features are always important.
• Similarity measure is always changing
• The system has to weight features on the fly.
• How ?

28
Q A