Learning in ContentBased Image Retrieval

1
Learning in Content-Based Image Retrieval
2
Outline

• Introduction
• Small sample Learning Issue
• Word Association via Relevance Feedback
• ImageGrouper
• Use Unlabeled Data
• Conclusion

3
Introduction

• In Content-based image retrieval (CBIR)

• Small Sample learning issue

• There is a gap between high-level semantics in
  human minds and low-level feature computable
  by machines
• A major difficulty for learning during relevance
  feedback is relatively small number of
  training samples available from users.

• Word Association

• Some of image databases have limited keyword or
  key-pharse annotations. It is desirable to
  seamlessly incorporate low-level features and
  high level semantic keywords.

4
Introduction

• ImageGrouper

• An interface design for users that allow to
  select and group examples from mutiple rounds
  of interactions , and make overlapping
  annotations to different groups of image

• Using Unlabeled Data

• use unlabeled data for enhancing classification
  accuracy, learning process and improve
  classification results.

5
Small sample Learning Issue

• Relevance feedback problem

• Positive Examples
• Negative Examples
• (they can belong to any class)

6
Small sample Learning Issue
Positive Examples
Negative Examples
Negative Examples
X
Negative Examples
Negative Examples
7
Biased Discriminant Analysis (BDA)

8
Nonlinear BDA using Kernel
9
Nonlinear BDA using Kernel
10
Result
Visual features color moments , wavelet
moments, water-filling structural features.
11
Word Association via Relevance Feedback (WARF)

• The performance of CBIR is inherently constrained
  by use of the low level features.
• Word Association Via relevance Feedback (WARF)
  add the unification of keywords to improve the
  retrieval performance.

12
WARF
CBIR
Low LevelFeatures
Improved Retrieval Result
Texture Annotations
(Keywords spotting from text such as HTML text on
webpages or manual annotations)
13
WARF
14
ImageGrouper

• The more query examples are available ,the better
  result we can get
• Refine the search adding example images.
• Additional examples may contain undesired
  features and degenerate the retrieval performance

15
ImageGrouper

• Traditional interface does not allow the user to
  put aside the query results for later uses.
• Because of users of CBIR are not necessarily
  looking for only one type of image. Interest may
  change during retrieval

16
ImageGrouper
17
ImageGrouper

• Left Pane ResultViews the result of both
  Content-based and Keyword-based search.
• Right Pane GroupPaletteuser handles image
  groups and specifty it as relevant, irrelevant or
  neutral.( Neutral do not contribute to the
  search at the moment )

18
ImageGrouper

• Compare with the traditional GUI for CBIR.
  ImageGrouper is much easier to try different
  combinateion of query examples.
• It provides storage area for images that are not
  used for query at the moment.

19
ImageGrouper
Example

• Assume the user is looking for red car
• She can initially collect car with any color.
• Once she found enough number, she can divide them
  into 2 groupsred cars ( set as relevant )other
  cars ( irrelevant )
• Furthermore, the user can annotate text
  information to each group , this text can be used
  for keyword-based search

20
Using Unlabeled Data

• Expectation-Maximization (EM)

• E-step
• M-step
• Loop

21
Using Unlabeled Data

• Discriminant Expectation-Maximization (D-EM)

• E-step
• D-step
• M-step
• Loop

22
Using Unlabeled Data
23
Using Unlabeled Data
24
Using Unlabeled Data