A Novel Logbased Relevance Feedback Technique in Contentbased Image Retrieval

1
A Novel Log-based Relevance Feedback Technique in
Content-based Image Retrieval
ACM Multimedia 2004 12th Annual Conference,
October 10 -16, 2004 New York City, Columbia
University

• Steven Chu-Hong Hoi Michael R. Lyu
• Department of CSE
• The Chinese University of Hong Kong
• Shatin, Hong Kong SAR
• chhoi, lyu_at_cse.cuhk.edu.hk

2
Outline

• Introduction Motivation
• Log-based Relevance Feedback
• Soft Label Support Vector Machine
• Experimental Results
• Conclusions and Future Work

3
Introduction

• Content-based Image Retrieval (CBIR)
• Attract much interest, studied for many years
• An important component in multimedia retrieval
• Query based on low-level visual content color,
  texture, shape, etc.

QBE
Challenge the semantic gap between low-level
features and high-level concepts
4
Introduction

• Relevance Feedback (RF) in CBIR
• A powerful technique, attack the semantic gap
  problem
• Using interactive mechanisms, soliciting users
  interactions, learning users high-level concepts
• Boosting retrieval performance effectively
• Many popular techniques MARS, QEX, MindReader,
  Optimizing learning, SVM (active), Boosting, etc.
  
• Problems
• Regular relevance feedback techniques a lot of
  times of feedback which will cost much time and
  make users boring

5
Motivation
Relevance Feedback
?
Users Feedback Logs
Can users feedback logs information be used to
improve the regular relevance feedback?
Problem
6
LRF Log-based Relevance Feedback

• Problem Formulation
• Construct a Relevance Matrix RM
• Each log session (N ) N
  images are marked relevant
  irrelevant instances
• Values relevant (1), irrelevant (-1), unknown
  (0)

Image samples in the image database
-1
-1
1
1
1
-1
-1
0
1
-1
1
Log Sessions
-1
1
-1
-1
-1
-1
-1
0
-1
1
-1
7
Log-based Relevance Feedback (contd)

• Relationship Measurement
• For each given session k , if the image i is
  marked as relevant (positive) and the image j
  is marked as irrelevant (negative), then the
  elements are represented as
• RM (k, i) 1 and RM (k, j) -1
• For every two images i and j, their relationship
  can be measured by a modified correlation
  function

8
LRF Algorithm

• Collection of training Samples
• Regular relevance feedback
• Learn only with a limited number of training
  samples
• Cannot achieve good performance without enough
  training samples
• Idea finding more samples based on N initial
  samples
• For an initial positive sample i, the relevance
  degrees between every image sample j of
  the database are computed by a soft label
  function
• By ranking the soft label values, we can collect
  a number of samples with larger soft label values
  corresponding to the sample i.

9
LRF Algorithm (contd)

• The learning issue of the algorithm
• Based on the initial marked samples and the log
  information, we can collect a large number of
  positive and negative training samples associated
  with soft labels which represent their confidence
  degrees.
• Problem how to develop the algorithm to learn
  the data associated with soft labels ?
• Proposed Solution Soft Label Learning
• Soft Label Support Vector Machine (SLSVM)

10
SLSVM Soft Label Support Vector Machine
1

• Problem Formulation

1
1
SVM
0.5
1
1
1
11
SLSVM (contd)
12
SLSVM (contd)
13
Experimental Results

• Datasets
• Images selected from COREL image CDs
• 20-Category 2000 image instances
• 50-Category 5000 image instances
• Each category contains a specific semantic meaning

14
Experimental Results (contd)

• Image Representation
• Color Moment
• 9-dimension
• Edge Direction Histogram
• 18-dimension
• Canny detector, 18 bins of 20 degrees
• Wavelet-based texture
• 9-dimension
• Daubechies-4 wavelet, 3-level DWT
• 9 subimages are selected to generate the feature

15
Experimental Results (contd)

• Log Format
• Define a Log Session (LS) as a basic log unit,
  that corresponds to a relevance feedback round
• Each log session contains 20 images marked by
  users
• Log Collection
• Collect logs from 10 users
• Non-noisy logs 100 LS
• Noisy logs
• 20-Category 103 LS, 7.2 noise
• 50-Category 138 LS, 8.1 noise

16
Experimental Results (contd)

• Compared Schemes
• EU (Euclidean distance - baseline)
• RF_QEX (QEX query expansion)
• Multiple instance sampling, pick N nearest
  samples recursively
• RF_SVM
• Regular relevance feedback by SVM
• LRF_QEX
• Similar to RF_QEX, but we pick the samples
  weighted by soft labels in our framework (the
  larger the label, the smaller the distance)
• LRF_SLSVM

17
Experimental Results (contd)

• Settings
• Same Kernels e.g. RBF kernel
• Evaluation metric Average Precision of
  relevance / of returned
• Automatic evaluation Taking average precision
  over 200 query executions

18
Experimental Results (contd)

• Performance Comparison

19
Experimental Results (contd)

• Performance Comparison

20
Experimental Results (contd)

• Performance Comparison

21
Conclusions

• In this paper we proposed a new scheme to study
  users feedback logs for improving the
  performance of regular relevance feedback in
  CBIR.
• We introduce the soft label learning concept and
  developed a modified SVM technique, i.e. Soft
  Label SVM, to construct the algorithm for
  log-based relevance feedback.
• We evaluate our proposed method compared with
  traditional techniques and demonstrate promising
  results.

22
Limitations Future Work

• The proposed LRF with SLSVM algorithm still
  suffers performance drop when many noisy logs are
  appeared.
• Much noise may be involved when the scale of the
  image database is increased.
• When the number of log sessions is large, the
  dimension of the relevance matrix may be a
  problem.
• Training time of SLSVM need be considered for
  large scale datasets.
• Open questions
• Can we work out more effective Soft Label
  Learning techniques in the future?
• Can we include some noise filtering techniques
  into our framework?

23
Thank You! Q A
24
References (part)

• He King 2003 X. He, O. King, W.-Y. Ma, M. Li,
  and H. J. Zhang. Learning a semantic space from
  users relevance feedback for image retrieval.
  IEEE Transactions on Circuits and Systems for
  Video Technology, 13(1)3948, Jan. 2003.
• Huang Zhou 2001 T. S. Huang and X. S. Zhou.
  Image retrieval by relevance feedback from
  heuristic weight adjustment to optimal learning
  methods. In Proceedings of IEEE International
  Conference on Image Processing (ICIP01),
  Thessaloniki, Greece, Oct. 2001.
• Hong Huang 2000 P. Hong, Q. Tian, and T.
  Huang. Incorporate support vector machines to
  content-based image retrieval with relevant
  feedback. In Proc. IEEE International Conference
  on Image Processing (ICIP00), Vancouver, BC,
  Canada, 2000.
• Rui Huang 1999 Y. Rui and T. S. Huang. A
  novel relevance feedback technique in image
  retrieval. In Proc. ACM Multimedia (MM99), pages
  6770, Orlando, Florida, USA, 1999.on Image
  Processing (ICIP00), Vancouver, BC, Canada,
• Tong Change 2001 S. Tong and E. Chang.
  Support vector machine active learning for image
  retrieval. In Proceedings of the ninth ACM
  international conference on Multimedia, pages
  107118. ACM Press, 2001.

25
Appendix

• Kernel Comparison

26
CBIR