Learning in Large Scale Image Retrieval Systems

1
Learning in Large Scale Image Retrieval Systems
by Pradhee Tandon Roll No. 200607020

• Under the guidance of
• Dr. C. V. Jawahar Dr. Vikram Pudi

2
Image Retrieval

• Explosive growth in images
• Easy access to most of these on the web
• Contemporary systems used tags
• The best commercial systems are still tag based
• Inadequate and unreliable
• Manual tagging is infeasible
• Content based retrieval is the best option

3
Content Based Image Retrieval
Query
Feature Extraction
Comparison Module
Results
Image Feature Database
Query
Results
4
Content Based Image Retrieval
Query
Feature Extraction
Comparison Module
Results
Image Feature Database
Feature Index
5
Content Based Image Retrieval
Query
Feature Extraction
Comparison Module
Results
Rf
Relevance Learning
Feature Index
Memory Logs
6
Content Based Image Retrieval
Query
Feature Extraction
Comparison Module
Results
Rf
Relevance Learning
Learning Memory
Feature Index
7
Scope of work

• Features
• Color Histograms
• Texture Filters
• Shape Context
• SIFT
• GLOH
• Spatial indexing methods
• Kd trees
• R-tree
• Distance Metrics
• Euclidean
• Mahalanobis
• KL Divergence

• Relevance Feedback
• Short term learning
• Long term learning
• Content Free Retrieval
• Active Learning
• Diversity Retrieval

8
Requirements

• Efficient Image Retrieval
• Learning relevant features in images
• Learning image-image relationships
• Diversity in retrieval for improved learning

9
Requirements

• Efficient Image Retrieval
• Learning relevant features in images
• Learning image-image relationships
• Diversity in retrieval for improved learning

10
FISH The System
11
Implementation of FISH

• Image Representation in FISH
• MPEG-7 Colour Structure Descriptor
• Maximum Response Filters for Textures

Developed on the LAMP stack, using C/C, Perl,
PHP, HTML, MySQL and Apache TPIE toolkit from
Duke University for B tree implementation
12
Indexing Scheme
Interactive response over large databases (less
than a second) Efficient scalable index (dynamic
with data) Similarity indexing scheme (r-tree,
kd-tree, ss-tree) Support for changing
similarity metrics (metric changes with learning)
B tree based index
Nataraj et. al, MMM 2007, Efficient Search with
Changing Similarity Measures on Large Multimedia
Datasets
13
The Retrieval Algorithm
Query
Feature Extraction
Comparison Module
Results
Rf
Relevance Learning
Feature Index
Learning Memory
Retrieval in FISH
14
Retrieval Performance
Retrieval times with increasing DB size in (secs)
dimensions fixed at 10
Retrieval times with increasing Dimensions in
(secs) DB size fixed at 1 lakh
15
Requirements

• Efficient Image Retrieval
• Learning relevant features in images
• Learning image-image relationships
• Diversity in retrieval for improved learning

16
Content Based Image Retrieval
Query
Feature Extraction
Comparison Module
Results
Rf
Relevance Learning
Feature Index
Memory Logs
17
Learning - expectations

• Effective capture user intent correctly
• Efficient interactive retrieval response
• Scalable limited computational overhead for
  large collections
• Adaptive caters to individual users
  subjectivities
• Intra-query or short term learning (STL)
• Evolving incrementally improves across users
  and queries
• Inter-query or long term learning (LTL)
• Dynamic seamlessly absorbs changes in the
  collection

18
Learning - Method

• Relative relevance of features using feedback
• Numerous methods can be used
• Discriminative variance is as -
• Weights are incrementally learnt over iterations
  using
• At the end of the session long term learning is
  updated for the relevant images using
• Image to image dissimilarity is computed using
• Weighted Mahalanobis

19
Improved accuracy
Precision across sessions using LTL
Rank Convergence of top N relevant samples
Sum of ranks of Top 10 relevant images converges
close zero (downshifted) over multiple sessions
with long term learning
20
Improved retrieval
System learns the rock and sky pattern over
sessions
System learns the yellow flower in the hedge over
sessions
Top 9 results for queries across 3 different
sessions (left-most are queries too)
21
Optimized Retrieval
22
Content from Learning
Long term memory allows learning of relevant
image features Converges to popular content over
sessions For example, Assume, features are
associated with individual pixels,
colors Consider a gray image, pixels for more
relevant features are colored brighter
Actual image
Content image
23
Visual Content Extraction
Over sessions
After a large number of sessions
24
Requirements

• Efficient Image Retrieval
• Learning relevant features in images
• Learning image-image relationships
• Diversity in retrieval for improved learning

25
Content Based Image Retrieval
Query
Feature Extraction
Comparison Module
Results
Rf
Relevance Learning
Feature Index
Memory Logs
26
Image-Image Relations
Query
Given a history of patterns in behavior and a
current partial pattern, collaborative filtering
predicts the next pattern for the latter Content
Free Image Retrieval or CFIR, uses feedback logs
to predict the next set of results for the
current pattern
27
Hybrid Image Retrieval

• We integrate them in a Bayesian inference like
  framework,
• The a priori relationships from logs
• The evidence from visual similarity
• Retrieval is an a posteriori estimation problem

28
Bayesian Image Retrieval System
Architecture of the proposed Bayesian Image
Retrieval System
29
Bayesian Image Retrieval...

• posterior prior evidence
• Efficient a priori updates
• The prior probabilities are not stored, reduces
  updates
• Co-relevance between images are stored in a
  matrix
• The a priori is estimated using the co-relevance
  values
• Evidence computation
• Weights are learnt using discriminative variance
  method
• Weighted Mahalanobis for (dis)similarity

30
Concept Discovery

• a priori matrix has embedded patterns of similar
  co-relevances
• Co-relevance patterns can be summarized into k
  concepts
• cluster the patterns into V concepts 1k.
• clustering is repetitive but offline
• exhaustive comparisons are avoided

31
Accuracy with Bayesian
Gain in precision with Bayesian
Gain in precision across sessions
Using real human user feedback logs
Using annotation based feedback logs
32
Accuracy with Bayesian
CBIR results and Bayesian results
33
Requirements

• Efficient Image Retrieval
• Learning relevant features in images
• Learning image-image relationships
• Diversity in retrieval for improved learning

34
Diversity in Image Retrieval
Query
Query
35
Skylines the natural solution

• Results should be similar in a variety of
  different ways
• Skylines return non-dominated samples
• Non-dominated samples are closer to the query
  than all the others, in at-least one way
  (attribute)

36
Skyline Extraction
Architecture of the proposed skyline based
similarity retrieval system
37
Diversity with Skylines
38
Efficient Skylines
Synthetic data with 10 dimensions and 10000 and
15000 data points
Real image data with 12 and 9 dimensions with
11901 real images
39
Preferential Skylines

• Relevance feedback represents users preference
• Weights learned using feature relevance
• Skylines are then computed in user space

40
Contributions

• Designed and implemented a web-based image
  retrieval system, called FISH
• Proposed an efficient feature relevance learning
  algorithm
• Integration of complimentary CFIR and CBIR a
  Bayesian inference framework
• Skylines to retrieve diversely similar samples
  for a given query

41
Future directions

• Videos are richer and the next step
• Efficient higher level concept discovery is
  needed
• Skylines with preference should be explored
  further

42
Publications

• Pradhee Tandon, Piyush Nigam, Vikram Pudi, C. V.
  Jawahar, FISH A Practical System for Fast
  Interactive Image Search in Huge Databases, in
  Proceedings of the 7th ACM International
  Conference on Image and Video Retrieval (CIVR
  08), July 6-8, 2008, Niagara Falls, Canada.
• Pradhee Tandon, C. V. Jawahar, Long Term
  Learning for Content Extraction in Image
  Retrieval, in Proceedings of the 15th National
  Conference on Communications (NCC 09), January
  16-18, 2009, Guwahati, India.
• Pradhee Tandon, C. V. Jawahar, Bayesian Image
  Retrieval submitted to 3rd
• International Conference on Pattern Recognition
  and Machine Intelligence (PReMI 09), December
  16-20, 2009, New Delhi, India.

43

• Thank you ?

44

• Addendum

45
The Retrieval Algorithm
Learning discussed in detail later
46
Bayesian Image Retrieval

• The a priori probability of retrieving image a
  with query q is
• P(R) n(q,a)/n(a)
• where n(a) denotes relevant retrievals for a
• The evidence from visual similarity is computed
  as
• p(SR) f(w,q,a)
• where weights w are refined using relevance
  feedback
• The posterior probability of retrieval is
  computed as
• p(RS) p(SR) P(R)
• the denominator can be ignored
• PicHunter is a hybrid but does no feature
  learning
• Zhong et. al, use Bayes inference for a
  probabilistic decision only

47
Skyline Extraction