IMAGE RETRIEVAL

1

• IMAGE RETRIEVAL
• Y. Rui, T. S. Huang, and S.-F. Chang
• Image retrieval current techniques, promising
  directions, and open issues
• Journal of Visual Communication and Image
  representation, Vol. 10, March 1999
• J. P. Eakins
• Retrieval of still images by content
• Proceedings of the Third European Summer School
  on Lectures on Information Retrieval-Revised
  Lectures, Lecture Notes In Computer Science, Vol.
  1980

2
INTRODUCTION

• Image retrieval has been a very active research
  since the 1970s.
• The trust comes from two major research
  communities
• Datadase management
• Computer vision
• The text-based image retrieval can be traced back
  to the late 1970s.
• A very popular framework for image retrieval
  was to annotate the images by text and then use
  text-based DBMS to perform image retrieval.
• However, there are two major difficulties
• The vast amount of labor required in manual image
  annotation.
• Different people may have a different perception
  of the same image content
• In the early 1990s, the emergence of large-scale
  image collections made the manual annotation
  approach more acute.
• To overcome the difficulties, content-based image
  retrival was proposed.

3
FEATURE EXTRACTION - I

• Features may include both
• text-based features (keywords, annotations,
  etc.)
• visual features (color, texture, shape, faces,
  etc.)
• Visual features can be further classified
• General features color, texture, shape
• Domain-specific features human faces,
  fingerprints
• Color
• This feature is one of the most widely used
  visual features.
• Color histogram is the most commonly color
  feature representation.
• Statistically, it denotes the joint probability
  of the intensities of the 3 color channels.
• Texture
• Refers to the visual patterns that have
  properties of homogeneity.
• Innate property of virtually all surfaces
  (e.g., clouds, trees, bricks, hair).

4
FEATURE EXTRACTION - II

• Shape
• Shape representations can be divided into two
  categories
• boundary-based
• region-based
• Color layout
• Although the global color feature is simple to
  calculate, and can provide reasonable
  discriminating power in image retrieval, it
  tends to give too many false positives when the
  image collection is large.
• Using color layout (both color feature and
  spatial relations) is a better solution for
  image retrieval.
• Segmentation
• Segmentation subdivides an image into its
  constituent regions or objects.
• Image segmentation algorithms are generally
  based on one of 2 basic properties of intensity
  values
• Discontinuity
• Similarity

5
HIGH DIMENSIONAL INDEXING IMAGE RETRIEVAL
SYSTEMS

• To make the content-based image retrieval truly
  scalable to large size of image collections,
  efficient multi-dimensional indexing techniques
  need to be explored.
• There are two major challenges in such an
  exploration
• The dimensionality of the feature vectors is
  normally of the order of 102.
• Since Euclidean measure may not be able to
  simulate human perception for all images, other
  similarity measures need to be supported.
• Image retrieval systems
• Many image retrieval systems (commercial and
  research) have been developed.
• Most systems support one or more of the
  following options
• Random browsing Leisure users may not have
  specific ideas about images or videos that they
  want to find. In this case, an efficient
  interactive browsing interface is very important.
• Search by example Searching for images by
  examples or templates is probably the most
  classical method of image search.
• Search by sketch Some systems provide advanced
  graphic tools for users to directly draw visual
  sketches to describe the images or videos they
  envision.
• Search by text The use of comprehensive textual
  annotations provides one method for image and
  video search and retrieval. The approach using
  textual annotations is not sufficient for
  practical application.
• Navigation with customized image categories
  Images and videos in a large archive are usually
  categorized into distinctive subject areas (e.g.,
  sports, transportation, and life style). An
  effective method in managing a large collection
  is to allow for flexible navigation in the
  subject hierarchy.

6
CLASSIFICATION OF QUERIES - I

• Level 1 comprises retrieval by primitive
  features such as color, texture, and shape.
• Examples of such queries may include
• Find pictures with long thin dark objects in the
  top left-hand corner
• Find images containing yellow stars arranged in a
  ring
• Find me more pictures that look like this
• Level 2 comprises retrieval by derived
  features, involving some degree of logical
  inference about the identity of the objects in
  the image.
• This level of retrieval can be divided into 2
  classes
• Retrieval of objects of a given type (e.g., find
  pictures of a double-decker bus)
• Retrieval of individual objects or persons (e.g.,
  find a picture of Eiffel tower)
• To answer these questions at this level,
  reference to some outside store of knowledge is
  normally required.
• In the first example above, some prior
  understanding is necessary to identify an object
  as a bus rather than a truck.
• In the second example, one needs the knowledge
  that a given individual structure has been given
  the name Eiffel tower.

7
CLASSIFICATION OF QUERIES - II

• Level 3 comprises of retrieval by abstract
  attributes, and necessitates a significant amount
  of higher level reasoning about the meaning and
  purpose of the objects or scenes depicted.
• This level of retrieval can be divided into 2
  classes
• Retrieval of named events or types of activity
  (e.g., find pictures of Scottish folk dancing)
• Retrieval of pictures with emotional or religious
  significance (e.g., find a picture depicting
  suffering)
• Success in answering queries at this level can
  require some sophistication.
• Complex reasoning and subjective judgment may be
  required to make the link between image content
  and the abstract concepts.
• Queries at this level are often encountered in
  both newspapers and art libraries.

8
AN IMAGE RETRIEVAL SYSTEM ARCHITECTURE
Three databases Image collection raw
images Visual features visual features
extracted from the images Text annotation
keywords and free-text descriptions of images
Three modules Multi-dimensional indexing are
used to index high dimensional feature
vectors. Query interface graphics-based.
Collects information from users and displays back
the results in a meaningful way. Query
processing processes requests from users.
9
REPRESENTATIVE EXAMPLES OF IMAGE RETRIEVAL SYSTEMS

• Commercial systems
• QBIC (Query by Image Content)
  http//wwwqbic.almaden.ibm.com
• Virage http//www.virage.com/cgi-bin/query-e
• RetrievalWare http//www.excalibur.com
• Experimental systems
• Photobook http//www-white.media.edu/vismod/demo
  s/photobook
• VisualSEEk and WebSEEk http//www.ee.columbia.ed
  u/sfchang/demos.html
• MARS http//jadzia.ifp.uiuc.edu8001
• Surfimage
• http//www-syntim.inria.fr/htbin/syntim/surfimage
  /surfimage.cgi
• Netra httpivaldi.ece.ucsb.edu/Netra
• Synapse http//cowarie.cs.umass.edu/demo

10
CONCLUSIONS - I

• Image retrieval is a very active research area.
• Text-based image retrieval
• Content-based image retrieval
• Classification of queries
• Level 1 retrieval by primitive features like
  color, texture, and shape.
• Level 2 retrieval by derived features, involving
  some degree of logical inference about
  the identity of the objects in the image.
• Level 3 retrieval by abstract attributes.
  Necessitates a significant amount of higher
  level reasoning about the meaning and purpose
  of the objects or scenes depicted.
• Research has shown promising results in using
  both textual and visual features in automatic
  indexing of images.

11
CONCLUSIONS - II

• Future research directions
• Humans are indispensable for image retrieval
  systems.
• Humans tend to use high-level concepts in
  everyday life. However, current computer visions
  techniques extract mostly low-level features from
  images.
• To organize and retrieve information on the Web,
  Web-based solutions are needed. Although
  solutions exist for text-based information, we
  need to have interoperable image retrieval
  systems.
• Currently, most of the existing research
  prototype systems only handle hundreds or at most
  a few thousand images. However, as the image
  collections are getting larger, multi-dimensional
  indexing aspect of image retrieval should be
  explored.
• Performance evaluation criteria are needed for
  image retrieval systems.
• Unfortunately, the subjectivity of image
  perception prevents us from defining an objective
  evaluation criteria.

12
CONCLUSIONS - III

• An equally important task is to establish a
  well-balanced large scale test bed.
• For text-based information retrieval, standard
  large scale testbed exists.
• For image retrieval testbed, MPEG-7 community has
  started to collect test data.
• The ultimate end user of an image retrieval
  systems are humans. So, the study of human
  perception of image content from a psychophysical
  level is crucial.