CSM06 Information Retrieval

1
CSM06 Information Retrieval

• LECTURE 8 Tuesday 18th November
• Dr Andrew Salway
• a.salway_at_surrey.ac.uk

2
Lecture 8 Image Retrieval

• Image Data and Querying Strategies
• Different kinds of metadata for visual
  information
• Manual Annotation of Images
• Automatic Image Annotation using Collateral Text,
  e.g. WebSEEK system
• Similarity-based Image Retrieval using Perceptual
  Features, e.g. QBIC and Blobworld

3
Image Data

• Raw image data a bitmap with a value for every
  picture element (pixel) (cf. vector graphics)
• Captured with digital camera or scanner
• Different kinds of images
• Photographs people, scenes, actions holiday
  albums, criminal investigations
• Fine art and museum artefacts
• Medical images x-rays, scans
• Geographic Information Systems images from
  satellites
• Meteorological Images

4
Querying Strategies

• Language-based query
• Text-based query describing entities, actions,
  meanings, etc
• By visual example
• Sketch-based query draw coloured regions
• Image choose an image and ask for more which are
  visually similar

5
Metadata for Visual Information

• Del Bimbo (1999) content-independent
  content-dependent content-descriptive
• Shatford (1986) in effect refines content
  descriptive ? pre-iconographic iconographic
  iconological based on Panofskys ideas
• Chang and Jaimes (2000) 10 Levels in their
  conceptual framework for visual information

6
3 Kinds of Metadata for Visual Information (Del
Bimbo 1999)

• Content-independent data which is not directly
  concerned with image content, and could not
  necessarily be extracted from it, e.g. artist
  name, date, ownership
• Content-dependent perceptual facts to do with
  colour, texture, shape can be automatically (and
  therefore objectively) extracted from image data
• Content-descriptive entities, actions,
  relationships between them as well as meanings
  conveyed by the image more subjective and much
  harder to extract automatically

7
Three levels of visual content

• Pre-iconographic generic who, what, where, when
  
• Iconographic specific who, what, where, when
• Iconological abstract aboutness
• Based on Panofsky (1939) adapted by Shatford
  (1986) for indexing visual information

8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
10 Levels (Jaimes and Chang)

• They borrowed concepts from cognitive psychology,
  library sciences, art, and content-based image
  retrieval to develop a conceptual framework for
  indexing different aspects of visual information
• FURTHER READING
• Jaimes and Chang (2000)A Conceptual Framework
  for Indexing Visual Information at Multiple
  Levels available on-line.

12
Image Annotation manual

• Keyword-based descriptions of image content can
  be manually annotated
• May use a controlled vocabulary and consensus
  decisions to minimise subjectivity and ambiguity

13
Systems

• For examples of manually annotated image
  libraries, see
• www.tate.org.uk
• www.corbis.com
• Iconclass has been developed as an extensive
  classification scheme for the content of
  paintings, see
• www.iconclass.nl

14
(No Transcript)
15
(No Transcript)
16
Image Annotation using collateral text

• Images are often accompanied by, or associated
  with, collateral text, e.g. the caption of a
  photograph in a newspaper, or the caption of a
  painting in an art gallery
• Keywords, and possibly other information, can be
  extracted from the collateral text and used to
  index the image

17
Image Annotation using collateral text WebSEEK
System

• The WebSEEK system processes HTML tags linking to
  image data files, (as well as processing the
  image data itself), in order to index visual
  information on the Web
• Here we concentrate on how WebSEEK exploits
  collateral text, e.g. HTML tags, for image
  indexing-retrieval NB. Current web search
  engines appear to be doing something similar

18
Image Annotation using collateral text WebSEEK
System

• Term Extraction terms extracted from URLs, alt
  tags and hyperlink text, e.g.
• http//www.mynet.net/animals/domestic-beasts/dog37
  .jpg
• animals, domestic, beasts, dog
• Terms used to make an inverted index for
  keyword-based retrieval
• Directory names also extracted, e.g.
  animals/domestic-beasts

19
Image Annotation using collateral text WebSEEK
System

• Subject Taxonomy manually created is-a
  hierarchy with key-term mappings to map key-terms
  automatically to subject classes
• Facilitates browsing of the image collection

20
(No Transcript)
21
(No Transcript)
22
Image Annotation using collateral text WebSEEK
System

• The success of this kind of approach depends on
  how well the keywords in the collateral text
  relate to the image
• URLs, alt tags and hyperlink text may or may not
  be informative about the image content even if
  informative they tend to be brief perhaps
  further kinds of collateral text could be
  exploited

23
Similarity-based Retrieval

• Perceptual Features (for visual similarity)
• Colour
• Texture
• Shape
• Spatial Relations
• These features can be computed directly from
  image data they characterise the pixel
  distribution in different ways
• Different features may help retrieve different
  kinds of images

24
Perceptual Features Colour

• Colour can be computed as a global metric, i.e. a
  feature of an entire image or of a region
• Colour is considered a good metric because it is
  invariant to image translation and rotation and
  changes only slowly under effects of different
  viewpoints, scale and occlusion
• Colour values of pixels in an image are
  discretized and a colour histogram is made to
  represent the image / region

25
Perceptual Features Texture

• There is less agreement about what constitutes
  texture and a variety of metrics
• Generally they capture patterns in the image data
  (or lack of them), e.g. repetitiveness and
  granularity
• (Compare the texture of a brick wall, a stainless
  steel kettle, ripples in a puddle and a grassy
  field)

26
Perceptual Features Shape

• Unless a simple geometric form (e.g. rectangle,
  circle, triangle) then an objects shape will be
  captured by a set of features relating to, e.g.
• Area
• Elongatedness
• Major axis orientation
• Shape outline

27
Perceptual Features Shape

• Parametric Internal Features describe the
  enclosed region
• Geometric attributes region area minimum
  rectangle/ellipse/ circle which includes the
  region or maximum which can be included
  elongatedness, i.e. ration between max. length
  and max. width
• Digital Moments 

28
Perceptual Features Shape

• Parametric External Features describe the
  boundary of the region
• Pixel-based representations for salient points
  store co-ordinates, and information about
  direction of curve ? chain encoding

29
Similarity-based Retrieval

• Based on a distance function - one metric, or
  combination of metrics, e.g. colour, shape,
  texture, is chosen to measure similarity between
  images or regions
• Key features may be extracted for each
  image/region to reduce dimensionality
• Retrieval is a matter of finding nearest
  neighbours to query (sketch-based or example
  image)
• Similarity-based Retrieval is more appropriate
  for some kinds of image collections / users than
  others

30
Systems

• For examples of image retrieval systems using
  visual similarity see
• QBIC (Query By Image Content), developed by IBM
  and used by, among others, the Hermitage Art
  Museum
• http//wwwqbic.almaden.ibm.com/
• Blobworld - developed by researchers at the
  University of California
• http//elib.cs.berkeley.edu/photos/blobworld/start
  .html

31
(No Transcript)
32
(No Transcript)
33
Reading

• ESSENTIAL READING
• Smith and Chang (1996), Searching for Images and
  Videos on the World-Wide Web.
• ONLY Section 3 of this paper is essential
• Choose download option from
• http//citeseer.nj.nec.com/smith96searching.html
• Excerpts from del Bimbo (1999), Visual
  Information Retrieval available in library
  short-term loan articles.

34
Reading

• FURTHER READING
• Eakins (2002), Towards Intelligent Image
  Retrieval, Pattern Recognition 35, pp. 3-14.
• Enser (2000), Visual Image Retrieval seeking
  the alliance of concept-based and content-based
  paradigms, Journal of Information Science 26(4),
  pp. 199-210.