Query by Image Content

1
Query by Image Content
2
Content Based Image Retrieval

• Images generated by
• satellites, military reconnaissance/surveillance
  flights, fingerprinting, mug shots,
• scientific experiments, biomedical imaging
• Applications are
• Art Galleries and Museum Management
• Interior Design, Remote Sensing, Weather
  forecasting, Picture Archiving, Fabric and
  Fashion Design, Law Enforcement and Criminal
  Investigation, Scientific/Medical Database
  Management

3
Searching an image database
Is there a picture of a car?
4
Previous Approaches

• Set of attributes extracted manually and managed
  by standard DBMS
• Text phrases which describe content of picture
• Retrieval by text search
• Different people use different phrases to
  describe same picture
• Takes a lot of time especially for large databases

5
Searching an image database
Is there a picture of a car?
6
Automated Searching

• Feature extraction / object recognition system
• Automated approaches to object recognition are
  computationally expensive, difficult and tend to
  be domain specific
• Some features can be found semi- automatically or
  even manually
• Some (colour) are easier than others (shape,
  texture)

7
Features/attributes

• 3 levels of complexity
• Level 1
• Primitive features e.g. Colour, shape, texture
• Image of red circle, rough texture
• Level 2
• Logical features related to the object found
• Clip of aeroplane landing, car crashing
• Level 3
• Abstract information associated with the nature
  of the problem
• Clips of romantic landscapes, most exciting
  moment of the match

8
Query by Image Content(QBIC)

• QBIC allows queries on large image and video
  databases based on
• example images
• user sketches and drawings
• colour and texture patterns
• These are level 1 complexity
• Level 2 and 3 complexity is known as Semantic
  Gap

9
QBIC

• Database population
• images and videos are processed to extract
  features - colours, textures, shapes, camera and
  object motion
• Database query
• User comprises graphical query
• Features generated from query and input to
  matching engine that finds image or video with
  similar features

10
Histograms

• Illustration of a histogram for a grey scale
  image
• This is what information the computer sees
• You can tell from the histogram that the image is
  dark but nothing about the content.

0,0,25,50,86,92,100
11
Histogram matching

• Store histogram for each image
• Compare images using some statistical method
• Return sorted matches best fit first
• Problem - These would have the same histogram

12
Red Histogram
13
Green Histogram
14
Blue Histogram
15
Colour Histogram

• Build Histogram How?
• A true colour 24 bit image would have 16 million
  bins!!
• Takes too long
• Not necessary
• Use fewer bins - some systems uses as few as 64
  (4x4x4)

16
Colour Histogram example

• Divide each image colour into one of 64 colour
  bins
• 0,255,255 bin 0,3,3
• 153,51,0 bin 2,0,0
• 128,0,128 bin 2,0,2
• 204,153,255 bin 3,2,3
• 192,190,255 bin ?,?,?
• 190,20,0 bin ?,?,?

R
G
B
0
1
2
3
bin
17
Similarity

• Images are similar if their histograms are similar

10,0,8,20
Similarity( image1, image2) D(H1,H2) Where D is
some measure of distance between Histograms
0,8,20,10
10,1,8,19
18
Problems with histograms

• 0,255,255 bin 0,3,3 0,191,191 bin 0,2,2
• 153,51,0 bin 2,0,0 153,64,0 bin 2,1,0
• 128,0,128 bin 2,0,2 127,0,127 bin 1,0,1
• 204,153,255 bin 3,2,3 191,153,255 bin 2,2,2
• 192,190,255 bin 3,2,3 190,192,255 bin 2,3,3
• 190,20,0 bin 2,0,0 192,20,0 bin 3,0,0

19
Colour match

• Average colour
• adds up red, green, blue components of each pixel
• Colour position
• 6x8 or 9x12 grid overlays picture. For each block
  average Munsell colour and 5 most frequently
  occurring colours
• 48 or 108 vector match

QBIC interprets the virtual canvas as a grid of
coloured areas, then matches this grid to other
images stored in the database.
20
Shape matching
Aggregate shape measurement Major/minor axis
length and direction
Easier to automatically compute shape if
background is plain Need vector to describe
outline This was produced by PaintShopPro
21
Shape matching

• Need measurements that are invariant to scale,
  position and rotation
• To recognise a basic shape all of the below
  should have a similar vector

22
Textures

• Textures can be rough or smooth, vertical or
  horizontal etc

23
QBIC

• Queries can be
• Find images with approximately 30 red, 15 blue
• Find images with red round object

Drawn query
Best match
Matches in decreasing order of correlation
24
Object query

• To find red flower from simple query
• match each grid position until you get a best
  match
• May not take position into account

25
QBIC

• Queries can be
• Find images similar to this one
• Find sunsets with 30 red, 30 yellow
• Find images with 30 red and blue textured object
• Find images like this sketch

User sketch
Best match
26
Problems with shapes

• What would you input to retrieve both of these?
• Other problems
• Draw a horse returns dogs, cows etc
• Draw side view of cars doesnt retrieve front
  views
• Objects may be partially obscured

27
Colour similarity
All histograms should be similar to picture 1
28
Colour similarity and keyword
All histograms should be similar to picture
1 but keyword man was included. The picture of
the lion has the keyword MANE! The
following picture has keyword MANUFACTURING!
29
Colour Layout
Colour position should be similar to picture 1
30
Colour percentage
34 yellow 30 red 36 dont care
31
Colour percentage and keyword
34 yellow 30 red 36 dont care keyword SUNSET
32
Keyword only
keyword SUNSET
33
Custom Paint Query
Top two thirds of picture blue Lower
third yellow
34
Texture search
Texture search like first picture
35
Texture search
Texture search like first picture