Content Based Information Retrieval in Forensic Image Databases

1

• Content Based Information Retrieval in Forensic
  Image Databases
•  

2
Outline

• Introduction
• Techniques for searching in image databases
• Feature selection and indexing
• Examples of real image databases
• Conclusions and future work

3
Introduction

• Many forensic databases of images
• - fingerprints
• - shoeprints
• - tool marks
• - cartridge cases
• - logos on pills containing MDMA
• - hand writing
• - video and 3D-information
• - numerous other databases in laboratories

4
Problem

• Selecting the right images based on forensic
  knowledge and retrieve only the images that are
  relevant, but do not miss images that are
  relevant.
• Constraints
• The algorithm has to work fast
• The images in database might be captured in
  different ways

5
Techniques

• Text strings by user (first generation)
• Features as texture, color, shape selected by an
  algorithm
• Recognize important features for the forensic
  field the mark, and use semantics (e.g..
  Fingerprints)
• Match using relevancy
• Use index for faster searches
• View images in web browser or computer screen and
  user can iterate

6
Data entry

• Images are captured using a camera or scanner
• The administrative data is combined by a user or
  another database
• The user also classifies
• The computer selects features
• The user approves the features
• The user searches

7
Feature selection

• Pre-processing step (eg. Wavelets for filtering
  out the relevant information)
• Select the features (color, texture, shape, or
  anything else that is important)
• Manual interaction with user, who selects the
  parts that are relevant

8
Color

• Just text green, red, blue
• By color histograms
• By color spaces that are invariant to lighting

9
Texture

• Different frequencies in the image
• Difficult to express in words
• Granularity, directionality and repetitiveness
• Vectors often high frequencies in image

10
Shape

• Shape are object identities in a meaningful form.
  
• squares, rectangles and circles
• 1. area, local elements of its boundary
• 2. transformation approach transform a shape
  into another shape
• Goal make it geometric invariant

11
Structure

• Gestalt of an image
• Line drawings versus pictures
• deriving a set of edges, corners and their
  location in image space

12
Others

• Motion (in video)
• Contents semantics often used in forensic image
  databases
• 3D-features

13
Matching / ranking

• Metric model
• Euclidean distance, City-block distance,
• Minkowsky distance, Earths Movers distance
  
• Advantage easy to compute
• can be used in indexes
• Disadvantage Human Visual System
• sometimes works different

14
Other matching methods

• Transformational distances how much effort is
  needed to convert a shape into another shape

15
Indexing

• Used for improving speed of searching
• Classical indexing text strings with hashes
• Geometrical hashes
• Many other methods in development that are
  mathematically complex (eg. SS-trees, R-trees)
• Active indexes

16
Performance
   
17
Other performance measures

• Average number of examples needed to obtain a
  certain degree of satisfaction
• Average number of iterations to obtain a
  satisfactory result
• Computational complexity

18
Commercial Databases that exist

• Fingerprint
• Faces
• Cartridge cases
• Shoeprints
• Tool marks
• Images of web

19
Fingerprints

• Syntactic ridge patterns and minutiae - string
  of primitives.The pre-defined classes are
  modelled as a set of grammars from the training
  samples
• Structural features based on minutiae are
  extracted and then represented using a graph data
  structure.
• Neural network approach the feature vector is
  constructed and classified by a neural network
  classifier.
• Statistical approaches statistical classifiers
  are used instead of neural classifiers.

20
Faces

• Determine invariant points of face in the image
• Use these points for matching
• Other method is using Principal Components
  Analysis
• Gabor filters are used for being invariant for
  illumination

21
Experiments with our databases

• Pixel-wise approach compare all pixels
• Feature-based approach compare the features in
  the images

22
Toolmarks

• Toolmarks are caused by friction between a
  surface and a tool
• Eg impression marks / striation marks

23
Crimes

• Burglaries (screwdrivers, crowbars, pliers)
• Homicides
• Other crimes
• The toolmark can be identified by a forensic
  expert

24
Example
25
Database of burglary cases

• In the Netherlands these are collected by the
  police.
• Links can be made between burglaries and tools of
  a suspect
• Comparison of striation marks time consuming

26
Striation Marks

• Different angles of tool to surface will cause
  different striation marks
• One tool with one striation marks, means eight
  comparisons at least for the examiner.

27
Database
28
Real life Striation marks / side light

29
3D structured light

• Project different line patterns on a toolmark
• Acquire these images with a camera
• Calculate the depth of the surface

30
OMECA
With micromirror device
31
Correlation methods

• Continuation of Previous research in 1995 for
  side light
• User interactive signature selection
• Calculating the standard deviation of the
  difference after normalization steps

32
Selecting a part and previewing

33
Slope compensation of striation mark

34
Slope in z-direction

Compensate by selecting the edges and normalize
35
Experiment

• 6 screwdrivers
• Striation marks of 45 degrees to the surface
• Both gray values / 3D values

36
Results Gray values
37
Results Structured Light

38
Conclusion

• 3D will result in Higher correlation factors
• Less sensitive to the lighting of the surface
• Fast for capturing

39
Future Research

• Testing on Larger Database
• Preselection on the shape of the blade
• Using 3D-images of the blade of the tool

40
Introduction

• Forensic firearm comparison
• cartridge case specific marks caused by
  feeding, extraction and ejection mechanism of the
  firearm.
• Firing pin and breech-face-marks

41
Manual Comparison
42
Ballistic Imaging Systems

• IBIS
• Drugfire
• MRT GE/2
• CIBLE
• Fireball

43
Images for testing
44
Database
45
Comparison
46
Overview database

47
Content based retrieval

• Images with noise
• Images that are rotated or shifted
• Difference in light source
• Difference in cartridge case metal
• Wear of firearm
• Wear of cartridge case
• Marks between two shots differ for mechanical
  statistical reasons

48
Matching pairs

• From practical cases
• matches between cartridge case found at the scene
  of crime and from the firearm
• 49 cases
• 19 different firearms (2-5 cases per firearm)

49
Preprocessing step

• Since there are some light variations
• equalization / normalization / circle

50
Alternative polar coordinates

51
Rank on variance
52
Rotation sensitivity
53
Multiresolution Approach

• Why multiresolution / wavelet transform
• Split
• background
• noise
• observed object

54
A trous

• Means with holes interlaced convolution at
  decreasing resolution
• no aliasing
• scaling function B3-spline
• original image is sum of wavelet planes

55
Scales
1 2
3 4
56
Brute force registration

• Rotate and translate the cartridge case
• In our tests 360 degrees and 20 pixels in x and
  y-direction
• Computationally very expensive !

57
Experiments
58
Firing pin

• Influence of marks and shapes in firing pin had
  to be reduced to 50 percent for the raw images
• For log polar images to 20 percent

59
Computation with brute force

• Computationally very intensive
• 3 computers worked 91 days for computing these
  numbers

60
Rotation and translation invariant

• log-polar transform of the Fourier magnitude
• Suppose translation and rotation and some
  scaling

61
Polar Coordinates
Rectangular Log-Polar
62
Polar coordinates

• Mapping of Fourier magnitudes into polar
  coordinates
• logarithmic transformation of r-axis transforms
  scaling into a shift
• polar mapping followed by log-transform of r-axis
  log-polar transform

63
Crosscorrelation of Triple invariant Image
descriptors
64
Faster preselection

• Based on tracking research
• Good features are located by examining the
  minimum eigenvalue of each 2 by 2-gradient
  matrix.
• The features are tracked by a Newton-Raphson
  method of minimizing the difference between the
  two windows
• Multiresolution tracking allows for even large
  displacements between images.

65
KLT tracking method
66
KLT

• The number of points tracked is a measure for the
  ranking on the list
• In our test 41 were in the top positions for
  side light
• 49 are in the top five percent
• Rotation and translation are followed

67
Conclusions

• If the user follows the protocol and the marks
  are clear, good results are possible and fast
  algorithms are possible
• Brute force registration works, however it is
  computationally very intensive

68
Conclusions (2)

• Brute force registration of third scale gives
  good results
• Log polar transform in third scale appears to
  work for 41 out of 49 images

69
Conclusions (3)

• KLT-method works for side and ring light images
  for pre selection
• Further refinement by selection of areas that
  should be compared
• Relevance feedback of correlation should be used

70
Future Research

• Optical correlators
• Other databases of images
• Other algorithms of correlation

71
Introduction

• Drugs department of our institute
• Illicitly produced drugs
• MDMA / amphetamine
• Database of these pills for linking manufacturers

72
Introduction (2)

• In database
• image- diameter- shape- weight- chemical
  composition

73
Introduction (3)

• Logos contain all kind of figures
• Description is also in database

74
Requirements correlation

• Many kind of trademark images
• Fantasy images
• Pill itself may be damaged
• Other shapes of the pill
• Position of the pill
• Pill in top position
• 2D-image of 3D shape

75
Correlation Methods

• Color
• Texture
• Shape (needed for logos)
• Other types of primitive features
• To search on these features descriptors are
  necessary

76
Shape retrieval

• Global features
• Aspect ratio, circularity, moments invariants
• Local features
• Sets of consecutive boundary segments
• Other methods
• Elastic deformation of shapes / wavelets etc.

77
Implementations

• Commercial databases
• QBIC / Virage / Excalibur
• Imatch
• Others
• MPEG-7
• Log polar correlation
• Photobook / Chabot / Visualseek / MARS / Zomax /
  Surfimage

78
Trademarks

• Trademarks are similar
• However the 3D-shape is not involved
• Project with University of Amsterdam on Drugs
  pills and trademarks (ZOMAX)

79
Correlation Methods MPEG-7

• Due to be approved in September 2001
• Video streams
• Will have effects on CBIR activities
• Test systems for members available

80
Correlation Methods MPEG-7 shape

• Object bounding box
• Region Based Shape
• Contour Based Shape

81
Contour Based Shape

• The number of peaks in the image
• The highest peak height
• The eccentricity
• Contour curvature vector

82
Test database

• Of cases from 1991
• Over 600 images are stored in the database
• One drugs pill is entered in the database under
  20 different positions of 15 degrees rotation.
• Comparison of results based on this drugs pill

83
Test images
84
Results QBIC

85
Results Imatch

86
Results with QBIC
87
Results with Imatch
88
Results with Log Polar
89
Results with MPEG-7
90
Improvement Pre-processing

• Labeling pill / logo
• Split by subtraction

91
MPEG-7 after processing
92
Conclusions and discussion

• Color features appear to work better, however
  these do not always work for our case
• Light conditions might influence results
• Log Polar correlation takes long
• MPEG-7 Curvature Space representation fast
  and most in top positions after pre-processing
• 3D acquisition of the drugs pill could work better

93
Future developments

• Probably more MPEG-7 - standardization
• Optical computers or parallel processors might
  make more complex methods possible
• 3D images and search methods
• Third generation databases might take over ten
  years, since first the HVS has to be understood