Visualisation and Comparison of Based on Selforganising Maps

1
Visualisation and Comparison of Based on
Self-organising Maps

• Da Deng, Jianhua Zhang, Martin Purvis
• Dept. of Information Science
• University of Otago
• ddeng_at_infoscience.otago.ac.nz

2
Image Retrieval

• A picture is worth a thousand words.
• Rich semantics is intrinsic in images.
• Reverse engineering for semantics extraction from
  raw image storage yes, but how?
• Image understanding in general still impossible!
• Text-based queries needs ontology and thesaurus
  etc. to make sense.
• A work-around CBIR

3
Content-Based Image Retrieval

• Content defined as low-level visual features.
• Archive, browsing and query of image data based
  on content-based representations.
• Relevant techniques feature extraction, pattern
  recognition, indexing, data mining
• Imagists have done much better CBIR!
• The apparition of these faces in the crowd
• Petals on a wet, black bough.

4
Popular CBIR Features

• Colour Statistics
• Global/regional histograms
• Colour correlograms
• Texture Features
• Gabor filters
• Wavelet transform energy
• Co-occurrence matrix
• Shapes
• Moments
• Fourier descriptors etc.
• Texts, faces etc.
• Motions in video

5
Image Collection Profiling

• Facilitates visualisation, browsing, comparison,
  and search.
• Our content-based approach
• Cluster content-based image features from the
  image collections and generate certain profiles.
• Visualise profiles of different image
  collections.
• Develop distance measure defined over these image
  profiles.

6
Mapping of Feature Spaces

• Clustering
• K-means, GEM etc.
• Multi-dimension scaling (MDS)
• Does the job of dimension reduction.
• Principal Component Analysis A statistical
  method to extract vectors on which data have the
  largest variance.
• Sammons Mapping A gradient descent optimisation
  process aiming at keeping the order of data point
  distance
• Self-organising Maps (SOM)
• Kohonen 1981
• A clustering algorithm as well as doing MDS onto
  a fixed topology

7
SOM Characterisitcs

• Easy visualisation
• Prototypes located usually on 2-D lattices
• U-matrix for colouring cells
• Plus MDS
• Topology preserving
• Similar inputs mapped to the same node or nodes
  nearby
• Probability density matching
• Tends to represent a cluster of frequently
  occurring input stimuli with more nodes

8
SOMs in Information Retrieval

• Hierarchical maps (Lampinen 1992) are necessary
  to index the large storage of documents and
  multimedia contents.
• PicSOM (Laaksonen et al., 1999)
• TS-SOM
• SOMlib (Rauber Merkl, 1999)
• GHSOM

9
Visualisation of SOMs

• This needs to be stable and fast to generate.
• SOMs are subject to variance owing to random
  initialisation.
• Sammons mapping used in SOM visualisation also
  varies over random initialisation.
• Sammons mapping is slow to converge.
• Solution
• Initialisation using PCA.
• When visualising the trained map, apply 2-D PCA
  of the prototype vector space before doing
  Sammons mapping.
• Outcome stabilised map, and fast-to-converge
  visualisation.

10
COVIC
11
Travelling within the Hierarchy
12
Facilitating Image Search

• The hierarchical SOM also helps to speed up image
  search.
• However, with some inaccuracy

13
Comparing Image Collections
14
Mapping the Views
15
Mapping Vehicles
16
Visualisation of All Four Maps
17
Similarity of SOMs

• Kaski Lagus 1996 proposed a method based on
  the quality of maps (defined with the training
  data set).
• Direct comparison of feature maps is more
  desirable.
• Our approach
• Starts by viewing map as a point set of
  prototypes
• Then adapts the point-set distance for SOMs
• Compared with other point-set distances
• Use visual assessment as supplement.

18
Earth Moving Distance

• EMD (Rubner et al. ICCV98)
• Moving from Aai, wi to Bbj, uj.
• All feasible flows satisfy constraints

19
Hausdorff Distance

• Classifical point set distance measure used in
  image matching.
• Given two point sets X and Y, the Hausdorff
  distance from X to Y is defined by
• Hausdorff Metric

20
Hausdorff in Question

• SOM is not a prototype set only!
• But also a grid structure that reflects
  characteristics of the data set it is trained on.
• Whats different between the maps shown here?

21
Adapting HD on SOMs

• Combining difference in local deviation with
  point-to-point distance
• Hausdorff Distance with Local Deviation (HDLD)

22
SMD ? SAND

• SMD Sum of Minimum Distances
• SMD does not take the neighbourhood into account.
• SAND Sum of Average (matched) Neighbourhood
  Distance.

23
Distance Measures Compared
By Regional Average Colours.
By Gabor Filtering Energy.
24
Content-based? Still A Problem!

• Mapping CBIR features can help to discover
  conceptual clusters.
• This is however, most likely to be domain
  dependent.
• Semantic representation will help to reduce
  irrelevant results.

25
Map A Video?
26
Future Directions

• Similarity assessment over multiple feature
  spaces.
• Probabilistic modelling of prototype sets may
  give more options.
• Video comparison and retrieval.
• Make use of classification techniques for object
  recognition.
• Achieve some semantic representation over CBIR?

27
Conclusions

• CBIR offers an effective way for multimedia
  navigation retrieval.
• We extend the CBIR approach onto multi-media
  collection profiling for navigation and
  comparison purposes.
• Quantatitive comparison by using point-set
  distance measures.
• Future directions
• On-line image collection profiling with more
  efficient computational model.
• Exploration of visual semantics.