Data Structures

1
Data Structures
Levels of Image Representation Computer vision
systems operate over a vast range of levels of
abstraction and a wide variety of data
structures. There are (more or less) four
major classes of information, listed in order of
increasing abstraction - Generalized Images
Analogical representations of the sensed scene.
Pictures, of virtually any type. Even more
generally, spatial data. - Segmented Images
Organized images such that various regions,
contours, and so on are associated with
meaningful (one hopes) scene features. -
Geometric Representations In these we try to
capture the notion of shape (and things related
to shape) and to associate it with each image
segment. - Relational Models With these we try
to construct a scene description and to interpret
the image. NB There may be many levels of data
structures for any given member of the four
classes given above.
2
Processes (reminder)
Six steps to success in computer vision... To
move between these various levels of abstraction,
we need to define processes. These processes may
be categorized (roughly) as follows - Image
Formation Acquisition, and (speaking
generally) the computation of higher level
images. - Conditioning Noise and distortion
mitigation. - Labeling Detection of primitives
(edges, corners, blobs, textures, ...). -
Grouping Also called perceptual organization,
combining those primitives that seem to be
related. - Characterization Assigning
properties, or lists of attributes to groups. -
Matching Binding characterized groups to
concepts known to the system (recognition,
classification).
3
Combining Processes Representations
4
Lacing it Together This cycle of conditioning
-- matching may be repeated many times in the
analysis of a complex scene by a sophisticated
vision system. The demarcation between steps
may be a little fuzzy. The most important
thing is the concept of an orderly approach to
the problem. That is, the system design should
be partitioned into modules such that the
processes move us from level to level, and such
that these individual processes are not
prohibitively complex. Control Issues
Bottom-up As we describe it, but almost never
completely effective on its own. Search spaces
are too large, constraints too hard to apply, too
many sources of error. Top-down
Hypothesize and test, difficult in general
systems, but can be quite effective when combined
with bottom-up processes to limit the space of
hypotheses.
5
Basic Data Structures
Matrices Each matrix location may hold a
piece of scalar information about a particular
point in the image space. Multiple matrices in
register can hold vector information (like a
gradient). Examples include... - Brightness,
the image(s) we usually start with - Gradient
(magnitude), useful in detecting edges - Local
gray level variance, an indicator of texture -
Detection decisions (binary) on any of several
things There are other matrix data structures
for which the matrix elements are not in
one-to-one correspondence with image pixels.
Examples include cooccurrence matrices and
adjacency matrices (encoding a graph data
structure). Chains Chains are used to encode
the trajectory of contours or region boundaries
across the image. A numerical code describes a
fixed set of possible point-to-point moves.
6
Chain Code Example
(8-Adjacency)
The code 7 6 7 6 5 7 0 1 1 2 1 3 4 4
7
Crack Codes
Crack codes use the same directional encoding
as a 4-connected chain code, but describe the
moves one would make in the interstices between
pixels to circumnavigate an image region.
The code 3 2 3 0 3 3 2 3 0 1 0 0 3 0 1 2 1 1 2
1 1 2
8
Topological Data Structures
These are the relational data structures used
in the top levels of the processing and
representations diagram. They describe an
image as a set of elements and their
interrelationships (usually spatial, but also
photometric or temporal). Most of these are
based in graph theory. Relational databases can
be used to represent some of this, too. A
graph consists of a set of nodes (vertices) and
a set of arcs (edges). Each node represents an
entity, such as a region in an image, while an
arc joining two nodes indicates that some
(predefined) relationship exists between them.
Arcs may be undirected or directed. An
undirected set of arcs can encode a relationship
such as adjacency , while a directed set of arcs
could encode, for example, larger-than. Graphs
may be attributed , in which additional
information is attached to the nodes and arcs.
This information can include attributes
associated with the nodes (size, color, ...) and
relational measurements associated with the arcs
(distance-from, relative-orientation, ...).
9
Region Adjacency Graph
One graph of interest, and which serves to
illustrate the idea, is the region adjacency
graph. Suppose we have a segmented image. Let
each region (plus the image border) be assigned
to a node in the graph. Let the arcs of the
graph encode adjacency an arc is drawn between
two regions if they share a border.
10
Another Type of Graph We could define an
adjacent-and-brighter-than graph. Notice
that it is a directed graph.
The basic structure of a simple graph can be
represented compactly in an adjacency matrix.
For our two examples we have the following.
Undirected graphs have symmetric adjacency
matrices directed graphs do not (in general).
11
Structural Descriptions
12
SDs Formal Definition
Basic This definition is due to Linda Shapiro
A structural description is a pair D (P,
R) P a set of primitives P p1, p2, ...
, pn R a set of named M-ary relations over
P Each primitive is defined in the (Cartesian)
product set with A a
set of attributes (color, brightness, area,
...) V a set of values (red, green, ve
integers, ...) A primitive is completely
specified by associating with it a set of
attribute - value pairs . For example p9
(color, red) , (aspect-ratio, 10) , (boundary,
smooth) Now, for R we have
R R1, ... , RK And for k 1, ..., K we
define Rk (NRk, Rk) NRk is the name
of the kth relation in the set. Rk is a set
of Mk -tuples of primitives having the kth
relationship among them That is, Rk is a set
of Mk - tuples drawn from the set of primitives P.
13
SDs Example
NB This is a stupid, toy example just to
illustrate the idea!
P p1, p2, ... , p7 A shape, area, color V
triangle, circle, rectangle, 10, ... , 1,
black, white, gray R R1, R2, R3 NR1
left-of NR2 above NR3 spokes M1 2
(binary) M2 2 (binary) M3 4 (4-ary) p1
(shape, triangle), (area, 7), (color, white) p2
(shape, circle), (area, 10), (color, black)
etc..... R1 (2 3) (2 4) (2 6) (3 4) (3 6) (1
5) ... R2 (1 2) (5 7) R3 (4 5 6 7)
14
Some Observations This is a declarative model
(in the language of syntactic pattern
recognition). Formal grammars are procedural.
Member M-tuples in each relation are determined
by a membership predicate -- something we have
to define. The relations are crisp -- meaning
that a given M-tuple of primitives is either in
or out -- with no uncertainty. This can prove
problematic in real applications -- will fix
later. We will use structural descriptions
(and other graph-theoretical models) in -
Object recognition - Scene analysis - Stereo
matching - Spatial data registration Use of
these descriptions will require the development
of matching criteria and algorithms. This
matching will have to take place in the presence
of noise, clutter, distortion, and other sources
of uncertainty. We will have to develop means of
measuring the quality of a graph match, and of
seeking the best match possible (reasonable).
15
Another Example It is instructive to compare the
structural description with a plain old graph
-- to see that the structural description is
really a collection of attributed, possibly
directed, hypergraphs. Lets revisit the first
toy image and consider the following structural
description of it... A Outer-border-type,
Gray-level-type, Number-of-holes V smooth,
piecewise-linear, constant, varying, 0, 1, 2,
... p0 (outer-border-type,
piecewise-linear), (gray-level-type,
constant), (number-of-holes, 1) p1
(outer-border-type, piecewise-linear),
(gray-level-type, constant), (number-of-holes,
1) p2 (outer-border-type, piecewise-linear),
(gray-level-type, varying), (number-of-holes,
0) p3 (outer-border-type, smooth),
??? (gray-level-type, varying),
(number-of-holes, 0) p4 (outer-border-type,
smooth), (gray-level-type, constant),
(number-of-holes, 1) p5 (outer-border-type,
smooth), (gray-level-type, constant),
(number-of-holes, 0) p6 (outer-border-type,
smooth), (gray-level-type, constant),
(number-of-holes, 0)
16
Notice that p0 and p1 -- and p5 and p6 --
respectively, are indistinguishable, based on the
attribute-value pairs given for them. This
might be a problem, or not. It is certainly not
an incorrect description, although it might be
an incomplete description. Lets define the
following two relations for use in our structural
description NR1 Adjacent NR2
Brighter-than R1 (1 6) (1 3) (1 4) (1 0) (1
2) (2 0) (3 4) (4 5) R2 (6 3) (6 1) (1 2) (4
3) (4 5) (6 2) (4 2) (5 1) Notice that in R2
(brighter-than) we have to be careful. The pair
(6 3) is a member of this relation, but (3 6) is
not. This corresponds to a directed graph. But
for R1 (adjacent), both (1 6) and (6 1) can be
considered member tuples. For compactness we
would only include one of them the other adds no
new information.