Committee Update Building a visual hierarchy

1
Committee UpdateBuilding a visual hierarchy

• Andrew Smith
• 30 July 2008

2
Outline

• Confabulation theory
• Summary
• Comparisons to other AI techniques
• Human Visual System
• Building A Visual Hierarchy
• Learning
• Inference
• Texture modeling (applications)
• Future work (dissertation defence, Spring 2009)

3
Confabulation Theory

• A theory of the mechanism of thought
• Cortex/thalamus is divided into thousands of
  modules (1,000,000s of neurons).
• Each module contains a lexicon of symbols.
• Symbols are sparse populations (100s) of neurons
  within a module.
• Symbols are stable states of a cortex-thalamus
  attractor circuit.

4
Confabulation theory (1/4)

• Key concept 1
• Modules contain symbols, the atoms of our mental
  universe.
• Smell module Apple, flower, rotten,
• Word module rose the and it France
  Joe
• Abstract planning modules, etc.
• Modules are small patches of thalamocortical
  neurons.
• Each symbol is a sparse popuation of those
  neurons.

5
Confabulation theory (1/4)
6
Confabulation theory (2/4)

• Key concept 2
• All cognitive knowledge is knowledge links
  between these symbols.
• Smell module Apple, flower, rotten,
• Word module the and it France Joe
  apple
• Only symbols that are meaningfully co-occurring
  may become linked.

7
Confabulation theory (3/4)
8
Confabulation theory (3/4)

• Key concept 3
• A confabulation operation is the universal
  computational mechanism.
• Given evidence a, b, c pick answer x such that
• x argmaxx p(a, b, c x)
• We say x has maximum cogency.

9
Confabulation theory (3/4)

• Fundamental Theorem of Cognition1
• p(abgde)4 p(abgde)/p(ae)
• p(abgde)/p(be)
• p(abgde)/p(ge)
• p(abgde)/p(de)
• p(ae)p(be)p(ge)p(de)
• If the first four terms remain nearly constant
  w.r.t e, maximizing the fifth term maximizes
  cogency (the conditional joint).

10
Confabulation theory (3/4)
11
Confabulation theory (4/4)

• Key concept 4
• Each confabulation operation launches a control
  signal to other modules.
• Control mechanism of inference studied by
  others in the lab.
• (not here)

12
Similarities to other AI / ML

• Bayesian networks a special case
• A confabulation network is similar to a
  Bayesian Net with
• Symbolic variables (discrete finite exclusive
  state) with equal priors.
• Naïve-Bayes assumption for CP tables.
• Can use similar learning algorithms (counting for
  CPs)
• Hintons (unrestricted) Bolzman Machines
  generalized
• Do not require complete connectivity
• (many) more than two states.
• Can use stochastic (Monte Carlo) execution

13
Outline

• Confabulation theory
• Summary
• Comparisons to other AI techniques
• Human Visual System
• A Visual Hierarchy
• Learning
• Inference
• Texture modeling
• Future Work (i.e. my thesis)

14
Human Visual System

• Retina pixels
• Lateral Geniculate Nucleus (LGN)
• center-surround representation
• Primary() Visual cortex (V1 )
• Simple cells
• Hubel Weisel (1959)
• Modeled by Dennis Gabor features
• Complex cells
• more complicated (end-stops, bars, ???)
• Take inspiration for our first and second-level
  features

15
(No Transcript)
16
Outline

• Confabulation theory
• Summary
• Comparisons to other AI techniques
• Human Visual System
• Building A Visual Hierarchy
• Learning
• Inference
• Texture modeling
• Future Work (i.e. my thesis)

17
Confabulation vision

• Features (symbols) develop in a layer of the
  hierarchy as commonly seen inputs from their
  inputs.
• Knowledge links are simple conditional
  probabilities
• p(ae) where a and e are symbols in connected
  modules)
• All knowledge can therefore be learned by simple
  co-occurrence counting.
• p(ae) C(a,e) / C(e)

18
Building a vision hierarchy

• Can no longer use SSE to evaluate model
• Instead, make use of generative model
• Always be able to generate a plausible image.

19
Data set

• 4,300 1.5 Mpix natural images (BW)

20
Vision Hierarch level 0

• We know the first transformation from
  neuroscience research simple cells approximate
  Gabor filters.
• 5 scales, 16 orientations (odd even)

21
Vision Hierarch level 0

• Does the full convolution preserve information in
  images? (inverted by LS)
• Very closely.

22
Vision Hierarchy level 1

• We now have a simple-cell like representation.
• How to create a symbolic representation?
• Apply principle Collect common sets of inputs
  from simple cells similar to a Vector
  Quantizer.
• Keep the 5-scales separate
• (quantize 16-dimensions, not 80)

23
Vision Hierarchy level 1

• To create actual symbols, we use a vector
  quantizer
• Trade-offs (threshold of quantizer)
• Number of symbols Preservation of information
• Probability accuracy

• Solution Use angular distance metric
  (dot-product)
• Keep only symbols that occurred in training set
  more than 200 times, to get accurate p(ae).
• After training, 95 of samples should be within
  threshold of at least one symbol.
• Pick a threshold so images can be plausibly
  generated.

24
Vision Hierarchy level 1

• Oops!
• Ignoring wavelet
• magnitude makes all
• texture features
• equally prominent.

25
Vision Hierarchy level 1

• Solution, use binning (into 5 magnitudes), then
  apply vector quantizers).

26
Vision Hierarchy level 1

• 10,000 symbols are learned for each of the 5
  scales.
• Complex features develop.

27
Vision Hierarchy level 1

• Now image is re-represented as 5 planes of
  symbols

28
Outline

• Confabulation theory
• Summary
• Comparisons to other AI techniques
• Human Visual System
• Building A Visual Hierarchy
• Learning
• Inference
• Texture modeling
• Future Work (i.e. my thesis)

29
Texture modeling - Learning

• We can now represent an image as five
  superimposed grids of symbols.
• Transform data set
• Learn which symbols are typically next to which.
• (knowledge links)

30
Knowledge links

• Learn which symbols may be next to which symbols
  (conditional probabilities)
• Learn which symbols may be over/under which
  symbols.
• Go out to radius 5.

31
Texture modeling Inference 1

• What if a portion of our image symbol
  representation is damaged?
• Blind spot
• CCD defect
• brain lesion
• We can use confabulation (generation) to infer a
  plausible replacement.

32
Texture modeling Inference 1

• Fill in missing region by confabulating from
  lateral different scale neighbors (rad 5).

33
Texture modeling
34
Texture modeling
35
Texture modeling
36
Texture modeling

• Conclusions
• This visual hierarchy does an excellent job at
  capturing an image up to a certain order of
  complexity.
• Given this visual hierarchy and its learned
  knowledge links, missing regions could plausibly
  filled in. This could be a reasonable
  explanation for what animals do.

37
Texture modeling Inference 2

• Super-resolution
• If we have a low resolution image, can we
  confabulate (generate) a high-resolution version?
• Space out the symbols, and confabulate values
  for the new neighbors

38
Texture modeling
39
Texture modeling
40
Texture modeling

• Super-resolution conclusions
• Having learned the statistics of natural images,
  the generative properties of this hierarchy can
  confabulate (generate) plausible high-resolution
  versions of its input.

41
Outline

• Confabulation theory
• Summary
• Comparisons to other AI techniques
• Human Visual System
• Building A Visual Hierarchy
• Learning
• Inference
• Texture modeling
• Future Work (Dissertation)

42
The next level

• Level 2 symbol hierarchy
• Collect commonly recurring regions of level 1
  symbols.
• Symbols at Level 2 will fit together like puzzle
  pieces.

Thank you!