Neuromorphic Object Detection Recognition and Tracking

1
Neuromorphic Object Detection Recognition and
Tracking

• Fopefolu Folowosele
• TELL Research Overview
• September 17, 2009

2
Outline

• Introduction to Neuromorphic Engineering
• Computational Sensory Motor Systems Lab
• Research Focus
• Approach
• Neural Array Transceivers
• HMAX Model of Object Recognition
• Neural Algorithms
• Conclusion

3
Neuromorphic Engineering

• Neuromorphic was coined by Carver Mead to
  describe VLSI systems containing circuits that
  mimic neuro-biological architectures present in
  the nervous system

• Neuromorphic Engineering involves designing
  artificial neural systems whose physical
  architecture and design principles are based on
  those of biological nervous systems

4
Computational Sensory Motor Systems Laboratory
5
Research Focus

• Object detection, recognition and tracking are
  computationally difficult tasks
• Primates excel at these tasks
• Engineered systems are unable to match their
  level of proficiency, flexibility and speed
• Intelligent robots need to be able to interact
  with their surroundings with limited human
  involvement
• This interaction involves
• Detecting the presence of the object
• Recognizing the object
• Tracking the trajectory of the object

Agile Systems, 2009
Hyperexperience 2008
6
Approach

• Our overall goal is to work towards developing a
  real-time autonomous intelligent system that can
  detect, recognize and track objects under various
  viewing conditions
• Emulate cortical functions of primates to design
  more intelligent artificial systems
• Mimic the visual information processing of the
  primates visual system
• Model computationally-intensive algorithms in
  neural hardware

Population Surveillance and Visual Search Engines
Research Tool for Neuroscientists
Visual Prosthesis and Ocular Implants
Techarena 2009 Future Predictions 2008 R.
Friendman, Biomedical Computation Review 2009
7
Projected Contributions

• Develop a spike-based processing platform on
  which we can demonstrate object detection,
  recognition and tracking
• Design the next generation neural array
  transceiver
• Realize silicon facsimiles of cortical simple
  cells, complex cells and composite feature cells
• Implement neural algorithms analogous to
  cross-correlation and Kalman filtering for object
  detection and tracking respectively

8
Software vs. Hardware Models
Software models run slower than real time and are
unable to interact with the environment
Silicon designs take a few months to be
fabricated, after which they are constrained by
limited flexibility
IBM 2004 Tenore 2008
9
Solution Reconfigurable Models

• Neural array transceivers are reconfigurable
  systems consisting of large arrays of silicon
  neurons
• Useful for studying real-time operations of
  cortical, large-scale neural networks
• Able to leverage the known fundamental blocks
  such as the operation of neurons and synapses
• Flexible enough for testing out unknowns

10
Neuro-Computational Spiking and Bursting Models
E.M. Izhikevich, Neural Networks, 2004

• Izhikevichs model seems most appropriate but
• Result of highly nonlinear curve fitting offering
  little insight into the underlying biological
  mechanism
• Interference between the state variables
• Parameters are 4-5 orders of magnitude apart

• Mihalas-Niebur Neuron Model
• Generalized version of the leaky
  integrate-and-fire model with adaptive threshold
• More biologically relevant
• Suggested modifications to the threshold
  interpreted as nonlinear voltage dependent
  channels

11
Mihalas-Niebur Neuron Simulations

• Circuit

• MatLab

F. Folowosele et al., ISCAS, 2009
12
3D Design
In collaboration with the Sensory Communication
and Microsystems Lab
13
Visual Pathways

• Primary Visual Cortex V1 transmits information to
  two primary pathways
• Dorsal stream
• Ventral stream
• Dorsal pathway is associated with motion
• Ventral pathway mediates the visual
  identification of objects

T. Poggio, NIPS, 2007 Wikipedia, The Free
Encyclopedia
14
HMAX

• Summarizes and integrates large amount of data
  from different levels of understanding (from
  biophysics to physiology to behavior)
• Two main operations occur in the model
• Gaussian-like tuning operation in the S layers
• Nonlinear MAX-like operation in the C layers

M. Riesenhuber T. Poggio, Nature Neuroscience
1999
15
Preliminary Results S1 and C1 Stages

• S1 neurons are oriented spatial filters that
  detect local changes in contrast
• S1 cell integrates inputs from a 4x1 retinal
  receptive field
• C1 neurons take the MAX of similarly-oriented
  simple cells over a region of space
• C1 cell integrates inputs from an array of 5x5
  similarly-oriented S1 cells

F. Folowosele et al., BioCAS 2008
16
MAX Operation

• Nonlinear saturating pooling function on a set of
  inputs, such that the output codes the amplitude
  of the largest input regardless of the strength
  and number of the other inputs
• Set of input neurons X causes the output Z to
  generate spikes at a rate proportional to the
  input with the fastest firing rate

R.J. Vogelstein et. al, NIPS 2007
17
Test1 Test Images and Resulting Simple Cells

• (A1-4) Generated test images
• (B1-4) Horizontally-oriented simple cells that
  respond to light-to-dark transitions
• (C1-4) Vertically-oriented simple cells that
  respond to dark-to-light transitions

F. Folowosele et al., ISCAS 2007
18
Test 1 MAX Network Computation Results

• The ratio k obtained is approximately constant
  among all the simple cells, with a mean of 0.068
  and a standard deviation of 0.0006

F. Folowosele et al., ISCAS 2007
19
Neural Algorithms

• In computer vision, object detection and tracking
  algorithms are computationally-intensive
  processes
• Cross-correlation for object and pattern
  detection
• Kalman and particle filtering for object tracking
• Neural-based algorithms are potentially more
  flexible and less computationally-intensive than
  their traditional counterparts

20
Object Detection

• Computation of cross correlation is utilized for
  pattern and object detection
• Basis for neural cross-correlation is the
  autocorrelative nature of the interspike interval
  histogram (ISIH) in spiking neurons
• First proposed in 1951 (Licklider 1951)
• ISIH for the auditory nerve ensemble response has
  same shape as the autocorrelation function
  (Cariani 1996)
• Autocorrelation observed in accumulated output
  from a single integrate-and-fire neuron (Tapson
  1998)

21
Neural Cross-Correlation

• The neural cross-correlation engine was proposed
  by Jonathan Tapson in 2007
• It utilizes integrate-and-fire neurons to produce
  cross-correlation information in a novel way

J. Tapson R. Etienne-Cummings ISCAS 2007
22
MatLab Simulation Results

• ISIH as a proxy for correlation

Mathematical Computation
Interspike Interval (ISIH)
F. Folowosele et al., SPIE 2007
23
MatLab Simulation Results

• Extracting the phase information of a signal

Mathematical Computation
Interspike Interval (ISIH)
F. Folowosele et al., SPIE 2007
24
Object Tracking

• Optimal solution to tracking tasks is Kalman
  filtering
• Core function of cerebral cortex hypothesized to
  involve some mechanism of Kalman filtering
• Kalman filtering type algorithm used in
  hierarchical generative model for visual
  recognition (Rao Ballard 1997)
• At each hierarchical level
• Predicts current visual state at a lower level
• Adapts own recognition state using the residual
  error between the prediction and the actual
  lower-level state

25
Kalman Filtering

• Models natural processes in the external world
  using a stochastic linear differential equation
• External system is described by a state vector
• Each measurement vector satisfies
• Organisms do not have access to the internal
  states of the world causing their sensory
  experiences

R. Rao, Uni. Of Rochester 1996 T. Lacey, Georgia
Tech. 1998
26
Neural Kalman Algorithm

• Ralph Linskers Algorithm
• Utilizes recurrent neural network composed of
  linear-response nodes
• Requires noisy measurement data as only input
• Classical Kalman Approach
• Estimate state vector
• Neural Kalman Approach
• Estimate measurement vector

27
Simple Tracking Example 2D Plant State
28
Conclusion

• We are designing a neural array transceiver on
  which we intend to implement
• Stages of the HMAX hierarchical model of object
  recognition
• Neural algorithms for object detection and
  tracking
• Our overall goal is to work towards developing a
  real-time autonomous intelligent system with an
  artificial visual cortex

29
Acknowledgments

• Prof. Ralph Etienne-Cummings
• Computational Sensory Motor Systems Lab Members
• Sensory Communications and Microsystems Lab
  Members
• Collaborators
• Jonathan Tapson (University of Cape Town)
• Tara Hamilton (University of Queensland)
• Ernst Niebur Stefan Mihalas (JHU Mind-Brain
  Institute)
• UNCF-Merck

30
References

• R.R. Murphy and E. Rogers, Cooperative
  assistance for remote robot supervision,
  Presence Teleoperators and Virtual Environments
  Journal, vol. 5, no. 2, pp. 224-240, 1996.
• T. Serre, M. Kouh, C. Cadieu, U. Knoblich, G.
  Kreiman, and T. Poggio, A theory of object
  recognition computations and circuits in the
  feedforward path of the ventral stream in primate
  visual cortex, AI Memo, MIT, Cambridge 2005.
• M. Riesenhuber, and T. Poggio, Computational
  models of object recognition in cortex a
  review, Technical Report Artificial Intelligence
  Laboratory and Department of Brain and Cognitive
  Sciences, Massachusetts Institute of Technology,
  2000b.
• R.J. Vogelstein, U. Mallik, E. Culurciello, G.
  Cauwenberghs, R. Etienne-Cummings, A multichip
  neuromorphic system for spike-based visual
  information processing, Neural Computation, vol.
  19, pp. 2281-2300, 2007a.
• D.H. Goldberg, G. Cauwenberghs, and A.G. Andreou,
  Probabilistic synaptic weighting in a
  reconfigurable network of VLSI integrate-and-fire
  neurons, Neural Networks, vol. 14, pp. 781-793,
  2001.
• T.Y.W. Choi, P.A. Merolla, J.V. Arthur, K.A.
  Boahen, and B.E. Shi, Neuromorphic
  implementation of orientation hypercolumns,
  IEEE ISCAS 2005.
• R.J. Vogelstein, U. Mallik, J.T. Vogelstein, G.
  Cauwenberghs, Dynamically reconfigurable silicon
  array of spiking neurons with conductance-based
  synapses, IEEE Transactions on Neural Networks,
  2007b.
• A. Cassidy, S. Denham, P. Kanold, and A.G.
  Andreou, FPGA-based silicon spiking neural
  array, IEEE BioCAS 2007.
• B. E. Shi, E. K. C. Tsang, S. Y. M. Lam and Y.
  Meng, "Expandable hardware for computing cortical
  maps," IEEE ISCAS 2006.
• D.H. Hubel and T.N. Wiesel, Receptive fields,
  binocular interaction and functional architecture
  in the cat's visual cortex, Journal of
  Physiology, vol. 160, no. 1, 1962.
• L.G. Ungerleider, and J.V. Haxby, What and where
  in the human brain, Curr. Opin. Neurobiol., pp.
  157-165, 1994.
• E. Rolls and T. Milward, A model of invariant
  object recognition in the visual system Learning
  rules, activation functions, lateral inhibition,
  and information-based performance measures,
  Neural Computation, vol. 12, pp. 2547-2572, 2000.
  
• P Merolla and K Boahen, A recurrent model of
  orientation maps with simple and complex cells,
  Advances in Neural Information Processing Systems
  (NIPS) 16, S Thrun and L Saul, Eds, MIT Press, pp
  995-1002, 2004.
• R.P.N. Rao, Robut Kalman filters for prediction,
  recognition, and learning, Technical Report 645,
  Computer Science Department, University of
  Rochester, 1996.
• J. Licklider, A duplex theory of pitch
  perception, Cellular and Molecular Life Sciences
  (CMLS), vol. 7, no. 4, pp. 128-134, 1951
• J. Tapson, Autocorrelation properties of single
  neurons, Proceedings of the 1998 South African
  Symposium on Communication and Signal Processing,
  1998.
• J. Tapson, C. Jin, A. van Schaik and R.
  Etienne-Cummings, A First-Order Nonhomogeneous
  Markov Model for the Response of Spiking Neurons
  Stimulated by Small Phase-Continuous Signals,
  Neural Computation, vol. 21, no. 6, pp.
  1554-1588, June 2009.
• T. Lacey, Tutorial The Kalman filter, Lecure
  Notes, Department of Computer Science, Georgia
  Institute of Technology, 1998.
• R. Linsker, Neural network learning of optimal
  Kalman prediction and control, Neural Networks,
  vol. 21, no. 9, pp. 1328-1343, 2008.

31
Figure References

• Self-driving Car
• http//hyperexperience.com/wp-content/uploads/200
  8/01/nissanpivot.jpg
• Space-station Builder http//www.agilesystems.com
  /images/happy20space20rigger2.jpg
• Population Surveillance Visual Search Engine
  http//gallery.techarena.in/data/516/New-visual-se
  arch-engine-TinEye.jpg
• Visual Prosthesis Ocular Implants
  http//futurepredictions.files.wordpress.com/2008/
  12/080129-bionic-eye_big.jpg
• Research Tool
• R. Friedman, Reverse engineering the brain,
  Biomedical Computation Review, vol. 5, no. 2, pp.
  10-17, Spring 2009
• Blue Gene
• http//agix.files.wordpress.com/2008/11/blue_gen
  e.jpg
• Hardware Chip
• http//etienne.ece.jhu.edu/ftenore/research/publ
  ications/tenore_biocas08.pdf

32
Relevant Publications

• F. Folowosele, T.J. Hamilton, A. Harrison, A.
  Cassidy, A.G. Andreou, S. Mihalas, E. Niebur and
  R. Etienne-Cummings, A Switched Capacitor
  Implementation of the Generalized Linear
  Integrate-and-Fire Neuron, Submitted to
  Proceedings of IEEE International Symposium on
  Circuits and Systems (ISCAS), 2009
• F. Folowosele, R.J. Vogelstein, and R.
  Etienne-Cummings, Real-Time Silicon
  Implementation of V1 in Hierarchical Visual
  Information Processing, Proceedings of IEEE
  Biomedical Circuits and Systems Conference
  (BioCAS), Baltimore, Maryland, November 2008.
• S. Chen, F. Folowosele, D. Kim, R.J. Vogelstein,
  E. Culurciello and R. Etienne-Cummings, Size and
  Position Invariant Human Posture Recognition
  Algorithm with Spike-Based Image Sensor
  Proceedings of IEEE Biomedical Circuits and
  Systems Conference (BioCAS), Baltimore, Maryland,
  November 2008.
• F. Folowosele, F. Tenore, A. Russell, G. Orchard,
  M. Vismer, J. Tapson, and R. Etienne-Cummings,
  Implementing a Neuromorphic Cross-Correlation
  Engine with Silicon Neurons, Proceedings of IEEE
  International Symposium on Circuits and Systems
  (ISCAS), Seattle, Washington, May 2008.
• J. Tapson, M.P. Vismer, C. Jin, A van Schaik, F.
  Folowosele, and R. Etienne-Cummings, A
  Two-Neuron Cross-Correlation Circuit with a Wide
  and Continuous Range of Time Delay, Proceedings
  of IEEE International Symposium on Circuits and
  Systems (ISCAS), Seattle, Washington, May 2008.
• F. Folowosele, R.J. Vogelstein, R.
  Etienne-Cummings, Spike-Based MAX Network for
  Nonlinear Pooling in Hierarchical Vision
  Processing, Proceedings of IEEE Biomedical
  Circuits and Systems Conference (BioCAS),
  Montreal, Canada, November 2007.
• F. Folowosele, J. Tapson, R. Etienne-Cummings, A
  Wireless Address Event Representation System for
  Biological Sensor Networks, Proc. SPIE
  (Bioengineered and Bioinspired Systems), 2007.