Neuromorphic Image sensors

1
Neuromorphic Image sensors
Eugenio Culurciello Yale University EENG427 Lesso
n 8
2
Biomimetic Circuits

• Taking hints from nature
• How does nature solve everyday problems
• Can we implement natures solutions?
• in Silicon?

3
Biomimetic Circuits

• Human Eye a wonderful machine
• Small and light 1 inch, 7 grams
• Retina neural sensor network, rods and cones
• Optic nerve carries digital signals to the brain

http//webvision.med.utah.edu/anatomy.html
4
Structure of the Eye
5
The Retina
6
Biomimetic Circuits

• Dynamic Range 10 orders of magnitude
• Bandwidth 100M sensors, 1M fibers in optic nerve
• Specialization
• Cones in color, high resolution - fovea
• Rods in the dark / motion

http//webvision.med.utah.edu/anatomy.html
7
Digital Cameras and Si Eye

• Everyone wants silicon eyes!
• Small
• Light
• Acute now gt 1Mpixel
• Must work in
• Dim restaurant
• Outside BBQ
• Long life Low power
• Almost like a human eye!

http//www.panasonic.com.au/product_pdf/EB-X70.pdf
8
Neuromorphic Image Sensors
Smart-sensors are those devices in which the
sensors and circuits co-exist, and their
relationship with each other and with
higher-level processing layers goes beyond the
meaning of transduction. Smart-sensors are
information sensors, not transducers and signal
processing elements. Smart sensors are not
general purpose devices. Everything in a smart
sensor is specificaly designed for the
application targeted for.
9
Neuromorphic Image Sensors
When compared to a vision processing system
consisting of a camera and a digital processor, a
vision chip provides many system level
advantages. Speed The processing speed
achievable using vision chips exceeds that of the
camera-processor combination. A main reason is
the information transfer bottleneck between the
imager and the processor. In vision chips
information between various levels of processing
is processed and transferred in parallel. Large
dynamic range Many vision chips use
photodetectors and photocircuits which have a
large dynamic range over at least 7 decades of
light intensity. Many also have local and global
adaptation capabilities which further enhances
their dynamic range. Conventional cameras are at
best able to perform global automatic gain
control.
10
Neuromorphic Image Sensors
Size Using single chip implementation of vision
processing algorithms, very compact systems can
be realized. The only parts of the system that
may not be scalable are the mechanical parts
(like the optical interface). Power dissipation
Vision chips often use analog circuits which
operate in subthreshold region. There is also no
energy spent for transferring information from
one level of processing to another level. System
integration Vision chips may comprise most
modules, such as image acquisition, and low level
and high level analog/digital image processing,
necessary for designing a vision system. From a
system design perspective this is a great
advantage over camera-processor option.
11
Neuromorphic Image Sensors
Although designing single-chip vision systems is
an attractive idea, it faces several limitations
Reliability of processing Vision chips are
designed based on the concept that analog VLSI
systems with low precision are sufficient for
implementing many low level vision algorithms.
The precision in analog VLSI systems is affected
by many factors, which are not usually
controllable. As a result, if the algorithm does
not account for these inaccuracies, the
processing reliability may be severely affected.
Vision chips also use unconventional analog
circuits which may not be well characterized and
understood. Resolution In vision chips each
pixel includes a photocircuit which occupies a
large proportion of the pixel area. Therefore,
vision chips have a low fill-factor and a low
resolution. The largest vision chip reported has
only 210 230 pixels, for a photocircuit
consisting of six transistors only.
12
Neuromorphic Image Sensors
Difficulty of the design Vision chips implement
a specific algorithm in a limited silicon area.
Therefore, often off-the-shelf circuits cannot be
used in the implementation. This involves
designing many new analog circuits. Vision chips
are always full custom designed, and full custom
design is known to be time consuming and
error-prone. Programming None of the vision
chips are general purpose. In other words, many
vision chips are not programmable to perform
different vision tasks. This inflexibility is
particularly undesirable during the development
of a vision system.
13
Mahowald, Mead's silicon retina
Mahowald's silicon retina chip is among the first
vision chips which implemented a biological facet
of vision on silicon. The computation performed
by Mahowald's silicon retina is based on models
of computation in distal layers of the vertebrate
retina, which include the cones, the horizontal
cells, and the bipolar cells. The cones are the
light detectors. The horizontal cells average the
outputs of the cones spatially and temporally.
Bipolar cells detect the difference between the
averaged output of the horizontal cells and the
input.
14
PASIC sensor from Linköping University The
Processor ADC and Sensor Integrated Circuit''
(PASIC) as the name suggests consists of a sensor
array, A/D converters, and processors. Each
column has its own ADC and processor.
15
Andreou and Boahen's silicon retina
This silicon retina is an implementation of the
outer-plexiform of retinal processing layers. The
design has a distinctive feature that separates
it from all other silicon retinas. The
implementation uses a very compact circuit, which
has enabled the realization of a 210 x 230 array
of image sensors and processing elements with
about 590,000 transistors, which is the largest
among all reported vision chips. This silicon
retina uses a diffusive smoothing network. The
function of this one-dimensional network can be
written as                                    
  dQn/dt is the current supplied by the network
to node n, and D is the diffusion constant of the
network, which depends on the transistor
parameters, and the voltage     .
16
Andreou and Boahen's silicon retina
The function of the network can be approximated
by the biharmonic equation                      
                                 
                                              
where g and h are proportional to the the
diffusivity of the upper and lower smoothing
layers, respectively. More details about the
function of the circuit can be found in relevant
references. All the 2D chips use a hexagonal
network with six neighborhood connection. The
largest chip occupies an area of 9.5x9.3mm, in a
1.2um CMOS process with two layers of metal and
poly. A cell size of about 40x40um has been
achieved for this implementation. Under typical
conditions the chip dissipates 50mW.
17
Andreou and Boahen's silicon retina
Andreou and Boahen have encapsulated the model of
the retina in a neat and small circuit (below).
This circuit includes two layers of the diffusive
network. The upper layer corresponds to
horizontal cells in retina and the lower layer to
cones. Horizontal N-channel transistors model
chemical synapses.
18
Biomimetic Circuits

• What would it take to reproduce the human eye in
  Si?

3D Fabrication Process High Connectivity
http//www.nips.cc/Web/Groups/NIPS/NIPS2000/00pape
rs-pub-on-web/KurinoNakagawaLeeNakamuraYamadaParkK
oyanagi.pdf
19
Biomimetic Circuits
And with a conventional process? NEURONS
Advantage IN SPACE Neurons in the human brain
make up to 105 connections with their neighbors
CIRCUITS Advantage IN TIME Integrated circuits
handle communication cycles six orders of
magnitude smaller than the inter-event interval
for a single neuron or cell
20
Conventional Image Sensors

• Integrate light on a capacitor for a fixed time
• Sample the analog capacitor voltage
• Pixels are synchronously scanned

21
Address-Event Image Sensors

• Measure the time to integrate to a fixed voltage
• Light triggers a digital event
• Integrate (to threshold) and fire

Event Driven!
22
Pixel Operation

• Photocurrent is integrated on a 0.1pF capacitor.
  Slew Rate of 0.1V/ms in typical indoor light of
  0.1mW/cm2
• Pixel is reset to Vdd_r
• While integrating light, the voltage on the
  capacitor will decrease down to the threshold of
  the inverter

23
Pixel Operation

• The switching current of the inverter is fed back
  by a current mirror to sharpen the transition.
  The integrating capacitor is disconnected to
  minimize power consumption during reset.
• Reduced power consumption when compared to an
  inverter
• Slew rate gain

24
Pixel Operation
25
Pixel Operation

• Equation of the switching point (voltage)

• In time domain

26
Address-Event

• Address-Event Representation asynchronous
  protocol for communication between large arrays
• The AER model trades complexity in wiring of the
  biological systems for processing speed of
  integrated circuits

27
Address-Event Architecture
28
Address-Event Architecture
29
Sample Images from Sensor
100k samples
10k samples
Inter-Event Image
Histogram Image
30
Chip layout
E. Culurciello, R. Etienne-Cummings, K. A.
Boahen, A Biomorphic Digital Image Sensor',
IEEE Journal of Solid-State Circuits, Vol. 38,
No. 2, February 2003.
31
Sensor Performance