On Implementing Reservoir Computing

1
On Implementing Reservoir Computing

• Benjamin Schrauwen
• Electronics and Information Systems Department
• Ghent University Belgium
• December 9 2006 - NIPS 2006

2
Outline

• Introduction
• Software Reservoir Computing Toolbox
• Hardware Digital spiking neurons
• Future hardware
• Conclusions

3
Introduction

• LSM, ESN, BPDC, SDN, are all the same concept,
  just use different nodes and topologies
  Reservoir Computing
• How to evaluate RC performance across node types?
• Opensource MATLAB toolbox for reservoir computing
  research
• A box of tools examples a large scale
  explorer
• Because all techniques in single flow able to
  focus on specific influence of
• Topology
• Node type
• Reservoir adaptation

4
Reservoir Computing Toolbox

• Generic way to construct topologies and weight
  scaling
• Various node types supported linear, TLG, tanh,
  fermi, spiking (LIF, synapse models, dynamic
  synapses)
• Event based simulator for spiking neurons
  ESSpiNN
• Supports batching for large datasets
• Currently focused on off-line training (on-line
  in construction)
• Resampling and post-processing pipeline
• Linear, ridge-regression, non-linear readout
• Cross-validation, grid-search
• Reservoir adaptation

5
The RC Toolbox
Input data generation
Topology
Adaptation
ESSpiNN (CSIM)
Simulation
Readout pipeline
Cross-val/grid
6
The RC Toolbox topology
Connection structure
Rewiring
Assign weights
Scaling
7
The RC Toolbox readout
Spatial non-linearity
Filtering/mean
Sp./temp. non-linearity
Scoring
8
The RC Toolbox
SOON
http//www.elis.UGent.be/rct
9
Hardware

• Hardware advantages of RC
• Sparse/local connectivity is good
• Random weights are allowed
• (mild) node and network chaos can be taken
  advantage of
• Weights are fixed or can only change locally with
  RA
• Various HW implementations possible
• Spiking/analog/non-linear
• Digital/aVLSI/

10
Digital spiking neurons

• SNN mathematically a more complex model than ANN
• But better implementable in hardware
• No weight multiplications table look-up
• Filtering can be implemented using shifts and
  adds
• Interconnection only single bit, and sparse
  communication
• Asynchronous communication easily implementable

11
Digital spiking neurons

• Hardware can take advantage of parallelism
• But area-speed trade-off we dont have to make
  the implementation faster than needed by the
  application
• For trade-off different implementations with
  other area-speed needed
• Possible parallelisms
• Network parallelism
• Neuron/synapse parallelism
• Arithmetic parallelism
• We implemented
• SPPA network parallel, neuron serial, arithmetic
  parallel
• PPSA network parallel, neuron parallel,
  arithmetic serial
• SPSA network serial or parallel, neuron serial,
  arithmetic serial

12
Digital spiking neurons PPSA
13
Digital spiking neurons SPPA
14
Digital spiking neurons SPSA
15
Results
sppa
spsa
ppsa
Number of inputs per neuron
16
Area-speed trade-off for speech task

• Speech task in hardware
• LSM with 200 neurons
• 12 kHz processing speed
• Real-time requirement

LUTs memory Real-time
SPPA 13812 900 kbit 347
PPSA 13426 58 kbit 205
SPSA 10PE 488 144 kbit 2.2
SPSA 5PE 489 144 kbit 1.1
SPSA 1PE 489 144 kbit 0.23
17
Digital spiking neurons and RCT

• Topology can be exported from RCT to different HW
  models
• Exploration in SW ? export to HW for deployment
• Basic HW simulation model in RCT

18
Intermezzo some science

• Most valuable resource in hardware long
  connections
• Impact for RC readout is hardest part
• Solution only do partial readout
• What is performance penalty of this?

19
Intermezzo some science

• Most valuable resource in hardware long
  connections
• Impact for RC readout is hardest part
• Solution only do partial readout
• What is performance penalty of this?

Moore-Penrose pseudo inverse
20
Intermezzo some science

• Most valuable resource in hardware long
  connections
• Impact for RC readout is hardest part
• Solution only do partial readout
• What is performance penalty of this?

Ridge regression Tikhonov regularization
21
Intermezzo some science

• Most valuable resource in hardware long
  connections
• Impact for RC readout is hardest part
• Solution only do partial readout
• What is performance penalty of this?

22
Future parallel event based
23
Future parallel event based
24
Future parallel event based

• Network communication needs to be minimized
• Best for networks with much local and few global
  connections
• High speed-up possible due to
• Event based
• Parallel
• Hardware implementation

25
Future CNN

• Cellular Neural/Non-linear Network as reservoir
• Outlook
• Very fast, analog non-linear network with only
  nearest-neighbor connections (128x128)
• Analog computer instruction flow possible that
  implements reservoir and full parallel read-out
• Intrinsically random connections corrections
  needed when deterministic computations on CNN
• Parallel image input via CCD layer
• With Samuel Xavier de Souza and Johan Suykens
  from KULeuven
• On ACE16k_v2 chip from AnaFocus

26
Future photonic
Photonics is the science and technology of
generating, controlling, and detecting photons,
particularly in the visible light and near
infra-red spectrum Wikipedia.org

• Currently mainly focused on communication
• Long standing photonicist dream photonic
  computing
• Problems
• Feature size at least order of wavelength (1µm)
• Implementing memory is complex
• Change light with light only possible through
  medium slow
• Laser is intrinsically non-linear/chaotic
• Problems with fabrication variances

27
Future photonic

• Possible implementation of reservoir photonic
  crystal
• Semi-crystal fabricated on silicon to affect the
  path of light
• Creates stop band where light of given bandwidth
  cant exist
• Light can be bend in any direction
• Single crystal flaw can be a laser

28
Future photonic

• Idea use photonics to implement a reservoir
• Why
• Nodes (lasers) intrinsically non-linear/chaotic
• Possibly very fast (ps timescale)
• Full parallel readout and linear regression
  trivial
• Random (but fixed) process variation is
  allowed/desired
• Research project recently started together with
  Roel Baets and Peter Bienstman from photonics lab
  at Ghent University

29
Future photonic
30
Future photonic

• Possible applications
• Full optical signal reconstruction in optical
  communication
• Optical image processing
• Very high speed signal processing
• Questions/problems
• Harness laser chaos or use it to our advantage
• Information in light in multiple physical
  properties energy, polarisation, EM field,

31
Conclusions

• The reservoir computing concept is very suited
  for hardware implementation
• or no much hardware is very suited to be used
  as a reservoir