Neural Network Implementations on Parallel Architectures

1
Neural Network Implementations on Parallel
Architectures
2
Index

• Neural Networks
• Learning in NN
• Parallelism
• Characteristics
• Mapping Schemes Architectures

3
Artificial Neural Networks

• inspired from human brain
• parallel,distributed computing model
• consists of a large number of simple,neuron-like
  processing elements called units
• weighted,directed connections between pairs of
  units.

4
Artificial Neural Networks-2

• weights may be positive or negative
• each unit computes a simple function of its
  inputs,which are weighted outputs from other
  units

5
Artificial Neural Networks-3

• a threshold value is used in each neuron to
  determine the activation of the output
• learning in NN
• finding the weights and threshold values
• training set
• multi-layer,feedforward networks input
  layer,hidden layer,output layer

6
Learning in ANN

• initialize all weights
• apply input vector to network
• propagate vector forward and obtain unit outputs
• compare output layer response with desired
  outputs
• compute and propagate error measure backward,
  correcting
• weights layer by layer
• iIterate until good mapping is achieved

7
Learning in ANN-2
8
Parallelism

• further speed-up of training
• neural networks exhibit high degree of
  parallelism(distributed set of units operating
  simultaneously)
• process of parallelism
• what type of machine?
• how to parallelize?

9
Parallelism-2

• different neural network models
• highly dependend on the model used
• SIMD(small computation a lot of data exchange)
• one neuron for one processor
• MIMD (distributed memory message passing)
• bad performance in frequent communication

10
Characteristics

• theoretical analysis of the inherent algorithm
• portability
• ease of use
• access to ANN model description

11
Historical Data Integration

• prediction of the sensor output
• two parallelism methods
• parallel calculation of weighted sum
• time increases with the number of processors
• parallel training of each seperate NN
• time decreases with the number of processors
• 8 RISC processors
• 4 MB cache memory
• 512 RAM

12
Method-1
13
Method-2
14
Distributed Training Data
15
A library on MIMD machines

• distributed shared memory

16
A library on MIMD machines-2

• several communication and syncronization schemes
• message passing or shared memory
• thread programming with shared memory has the
  best performance
• every data is shared but handled only by one
• processor
• training of a Kohonen map of 100100 neurons with
  100000 iterations with 8 processors are 7 times
  faster than the sequential execution.

17
A library on MIMD machines-3
18
AP1000 Architecture
19
AP1000 Architecture-2

• MIMD computer with distributed memory
• vertical slicing of the network
• 3 methods for communication
• One to one communication
• Rotated messages in horizontal and vertical rings
• Parallel routed messages
• different neural network implementations

20
AP1000 Architecture-3

• different mappings according to the network and
  the training data
• heuristic on training time
• combine multiple degrees of parallelism
• training set parallelism
• node parallelism
• pipelining parallelism

21
References

• APPROACH TO PARALLEL TRAINING OF INTEGRATION
  HISTORICAL DATA NEURAL NETWORKS, V. TURCHENKO1,
  C. TRIKI2, A. SACHENKO1
• A LIBRARY TO IMPLEMENT NEURAL NETWORKS ON MIMD
  MACHINES,Y.BONIFACE, F.ALEXANDRE,S. VIALLE
• A BRIDGE BETWEEN TWO PARADIGMS FOR PARALLELISM
  NEURAL NETWORKS AND GENERAL PURPOSE MIMD
  COMPUTERS, Y.BONIFACE, F.ALEXANDRE,S. VIALLE
• PARALLEL ENVIRONMENTS FOR IMPLEMENTING NEURAL
  NETWORKS, M. MISRA