Parallel Programming in Artifitial Neural Networks

1
Parallel Programming in Artifitial Neural Networks

• By
• Harold Cruzado J.

2
Contents

• Overview of Artificial Neural Network (ANN)
• Why Parallelizing?
• Parallelizing by use MPI on C/C
• An perceptron, back propagation, some functions
  useful, phases in the training, C structure to
  MPI structure
• Conclusions

3
Artifitial Neural Network (ANN)

• The ANN (or net) is used to solve many problems
  sometimes that not have direct solutions.
• The ANN is composed by perceptrons ( neurons).
  Each percetron (also called neuron) have input
  output and an constant called bias

4
Why to parallelize?

• An ANN have a high time consuming in the process
  called training (also called learning ), which is
  achieved mean some algorithms, so as back
  propagation and others those algorithms involve
  the matrix operations.

5
An Perceptron and its training
Training theperceptron, lets t a target, then
- Forward phase (1) Calculate the actual
outputs - Backward phase (2) Adjust the
weights to reduce error
6
An net of three layers
7
Back Propagation Algorithm

1. Define Net
2. Initialize Net
3. Forward Propagate net
4. Backward Adjust weights on net
5. Send (from process root) new Net all process
6. If SSEgte repeat from step 3 to 6

8
Parallelizing by using MPI
9
forward
if SSEgte

backward
10
Function main
void main( int argc, char argv ) Ctype
declations... MPI_Init(argc, argv )
MPI_Comm_rank( MPI_COMM_WORLD, rank)
MPI_Comm_size( MPI_COMM_WORLD, size)
Ctype to MPItype Initializing the Net
Forward fase Backward fase
MPI_Finalize() exit(0)
11
Dartitioning data
The following function in C define the
corresponding part of an interval (for example,
indices of rows of the matrix ) for each process
if(iltr) I0i-1 I1i
if(igtr) I0r I1r
Void Partition(int n,int rank,int d,int I2)
int rnd,q drank1 q(int)n/d
I0q(i-1) I1qi
12
Product matrix-vector
void Ax(int Afc,int xc,int I2,int
rank,int size,int yf) int c,auxf
for(int iI0iltI1i) yi-I00 for(int
j0jlt4j) yi-I0Aijxj

13
C type structure to MPI type structure
typedef struct
double YN1
double DeltaN
double WeightsNN1 Int
bias LAYER
typedef struct LAYER LayerL
double TargetNN double
Eta double
SSE double
xDataRowsDataCols NET
14
C structure to MPI structure

• int blocklengths2
• MPI_Aint displacements2,double_extent,layer_ex
  tent
• MPI_Datatype dtypes2 MPI_DOUBLE,MPI_INT,
  mpi_layer
• MPI_Type_extent(MPI_DOUBLE,double_extent)
• displacements0 0
• displacements1 (processors(2N1)
  N(N1))double_extent
• blocklengths0processors(2N1)N(N1)
• blocklengths11
• MPI_Status status
• MPI_Type_struct(2, blocklengths,
  displacements, dtypes, mpi_layer)
• MPI_Type_commit(mpi_layer)

15
C structure to MPI structure

• MPI_Type_extent(mpi_layer,layer_extent)
• MPI_Datatype dtypes12 mpi_layer,MPI_DOUBLE
  , mpi_net
• displacements0 0
• displacements1 Llayer_extent
• blocklengths0L
• blocklengths12NNDataRowsDataColsprocesso
  rs
• MPI_Type_struct(2, blocklengths, displacements,
  dtypes1, mpi_net)
• MPI_Type_commit(mpi_net)

16
Initializing the Net
.. if(rank0) LoadData("TrainingData.txt"
,Rows,Cols,Net.X) SetBias(Net,size)
GenerateTarget(Net) RandomWeights(Net)
Net.Eta0.02 MPI_Bcast(Net,1,NET,0,MPI_COMW
ORLD) ..
17
Forward phase
.. Partition(f,rank,size,I,n_loc) for (int
i0iltL,i) Ax(Net,Layerl.Weights,Net.x,I,
rank,size,y_loc) if (rank0)
MPI_Gather(y_loc,n_loc,MPI_INT,Net.Layerl.y,n_lo
c, MPI_INT,0,MPI_COMM_WORLD)
if(rank0) MPI_Bcast(Net,1,NET,0,MPI_
COMWORLD) EvaluateSSE(Net,0,rank,size) ..
18
Backward phase
.. DeltaAdjust(Net,Net.Targetsi,rank,size)
WeightsAdjust(Net,rank,size) if(rank0)
MPI_Bcast(Net,1,NET,0,MPI_COMWORLD) ..
19
Conclusions

• The MPI is good alternative to implemented the
  parallel code
• It is possible to parallelize the serial code
  which use defined types by programmers

20
Questions?