PGP Project

1
PGP Project
Viktor Yarmolenko Lewis Mackenzie Paul
Cockshott Ewan Borland
2
Contents

• Background
• Motivation
• Proposed Solution
• Status
• Questions

3
Background
Setup
4
Background
Algorithm
View from 3 cameras
5
Background
Algorithm
X warp
Y warp
Correlation
6
Background
Algorithm

Reconstructed model viewed from a Virtual
perspective position.
7
Background
The problem
Highly parallelizable problem the following can
be done independently

• 8 Matching jobs per group
• N process groups per sequence
• 1min on Matching
• 1min on Building (2GHz)

8
Motivation
Dynamic data graphs
S
M
1
S
M
1
T
9
Motivation
Dynamic data graphs
S
M
1
S
M
B
1
1
T
10
Motivation
Dynamic data graphs
S
M
M
2
M
2
1
S
M
B
1
1
T
11
Motivation
Dynamic data graphs
S
M
M
2
2
S
B
1
T
12
Motivation
The search for silver bullet
Our Aim

1. To develop a highly parallel system for 3D
   computer vision algorithms,
2. Which also can be a general framework for
   distributed processing.

Primary Requirements

• Allows dynamic creation of processes (arbitrary
  code)
• Allows creation of communications channels
  between processes.
• Allows channels to be dynamically reconfigurable.

13
Motivation
The search for silver bullet
GRIDDynamicFlexibleJava...

• Use of GRID as a transport layer sounds good
• The requirements could not be readily met by
  existing GRID protocols
• No current API (MPI, RMI, PVM, DSM) can readily
  meet the requirements
• We need a conceptually new parallel architecture

J
p

14
Solution
What is J??
J? is a Java interface loosely modelled on the
primitives of the ?-calculus (Milner) to be used
as a substratum for GRID based parallel computing.
The key concepts taken from Milners calculus are
the ability to dynamically create processes and
communication channels and to transmit
communication channels along other channels.

• "The Polyadic ?-calculus, a tutorial, Milner,
  R., (1991)
• A calculus of mobile processes, Milner, R.,
  Parrow, J., Walker, D., Information and
  Computation, 1001-77, (1992)

(Milner)
15
Solution
J? primitives
JPieTask implements Runnable JPieFunnel
extends OutputStream JPieTap extends
InputStream JPiePipe contains connected
JPieTap and JPieFunnel
1000km
16
Solution
J? example
HDD
VM
VM
VM
17
ILS model
Initiator Locator Servent
Locator currently at EPCC Runs web
services Maintains MySQL database of servents

Reply with Available servents
Register With locator
Servent runs JPie daemon
Query locator
Initiator which Starts job
Any servent can also become an initiator during
the course of a computation and spawn more tasks

18
ILS model
Initiator Locator Servent
Locator keeps track of available memory and
performance of servents
Reply with list of suitalble servents
Register performance and memory
Servent
Queries locator For machines of Above
performance Level X
Java Grande Benchmark used To rate
Servent performance
Initiator
Starts job

19
Status
Need Your Thoughts

• Performed network tests using the data demanding
  part of the algorithm. Data transfer is not a
  bottleneck in this problem.
• Completed the multi processor implementation of
  J?, using sockets.
• Currently installing resource locator using Web
  services to find free cpus.
• Would like to implement J? transport using a GRID
  layer.

Acknowledgements
NeSC Sponsors of the project EPCC Discussions
20
Examples
Conformed sequence
21
Examples
Conformed mesh
22
Examples
Landmark grid
23
Examples
Textured
24
Examples
Textured mesh
25
Stream down the stream in pictures
HDD
VM
VM
VM
26
Preliminary Tests

• A part of algorithm was used
• Total data transfer is over 5GB
• Processed 12 sec of 3D video
• That is 3600 images (12?25?4?3)
• The bandwidth at its bottleneck 100Mbits
• Virtually theoretical speedup
• Can be improved by using J?

Exp Total CPUs LocalCPUs Remote CPUs Time (min) Comments
0 4 4 0 215 Four local PCs
1 12 12 0 75 12 local PCs (4 Dual 4 Single, in the same room)
2 16 0 16 48 16 remote PCs (16xCPU IBM machine 60 miles away)
3 28 12 16 29 16 remote 12 local
4 43 12 31 21 16 remote 12 local 15 other PCs from the department