Stream-Components:

1
Stream-Components Component based
computation on the Grid
Paul Martinaitis and Andrew Wendelborn School of
Computer Science, University of Adelaide, South
Australia. (paulm,andrew_at_cs.adelaide.edu.au
2
Introduction

• StreamComponents Stream processing using
  components.
• Stream comprises stream generator and transducers
  e.g.
• Any serializable object can be a stream-value.
• The stream is to be highly re-configurable. In
  particular
• Compute codes can be added/changed by the Client
  at anytime.
• This separates the topology of a Stream from its
  functional aspects.
• The implementation uses Proactive Grid Components
  (Fractive).
• We begin by looking at the structure of a
  StreamComponent application

3
A generic Stream Component Application
The Stream (red) component supplies values and is
therefore at the start of the Stream.
StreamF (green) components are stream
transducers which is to say that they transform
the input values by applying a (unary) function
to the input values.

• Shows StreamComponents (SG1, f2 .. f5)
  distributed across three hosts (H1 .. H3) with
  Proactive nodes (H1Node .. H3Node).
• There are two types of StreamComponent Stream
  and StreamF
• Each SC contains user Compute code.
• Application (via SC_GUI) controls deployment and
  reconfiguration
• Change Compute codes and/or topology e.g.

SG1
f2
f3
f4
f5
SG1
f2
f4
f6
might become
4

• Outline of Talk
• We will describe incremental development of
  StreamComponents (SC)
• Firstly, using ordinary Active objects.
• Then Fractal and the StreamComponents Stream and
  StreamF.
• Then our Re-configuration mechanism.
• We will then discuss WebService bindings between
  Components
• Finally, our proposed WebService Stream Deployer
  (WSSD) scheme
• to facilitate full configuration and control of a
  distributed Stream via WebServices.

5

• Basic Stream Model
• We begin with just ProActive and standard Active
  objects.
• All entities producing stream values implement
  the Stream Interface. This has just one method
  car()obtains the next value from the Stream.
• Stream objects generate values StreamF objects
  apply a unary function and act as stream
  transducers.
• We instantiate these as Active objects, to get
• representing

6

• IntsTimesTen cont.
• The StreamEater is a simple test client which
  calls car() on upstream and displays the result.
• When we press Next to obtain a value from the
  stream, demands flow from left to right through
  successive calls to car() (shown in red)
• This continues until the beginning of the Stream
  (the IntegerStream object).
• Generated values are then fed back through the
  Stream to the consumer (StreamEater) (also shown
  in red).

upStream.car()
StreamF
upStream.car()
StreamEater
car()
IntegerStream
upStream
upStream
myFunc
3
4
car()
30
Val
NEXT
TimesTen
30
3
7

• Code
• Below is code for the integer generator, the
  times10 transducer, and for creating the active
  objects

public class IntegerStream implements Stream
private int currentValue 0 public
IntegerStream() public valueOWB car()
int temp currentValue
currentValue return new valueOWB(new
Integer(temp))
public class TimesTen implements unaryF
public Object apply(Object x)
int temp ( (Integer)x ).intValue()
return new Integer (temp 10)
Ints (IntegerStream)newActive("IntegerStream",
new Object ) Object params new Object
Ints, new TimesTen() TimesTenS
(StreamF)ProActive.newActive("StreamF", params)
StreamEater SE new
StreamEater(TimesTenS)
8

• IntsTimesTen Running
• Throughout our examples, we use IC2D to monitor
  and control the location and Migration of the
  StreamComponents/Objects.

Note that the StreamEater is not an Active object
therefore doesnt appear in IC2D.

• State of Stream was that of the last slide the
  next value was then obtained yielding 40.
• Due to the location tranparency of Active
  objects, the operation of the Stream is
  unaffected by the distribution across multiple
  nodes.

9

• Summary Active object implementation
• We have demonstrated the basic Stream mechanism
  using Active objects.
• This gives transparent distribution and
  migratability.
• But the composition of the Stream is fixed.
• We now describe the development of the
  StreamComponents themselves which will overcome
  this limitation.

10
Fractal Components

• A component consists of a Content surrounded by a
  membrane.
• The Content provides functional code for the
  component.
• Fractal is reflective the control interfaces
  control non-functional aspects.

11

• The Stream Component
• This is a Component version of the Stream object
  described previously.

• There is single Server interface, StreamSource
  which provides the car() method to supply the
  next value in the Stream.
• Compute code is a Stream object and is assigned
  to the attribute myStream.
• StreamCompAttributes is provided to set / change
  the Compute code. This is used for
  re-configuration.
• A LifeCycle Controller is provided to Start/Stop
  the component

12

• Fractal Components Function Interfaces and
  Binding
• Binding between components occurs from Client to
  Server and can be (in principle) done at any
  time.
• The diagram shows the IntTimesTen example in
  terms of Fractal components.
• After binding, values flow from the Server to the
  Client therefore binding direction is opposite
  to information flow.

13
IntTimesTen - Running

• Note StreamComponent GUI

14

• Static and Dynamic Reconfiguration
• Thus far, we have demonstrated how Compute codes
  can be changed via special Attribute controllers.
• The Attribute controllers as defined can only
  deal with already instantiated objects.
• This is static reconfiguration to be useful, the
  StreamComponent system must be able to accept any
  appropriate classes.
• We have developed a Dynamic Reconfiguration
  Mechanism whereby
• Compute code can be specified either as a Class
  name, or as bytecode
• If a classname is specified, it is loaded from
  the local file system via the standard java
  classloader, instantiated, then swapped into
  the Component
• If bytecode is given, a custom class-loader loads
  the bytecode, instantiates it, then swaps it into
  the Component a web service method enables
  upload to a remotely deployed StreamComponent.
• In the StreamComponent GUI, we introduce Change
  Class and Load Class buttons.

15

• Reconfiguration Example
• We will demonstrate the dynamic reconfiguration
  mechanism by continuing on from the last example

• The value 40 has just been obtained from the
  Stream

16

• We initiate a reconfiguration of the TimesTen
  component by pressing Load Class.
• This brings up a dialogue where we can select
  TimesHundred.

17

• and obtain next value ..

• Its important to note that TimesHundred was
  loaded from the local filesystem (Orac) as
  byte-code, then transmitted to the remote StreamF
  on DHPC13.
• It was then loaded(in the Java sense) by the
  custom class loader, instantiated and swapped
  into the StreamF.

18

• Reconfiguration extended example
• An important feature of StreamComponents is
  value neutrality.
• any serializable object can be represented.
• The system is not limited to using Java classes
  as functions and values .
• We will now present an example using the
  ImageMagick library
• An image manipulation library written in C
• We use JMagicK which is a Java implementation
  of ImageMagick
• Really just a wrapper library that uses JNI to
  dispatch to the C methods.

19

• A simple implementation of SingleImageStream
  would be

public valueOWB car() MagickImage image
new MagickImage( new
ImageInfo( TestImage.jpg ) ) return new
valueOWB(image)

• Problem is that image is just a handle to C
  structure .
• Therefore last line must become

return new valueOWB( ImageToBlob (image) )

• A Blob is just a byte representation and thus
  serializable.

20

• Demonstration
• We take the IntsTimesTen Stream from before .
  and reconfigure the Compute classes

21

• we can now obtain the next value from the
  Stream .

22

• We now change the StreamIdentity to an
  IMRotate90

23

• Rotated image ..

24

• Composition Example
• We now form a Stream consisting of several
  StreamF components composed together .

25

• Perform 2 rotations plus Blur

26

• Reconfiguration Summary
• The Compute code of the StreamComponents can be
  supplied/changed at anytime prior to or after
  execution has commenced.
• Combination of Fractive location-transparency and
  a custom class-loader results in the ability to
  download compute classes, as byte-code, from
  client machine to any remote component.
• System is not limited to Java libraries
• Can use JNI interface to interface with many
  other languages.
• Only constraint is that a Serializable
  representation for the Stream values must be
  provided.

27

• StreamComponents and WebServices
• There are three aspects where WS can be usefully
  employed with StreamComponents
• Inter-component bindings (as an alternative to
  Fractive binding)
• Re-configuration of Compute code via a WS
  interface
• Full creation, binding and configuration of a
  Stream via WS.
• Providing a WS interface by which users can
  obtain the values of a Stream is straightforward
  we could just expose the car() method of the
  particular StreamComponent directly, using the
  ProActive exposeAsWebService mechanism for
  example.
• However, we have chosen to use a proxy component,
  the WSServerProxy (WSSP) which acts as the WS
  interface to an associated StreamComponent.

28

• WebService Binding
• We did this because to enable additional WS
  methods to be added, in addition to car(), such
  as the reconfComp method discussed later.
• For the client-side, we have built the
  WSClientProxy (WSCP) which enables client-side
  StreamComponents to transparently access an
  accociated WSSP.
• Thus a WS binding is effected by inserting a pair
  of WSCP WSSP components into the stream at the
  point where the binding is to occur.
• From a semantic point of view, the WSCP WSSP
  pair is just an Identity Transducer.

29

• Returning to the IntsTimesTen Stream .
• We can change the binding between D13 and D17
  from Fractive to WS by inserting a WSProxy pair
  between them

30
IntsTimesTen with WS binding
Fractive Binding
WS binding

• We can compose any number of WS bindings
  together.

31

• Remote Deployment
• There are two main limitations of the WS based
  scheme described thus far
• Manual Deployment
• Any remote StreamComponents must be pre-deployed
  on the remote Nodes prior to being used by the
  client.

pre-deployed
32

• Data centralization
• In the process of composing several remote
  Streams, the dataflow will be unnecessarily
  directed through the local node .

• We want Stream values to flow through the local
  (client) node only when values are used there
• for example, by the local StreamF Component
  LocalFunc shown.

33

• We therefore want a scheme where stream-values
  flow directly to the next component in the Stream
  and only return to the local node when needed.

• This is similar to choreography vs orchestration
  in workflow.
• The WSSD scheme that we propose overcomes these
  limitations.

34

• Web Service Stream Deployer (WSSD)
• Specifically
• It facilitates the creation and reconfiguration
  of Stream Components on a remote node via WS.
• Allows the inter-component bindings to be made
  and altered though a WS interface whilst
  overcoming the Data Centralization problem.
• Represents the remote StreamComponents by a
  text-based handle thus allowing the Stream to be
  configured in virtually any environment including
  workflow systems such as Kepler.
• Each participating node must offer a WSSD
  service.
• To illustrate, we return to the IntsTimesTen
  stream

35

• WSSD Example Initial State
• The initial state of the system is one WSSD
  service on each remote node.
• Each remote WSSD is accessed locally via a proxy
  WSSDClientProxy.

36

• First step is to create the remote components.
• We do this by calling the createStream and
  createStreamF methods

String Ints D17Dep.createStream(Integers) Str
ing TimesTenS D13Dep.createStreamF(TimesTen,
TimesTenS)

• Notice that the returned values are the handles
  which are Strings
• HOST CLIENT_ID TYPE ID

37

• Omitting details, we expose components as WS,
  bind and attach StreamEater

38

• Summary
• We have presented a simple model for Stream
  computation and shown, through a series of
  experiments, the incremental development of
  StreamComponents.
• We started with an Active object implementation
  which yielded location transparency and
  migratability and then extended it by using
  Fractive components to support
• value neutrality
• reconfigurability of the compute code.
• reconfigurability of the stream topology
• complete separation of stream topology and
  distribution from the compute code.
• Support for WebServices was then explored by
• facilitating transparent binding of
  StreamComponents via WS.
• demonstrating a mechanism to reconfigure the
  Compute code via a WS.
• discussing a proposed mechanism, WSSD, by which
  StreamComponents may be created and configured
  via a WS interface.

39

• Future Work
• Evaluate WSSD scheme is it the best way to
  represent the notion of handle?
• consider WS-RF? Futures?
• Explore more eager forms of evaluation to take
  advantage of the potential pipelining
  parallelism available in a long chain of Stream
  components.
• The WebServices aspects of the StreamComponents
  is currently implemented using Axis2. With the
  recent release of ProActive 4, we will examine
  the use of the ProActive WS mechanism for the
  implementation.
• Safety of reconfiguration operations.
• Apply to use case of supporting instrument
  analysis workflow.
• Mapping to workflow engine (eg Kepler)