Resource Allocation for Distributed Streaming Applications

1
Resource Allocation for Distributed Streaming
Applications
Qian Zhu and Gagan Agrawal Department of Computer
Science and Engineering The Ohio State University
ICPP 2008 Conference
Sept. 10th, 2008 Portland, Oregon
ICPP 2008
2
Data Streaming Applications

• Computational Steering
• Interactively control scientific simulations
• Computer Vision Based Surveillance
• Track people and monitor critical infrastructure
• Images captured by multiple cameras
• Online Network Intrusion Detection
• Analyze connection request logs
• Identify unusual patterns

2
ICPP 2008
3
Streaming Applications in Wide Area Environments

• Distributed high-volume data sources
• Increasing WAN bandwidths
• Better than secondary storage bandwidths
• Geographically distributed users / consumers of
  data
• Exploit flexibility in resource usage in Grid
  Environments

4
Our Previous Work

• A middleware system GATES
• Grid-based AdapTive Executions on Streams
• Integration with Grid Standards
• Support for self-adaptation
• Dynamic allocation and fault-tolerance

5
Resource Allocation in Streaming Grid Applications

• Challenges
• Pipeline of processing stages
• Computation and communication requirements
• Long running nature
• Dynamic grid resources
• Current Approach
• Ad Hoc and Heuristics-based
• Not considering both bandwidth and computing
  power

ICPP 2008
6
Overview of Our Research

• Static Resource Allocation
• Subgraph isomorphism based
• Handle Network bandwidth and Computing power
• Effectiveness value
• Goal
• To minimize the execution time of the data
  streaming applications

ICPP 2008
7
Outline

• Motivation and Introduction
• Resource Allocation in Data Stream Processing
• Resource Allocation Algorithm
• Experimental Evaluation
• Related Work
• Conclusion

7
ICPP 2008
8
Data Stream Processing Model

• Directed Acyclic Graph (DAG) Gp(Vp, Ep)

source
Computing Power Requirement
Bandwidth Requirement
Processing nodes
sink
ICPP 2008
9
Resource Model

• Directed Acyclic Graph (DAG) GR(VR, ER)

Computing power
Bandwidth
ICPP 2008
10
Problem Description

• To Allocate Resources to the Data Stream
  Application
• A mapping from Gp(Vp, Ep) to GR(VR, ER)
• Modified Subgraph Isomorphism Based
• To choose an isomorphic subgraph of GR
• Transporters
• Optimal Mapping
• Effectiveness value
• To minimize the execution time

ICPP 2008
11
Example
A 1000
B 400
C 200
E 2000
D 100
transporter
ICPP 2008
12
Effectiveness Value

• Bandwidth only
• Including Computing Power

Number of transporters
A sigmoid function
Overhead of adding transporters
Computing power match
ICPP 2008
13
Outline

• Motivation and Introduction
• Resource Allocation in Data Stream Processing
• Resource Allocation Algorithm
• Experimental Evaluation
• Related Work
• Conclusion

13
ICPP 2008
14
Proposed Algorithm

• Background VF algorithm (L.P.Cordella et al.)
• State Space Representation (SSR)
• Feasibility rules
• Depth-First Search
• Pros and Cons
• Efficient with small graphs (lt200 nodes)
• A large number of candidate partial mappings

ICPP 2008
15
Proposed Algorithm Step 1

• Prune Candidate Partial Mappings
• Candidate node list
• Reduce potential matches
• Multiple Partial Mapping set

B C D E F G
C D G
A
E F
A 1000
3 200
Cand(3)C,D,G
B 400
C 200
E 2000
D 100
ICPP 2008
16
Proposed Algorithm Step 2

• Modified Subgraph Isomorphism Mapping
• Transporters

B C D E F G
C D G
A
E F
A 1000
3 200
B 400
C 200
E 2000
D 100
Candidate pair (3,C)
transporter
Candidate pair (3,D)
ICPP 2008
17
Handle Computing Power

• Computing Node ? Network Link
• Computing power ? Network bandwidth
• Effectiveness Value Calculation
• Possible Issues high bandwidth and low computing
  power
• Map one node onto a cluster of network nodes

ICPP 2008
18
Outline

• Motivation and Introduction
• Resource Allocation in Data Stream Processing
• Resource Allocation Algorithm
• Experimental Evaluation
• Related Work
• Conclusion

18
ICPP 2008
19
Goals for the Experiments

• Demonstrate the Scalability of Our Resource
  Allocation Algorithm
• Demonstrate the High Performance of the
  Applications

ICPP 2008
20
Experiment Setup

• Algorithms Compared
• Optimal
• Streamline
• Streaming Applications
• Volume Rendering Application
• A Synthetic Application

ICPP 2008
21
Scalability of the Resource Allocation Algorithm
ICPP 2008
22
Application Performance

• Volume Rendering

33
29
27
Within 4
ICPP 2008
23
Application Performance

• A Synthetic Application

40
36
Within 3
34
ICPP 2008
24
Outline

• Motivation and Introduction
• Resource Allocation in Data Stream Processing
• Resource Allocation Algorithm
• Experimental Evaluation
• Related Work
• Conclusion

24
ICPP 2008
25
Related Work

• Resource Allocation for Stream Processing
• Tang et al. (HPCC 06), Ali et al. (PDPTA 02)
• Resource Allocation for Grid Computing
• Abdu et al. (IPDPS 01), Bhat et al. (Grid 07),
• Hong et al. (ICPP 03)
• Subgraph Isomorphism Algorithms and Applications
• Bioinformatics (Online Information 90), VLSI
  design (ISCAS 95), Mobile robot design (JPR 95)

ICPP 2008
26
Conclusion

• Modified Subgraph Isomorphism Algorithm for
  Resource Allocation in Grid Streaming
  Applications
• Handling Network Bandwidth and Computing Power
• Comparable Overhead with Streamline
• Improved Application Performance

ICPP 2008
27
Thank you!
ICPP 2008