1. Resource Allocation for Distributed Streaming Applications
Qian Zhu and Gagan Agrawal
Department of Computer Science and Engineering
The Ohio State University
ICPP 2008 Conference
Sept. 10th, 2008 Portland, Oregon
ICPP 2008

2. Data Streaming Applications
Computational Steering
Interactively control scientific simulations
Computer Vision Based Surveillance
Track people and monitor critical infrastructure
Images captured by multiple cameras
Online Network Intrusion Detection
Analyze connection request logs
Identify unusual patterns
ICPP 2008 2 2

3. Streaming Applications in Wide Area Environments
Distributed high-volume data sources
Increasing WAN bandwidths
Better than secondary storage bandwidths
Geographically distributed users / consumers of data
Exploit flexibility in resource usage in Grid Environments

4. Our Previous Work A middleware system GATES
Grid-based AdapTive Executions on Streams
Integration with Grid Standards
Support for self-adaptation
Dynamic allocation and fault-tolerance

5. Resource Allocation in Streaming Grid Applications
Challenges
Pipeline of processing stages
Computation and communication requirements
Long running nature
Dynamic grid resources
Current Approach
Ad Hoc and Heuristics-based
Not considering both bandwidth and computing power
ICPP 2008

6. Overview of Our Research
Static Resource Allocation
Subgraph isomorphism based
Handle Network bandwidth and Computing power
Effectiveness value
Goal
To minimize the execution time of the data streaming applications
ICPP 2008

7. Outline Motivation and Introduction
Resource Allocation in Data Stream Processing
Resource Allocation Algorithm
Experimental Evaluation
Related Work
Conclusion
ICPP 2008 7 7

8. Data Stream Processing Model
Directed Acyclic Graph (DAG) – Gp(Vp, Ep)
source
Computing Power
Requirement
Bandwidth
Requirement
Processing nodes
sink
ICPP 2008

9. Resource Model Directed Acyclic Graph (DAG) – GR(VR, ER)
Computing power
Bandwidth
ICPP 2008

10. Problem Description
To Allocate Resources to the Data Stream Application
A mapping from Gp(Vp, Ep) to GR(VR, ER)
Modified Subgraph Isomorphism Based
To choose an isomorphic subgraph of GR
Transporters
Optimal Mapping
Effectiveness value
To minimize the execution time
ICPP 2008

11. Example A
1000 B
400 C
200 E
2000 D
100
transporter
ICPP 2008

12. Effectiveness Value Bandwidth only Including Computing Power
Number of transporters
A sigmoid function
Overhead of adding transporters
Computing power match
ICPP 2008

13. Outline Motivation and Introduction
Resource Allocation in Data Stream Processing
Resource Allocation Algorithm
Experimental Evaluation
Related Work
Conclusion
ICPP 2008 13 13

14. Proposed Algorithm Background: VF algorithm (L.P.Cordella et al.)
State Space Representation (SSR)
Feasibility rules
Depth-First Search
Pros and Cons
Efficient with small graphs (<200 nodes)
A large number of candidate partial mappings
ICPP 2008

15. Proposed Algorithm – Step 1
Prune Candidate Partial Mappings
Candidate node list
Reduce potential matches
Multiple Partial Mapping set 1 2 3 4 A B C D E F G C D G E F A
1000 3
200 B
400 C
200 E
2000
Cand(3)={C,D,G} D
100
ICPP 2008

16. Proposed Algorithm – Step 2
Modified Subgraph Isomorphism Mapping
Transporters 1 2 3 4 A B C D E F G C D G E F A
1000 3
200 B
400 C
200 E
2000 D
100
Candidate pair: (3,C)
transporter
Candidate pair: (3,D)
ICPP 2008

17. Handle Computing Power
Computing Node  Network Link
Computing power  Network bandwidth
Effectiveness Value Calculation
Possible Issues: high bandwidth and low computing power
Map one node onto a cluster of network nodes
ICPP 2008

18. Outline Motivation and Introduction
Resource Allocation in Data Stream Processing
Resource Allocation Algorithm
Experimental Evaluation
Related Work
Conclusion
ICPP 2008 18 18

19. Goals for the Experiments
Demonstrate the Scalability of Our Resource Allocation Algorithm
Demonstrate the High Performance of the Applications
ICPP 2008

20. Experiment Setup Algorithms Compared Streaming Applications Optimal
Streamline
Streaming Applications
Volume Rendering Application
A Synthetic Application
ICPP 2008

21. Scalability of the Resource Allocation Algorithm
ICPP 2008

22. Application Performance
Volume Rendering
33%
29%
27%
Within 4%
ICPP 2008

23. Application Performance
A Synthetic Application
40%
36%
Within 3%
34%
ICPP 2008

24. Outline Motivation and Introduction
Resource Allocation in Data Stream Processing
Resource Allocation Algorithm
Experimental Evaluation
Related Work
Conclusion
ICPP 2008 24 24

25. Related Work Resource Allocation for Stream Processing
Tang et al. (HPCC 06), Ali et al. (PDPTA 02)
Resource Allocation for Grid Computing
Abdu et al. (IPDPS 01), Bhat et al. (Grid 07),
Hong et al. (ICPP 03)
Subgraph Isomorphism Algorithms and Applications
Bioinformatics (Online Information 90), VLSI design (ISCAS 95), Mobile robot design (JPR 95)
ICPP 2008

26. Conclusion
Modified Subgraph Isomorphism Algorithm for Resource Allocation in Grid Streaming Applications
Handling Network Bandwidth and Computing Power
Comparable Overhead with Streamline
Improved Application Performance
ICPP 2008

27. Thank you!
ICPP 2008