Composite Subscriptions in Contentbased PubSub Systems

1
Composite Subscriptions in Content-based Pub/Sub
Systems

• Guoli Li
• Hans-Arno Jacobsen

2
Agenda

• PADRES system
• Composite subscriptions
• Case Study workflow management
• Experimental evaluations

3
Motivation

• Emerging Applications
• Business process execution
• Business activity monitoring
• Workflow processing
• Characteristics of these applications
• Distributed
• Loosely coupled
• Decoupled
• Compositional
• Service-oriented

4
Publish/Subscribe Model
Stock markets
TSX
NYSE
NASDAQ
AMGN58
Publications
IBM84
ORCL12
JNJ58
INTC19
HON24
MSFT27
Broker Network
Subscriptions
5
Focus of This Work

• Support composite subscriptions
• Distributed composite event detection
• Decentralized workflow management based on
  publish/subscribe

A
B
C
D
6
Agenda

• PADRES system
• Composite subscriptions
• Case Study workflow management
• Experimental evaluations

7
The PADRES System

• PADRES Publish/Subscribe Applied to Distributed
  REsource Scheduling
• Collaboration with Cybermation, Inc.
• PADRES Features
• Rule-based matching and routing engine
• Covering and merging based routing
• Load balancing
• Fault detection repair
• JMS support Web service integration

8
Subscription Language Model

• Predicate-based subscription language
• (attribute, op, value) e.g. (job,,A)
• Extended to composite subscriptions
• Rule-based message matching
• Well-known in expert systems
• Publications are mapped to facts
• Subscriptions are mapped to rules
• Advertisement are mapped to rules

9
Rule-based Matching Engine

• SRT
• Advertisements are rules
• Subscriptions are facts
• PRT
• Subscriptions are rules
• Publications are facts

Rule-based Matching Engine
JESS
adv
Subscription Routing Table (SRT)
sub
sub
Publication Routing Table (PRT)
pub
e.g. Sub(class,,jobappl,,payrollna
me,,Aprice,lt,100)
Pub(class, job appl,payroll name,
Aprice,80)
Fact P (job (appl payroll)
(job A) (price 80))
Rule S (job (appl payroll)
(job A) (price lt 100)) ?
(forward to destination D1, D2)
10
Content-based Routing

• Advertising

Advertisement
Distributed Overlay Broker Network


PADRES
Adopted from SIENA, Gryphon, REBECA and Hermes
11
Content-based Routing

• Subscribing

Subscription
Distributed Overlay Broker Network


PADRES
Adopted from SIENA, Gryphon, REBECA and Hermes
12
Content-based Routing

• Publishing

Publication
Distributed Overlay Broker Network


PADRES
Adopted from SIENA, Gryphon, REBECA and Hermes
13
Agenda

• PADRES system
• Composite subscriptions
• Case Study workflow management
• Experimental evaluations

14
Composite Subscriptions

• Composite subscription consists of atomic
  subscriptions
• Provide a higher level view for subscribers
• Save network bandwidth
• Naturally enabled by rule-based approach
• Subscription language features
• Operators AND (), OR ()
• Variables (x)
• Advanced event patterns

CSS1 OR S2 ANDS3 OR S4 AND S5
15
Composite Subscriptions
e.g. rule ((job (appl X) (name A) (state
succ)) (job (appl X)
(name B) (state succ))) gt (forward
a notification to proper destination)

• Parallel execution pattern s1 s2
• Alternation pattern s1 s2
• Sequence pattern (s1mint1s2)maxt2
• Repetition pattern Repetition(S, n, attr, v)

e.g. rule ((job . (name A) (time Y))
(job . (name B) (time gt Yt1)
(time lt Yt2))) gt (forward a
notification to proper destination)
e.g. rule ((job (appl X) (name A) (state
succ)) (job (appl X)
(name B) (state succ))) gt (forward
a notification to proper destination)
e.g. rule ((job . (name A) (attr Z))
(job . (name A) (attr
Zv))
(job .
(name A) (attr Z(n-1)v)) ) gt
(forward a notification to proper destination)
16
Composite Subscription Routing
CSS1 AND S2 ANDS3
P
Publishers
CSS1 AND S2
Subscribers
S
P1
Distributed Overlay Broker Network
CS
B1
B5
S
S1
B3
CS
B4
S3
S2
B2
B6
P2
P3
17
Composite Event Detection
CSS1 AND S2 ANDS3
P
Publishers
CSS1 AND S2
Subscribers
S
P1
P1
Distributed Overlay Broker Network
S1
CS
B1
B5
S
CS
CS
P12
B3
B4
P123
B2
B6
S2
P2
P2
S3
P3
P3
18
Routing Algorithm

• Build a destination tree for a composite
  subscription
• Split the composite subscription according to its
  destination tree
• Each part is forwarded to its destinations

At Node 4
CSS1 AND S2 ANDS3
null
CSS1 AND S2
6
3
3
3
Destination ID
19
Agenda

• PADRES system
• Composite subscriptions
• Case Study workflow management
• Experimental evaluations

20
Case Study Workflow Management

• Decentralized Workflow Management based on
  Pub/Sub
• Functions
• Workflow transform
• Workflow deployment
• Workflow execution
• Workflow control and monitoring

21
Workflow Descriptions

• Workflows are described using XML-based
  languages, such as BPEL
• Activities defined in a workflow have
  dependencies among each other
• A workflow instance is generated by a trigger
• Failures are handled in the workflow description

trigger
A
B
C
D
22
Workflow Transformation

• The first job(s) subscribe to trigger messages
• e.g. Job agent A subscribes to trigger
  publication class,,trigger,workflow,,payr
  oll, instanceID,,x
• Job dependencies are implied by subscriptions
• e.g. Job agent E subscribes to job A failed
  
• class,,job_info,workflow,,payroll,
  instanceID,,x, job,,A, status,,fail
• e.g. Job agent D subscribes to B and C
  finished, which is a composite subscription
• class,,job_info,workflow,,payroll,
  instanceID,,x, job,,B, status,,succ
• class,,job_info,workflow,,payroll,
  instanceID,,x, job,,C, status,,succ
• Advertisements are generated from a workflow
  specification file

23
Workflow Deployment
sub/advs Job D
sub/advs Job C
sub/advs Job B
sub/advs Job A
Deployer
sub/advs Job D
Distributed Overlay Broker Network
Job D Agent
B1
B5
B3
B4
sub/advs Job A
B2
Job A Agent
B6
sub/advs Job C
sub/advs Job B
Job C Agent
Job B Agent
24
Workflow Execution (Success)
trigger
Workflow Manager
END
Distributed Overlay Broker Network
Job D Agent
B1
B5
B3
A
B4
BC
sub/advs Job D
A
B2
Job A Agent
B6
A
sub/advs Job A
Job C Agent
sub/advs Job C
Job B Agent
B
C
sub/advs Job B
25
Workflow Monitoring

• Monitor is a Pub/Sub client
• Visualize topology of broker overlay network
• Visualize message routing
• Monitor/control workflow execution
• Workflow level
• Activity level
• Support dynamic workflow modification

26
Agenda

• PADRES system
• Composite subscriptions
• Case Study workflow management
• Experimental evaluations

27
Evaluations

• Environment
• Intel Xeon 3GHz, 1GB RAM
• JDK 1.4.2
• Workload description
• Predicates based on 20 attributes with random
  operators
• Attributes are selected following Zipfs law
• Values are uniformly distributed in the value
  range
• Two workflows with different number of jobs
• Metrics
• Matching time
• Network traffic

28
Publication Matching Time
29
Composite Event Detection
30
Number of Notifications
31
Network Traffic
32
Related Work

• Snoop by S. Chakravarthy
• An expressive event specification language
• Composite events represented by trees
• Event consumption policy
• GEM by M. Sansouri-Samani
• Event-Action rule-based event monitoring language
• Cambridge Event Architecture by P.R. Pietzuch, B.
  Shand, and J. Bacon.
• The Core Composite Event Language
• Composite event detector using automata
• An extension of existing pub/sub middleware
  platform
• SIENA by A. Carzanig, D.S.Rosenblum, and A.L.Wolf
• Advertisement-based routing
• Patterns with limited expressiveness
• Table of available patterns

33
Conclusions

• The rule-based matching is about 20 times faster
  than the counting algorithm
• Composite subscriptions reduce the message
  traffic up to 65
• Applicable to workflow management
• Future work
• Scalability analysis evaluations
• Support BPEL4WS
• QoS-based workflow execution

34
QA
Thank You!

• Contact
• gli_at_cs.toronto.edu
• jacobsen_at_eecg.toronto.edu
• Middleware system research group, University of
  Toronto
• www.msrg.eecg.toronto.edu

35
Support Slides
36
Unsubscribe CS
CSS1 AND S2 ANDS3
P
Publishers
CSS1 AND S2
Subscribers
S
P1
Distributed Overlay Broker Network
S1
CS
B1
B5
S
CS
S1
CS
CS
CS
CS
S1
B3
B4
CS
CS
CS
S2
S3
B2
S2
B6
S2
CS
S3
Unsubscription
P2
S3
CS
Ack message
P3
37
Workflow Execution (Failure)
trigger
Workflow Manager
sub/advs Job D
Distributed Overlay Broker Network
Job D Agent
B1
B5
B3
B4
END
B2
Job A Agent
B6
Job F Agent
A
sub/advs Job A
Job C Agent
sub/advs Job F
Job B Agent
E
Job E Agent
sub/advs Job E
sub/advs Job C
sub/advs Job B
38
Architecture