PADRES A Contentbased PubSub System

1
PADRESA Content-based Pub/Sub System
Middleware Systems Research Group University of
Toronto Cybermation Inc.
Research Team Eli Fidler Vinod Muthusamy
Alex Cheung Alex Wun Guoli Li Songlin Hu
Matt Medland Pengcheng Wan Hans-Arno
Jacobsen (PI) Serge Mankovski (Cybermation)
2
The PADRES System

• PADRES Publish/Subscribe Applied to Distributed
  REsource Scheduling
• Features
• Distributed content-based publish/subscribe
  system using peer-to-peer content-based routing
• Subscription language supports advanced concepts
  such as composite events
• Tools and components to support workflow
  management applications

3
Project Context

• A distributed content-based Pub/Sub system
• Loosely coupled data sources and sinks
• Routing messages entirely on their content
  (Content-Based Routing)
• Based on distributed broker network
• Hybridisation of pub/sub and query-based data
  access in distributed environments
• Flexible and seamless messaging substrate for
  applications such as enterprise job scheduling
• Collaboration with Cybermation Inc.

4
Content-Based Routing

• Messages are routed based on their content, not a
  target address
• Clients indicate interests by subscribing to
  content filters (their C-B addresses)
• Publications are routed hop-by-hop according to
  the subscriptions at each broker

5
Content-based Routing

• Advertising

Advertisement
Distributed Overlay Broker Network


PADRES
6
Content-based Routing

• Subscribing

Subscription
Distributed Overlay Broker Network


PADRES
7
Content-based Routing

• Publishing

Publication
Distributed Overlay Broker Network


PADRES
8
System Concepts

• Message
• Advertisement
• Subscription
• Publication
• Event
• Atomic event publication in pub/sub literature
• Composite event a pattern of events
• Subscriptions
• Atomic subscription atomic event
• Composite subscription composite event

9
Features

• Rule-based matching and routing engine
• Unifies matching and routing
• Supports powerful subscription language
• Based on Rete matcher
• Historic and time-based query support
• Subscribe to past data
• Subscribe to future data
• Correlate past and future data
• Composite and meta subscriptions
• Subscribe to temporal and causal patterns
• Subscribe to system events
• Covering and merging based routing
• Perfect and imperfect merging
• JMS support
• Federation of JMS servers

10
System Architecture
11
Network Architecture

• Broker
• Forward messages using overlay network
• Provide RMI bindings as client connection points
• Clients
• Advertise, publish, and subscribe

12
Broker Architecture
13
Current/Future Work

• Composite subscriptions
• Historic data access
• Scalability, fault tolerance
• Security
• Benchmarking, load testing
• We badly need experience and data from the real
  world

14
Using Padres

• Job Scheduling
• And
• Workflow Management

15
System Functions

• Monitor
• Visualization of broker network topology
• Gather information about each broker
• Measure the performance
• Historic Data Access
• Subscribe to both future and past events
• Support complicated historic queries
• Load Balancing
• Overloaded brokers offload event delivery to
  other brokers
• Overloaded subscribers offload publications to
  event sinks
• Fault Tolerance
• Identify a failure
• Failure recovery
• Applications Enterprise Job Scheduling

16
Job Scheduling based on PADRES
Job Scheduling Application

• Job schedule is defined in XML
• Job dependencies can be transferred to a set of
  composite subscriptions ,e.g.
• Job DSub(job B done) AND Sub(job C done)
• Job scheduling deployer injects the subscriptions
  into PADRES federation, which are received by job
  agents
• Trigger starts the execution of deployed job
  schedule.

XML
Set of sub/advs
Job Scheduling Deployer
Trigger
PADRES Broker Network

Agent
Agent
Agent
17
Workflow Management

• Decentralized Workflow Management based on
  Pub/Sub
• Functions
• Workflow transform
• Workflow deployment
• Workflow execution
• Workflow control and monitoring

18
Workflow Descriptions

• Workflows are described using XML-based
  languages, such as BPEL
• Activities defined in a workflow have
  dependencies among each other
• A workflow instance is generated by a trigger
• Failures can be handled in the workflow
  description

trigger
A
B
C
D
19
Workflow Transform

• The first job(s) subscribe to trigger messages
• e.g. Job A subscribes to trigger
  publication class,eq,trigger,workflow,eq,pa
  yroll, instanceID,,x
• Job dependencies are implied by subscriptions
• e.g. Job E subscribes to job A failed
• class,eq,job_info,workflow,eq,payroll,
  nstanceID,,x, job,eq,A, status,eq,fail
• e.g. Job D subscribes to B and C finished,
  which is a composite subscription
• Advertisements are generated from a workflow
  specification file

20
Workflow Transform/Deployment

• Job agents are connected to brokers and maintain
  subscriptions (C-B addresses) indicating their
  capabilities
• Advertisements and subscriptions are packaged and
  injected to appropriate job agents
• Transformer and Deployer can decide which jobs
  should be assigned to which agent capabilities,
  although the CBR network handles the actual
  delivery of the assignments

21
Workflow Deployment
sub/advs Job D
sub/advs Job C
sub/advs Job B
sub/advs Job A
Deployer
sub/advs Job D
Distributed Overlay Broker Network
Job D Agent
B1
B5
B3
B4
sub/advs Job A
B2
Job A Agent
B6
sub/advs Job C
sub/advs Job B
Job C Agent
Job B Agent
22
Workflow Execution (Success)
trigger
Workflow Manager
END
Distributed Overlay Broker Network
Job D Agent
B1
B5
BC
B3
A
B4
sub/advs Job D
A
B2
Job A Agent
B6
A
sub/advs Job A
Job C Agent
sub/advs Job C
Job B Agent
B
C
sub/advs Job B
23
Workflow Execution (Failure)
trigger
Workflow Manager
sub/advs Job D
Distributed Overlay Broker Network
Job D Agent
B1
B5
B3
B4
END
B2
Job A Agent
B6
Job F Agent
A
sub/advs Job A
Job C Agent
sub/advs Job F
Job B Agent
E
Job E Agent
sub/advs Job E
sub/advs Job C
sub/advs Job B
24
Workflow Monitoring

• Monitor is a Pub/Sub client
• Monitor/control workflow execution status
• Workflow level
• Activity level
• Support dynamic workflow modification
• Visualize topology of broker overlay network
• Visualize message routing

25
Current/Future Work

• Define full BPEL to pub/sub mapping
• Define mapping of known workflow patterns to
  pub/sub
• Comparative evaluation with centralized
  orchestration
• Benchmarking, load testing
• We badly need experience and data from the real
  world (i.e., business processes etc.)

26
- END -