Design of HighPerformance Distributed EventProcessing Systems

1
Design of High-Performance DistributedEvent-Proc
essing Systems

• Roman Elizarov
• Devexperts
• SDBP 2007

2
Event processing?

• Common situation
• You are designing a system that processes events
  (stock quotes, sports bets, telemetry from
  factory/network hardware, etc)
• This system is usually distributed, because
  events come from different places.
• What design choices are you going to consider?

3
The usual design dichotomy
Remote Procedure Calls (RPC) for synchronous
event processing
Message Oriented Middleware (MOM) for
asynchronous event processing
V
S

• I will show that this question, as it is usually
  stated, is misleading. You should not ask this
  question at all.

4
Message Oriented Middleware?

• Popular approach to message passing
• Lots of books and publications
• Wikipedia definition
• Message Oriented Middleware is a category of
  inter-application communication software that
  relies on asynchronous message passing as opposed
  to a request/response metaphor.

5
MOM Advantages

• Asynchronous message transfer (sender is
  decoupled from receiver)
• Message persistence
• Transactional support
• Interoperability (cross-platform)
• Standards-based APIs (for example, JMS)

6
MOM Disadvantages

• It adds additional (and usually 3rd party)
  component to the system architecture (Message
  Transfer Agent)
• Harder to maintain
• Reduces reliability
• Reduces performance
• It requires learning of a large 3rd party or
  standard APIs.

7
More on performance

• Events that you are planning to process may
  happen 10K times per second or more often.
• This rate is above peak performance of most MOM
  systems. Rates at 100K events per second cannot
  be reached by any modern MOMs on a decent
  hardware.

8
Common misconception 1

• Myth Asynchronous event processing implies MOM
• Truth You can design high-performance system for
  asynchronous event processing yourself without
  MOM.

9
Common misconception 2

• Myth MOM vendors are spending millions of on
  their software. How could we possibly design
  something with a higher performance in our
  project within a much smaller budget?
• Truth Design is not about RPC vs MOM. Design is
  about data structures, algorithms, patterns, and
  constraints of your particular project.

10
Data structures algorithms?

• Hold on
• You said data structures and algorithms. What
  this has to do with design anyway? We always
  thought that those are the coders issues, not
  designers ... Designer is drawing diagrams,
  determines system components and interfaces. He
  does not implement any data structures and
  algorithms, does he?

11
Software Designer?

• Designer does not code data structures and
  algorithms.
• Designer defines interfaces that place
  constraints on implementation (data structures
  and algorithms).
• Which lead to constrains on maximal system
  performance.

12
Design example

• Compare two designs

interface Foo1 void bar(Set s) interface
Foo2 void bar(SortedSet s)
13
Design example contd

• SortedSet operations can be implemented
  efficiently in O(log n) time using different
  kinds of trees (B-Trees, Red-Black trees, Splay
  trees, etc).
• Set operations can be implemented efficiently in
  O(1) time using hashing.

14
Designing with algorithms in mind

• Design must enforce only constraints that are
  absolutely necessary.
• You should follow the rule of the least possible
  constraint.

15
MOM Performance demystified

• MOM systems are usually slow by design. It is not
  because of some negligence or oversight of coders
  who implement them.
• This is especially true for the
  standards-conforming MOM systems.
• No JMS implementation can be really fast because
  of the complexity of the JMS specification and
  its requirements.

16
Questions to ask before using MOM

• Does my system absolutely need message
  persistence, transactional processing, and/or
  interoperability (cross-platform)? For many
  systems the answer is NO.
• Does my system absolutely need maintainability,
  reliability, and high-performance? For many
  systems the answer is YES.

17
The Sample Design Problem

• Financial application with quote table (ticker).
• Keeps track of last known data and shows it to
  the client.
• Updates data as it changes.
• 100K (target1M) quotes per second on 300K
  distinct symbols enter into the system.

18
Deployment scenario (Cluster)

• Data events are distributed via a tree of
  multiplexing nodes (multiplexors)
• Data goes downwards, subscription goes upwards
• Each multiplexor, data source, and client uses
  Ticker Core

19
Overall Architecture

• Make two layers separate data structure layer
  (Ticker Core) from data transport layer (Socket
  Connector)
• Ticker core can be used with different transport
  layers
• There is no need for a dedicated MTA process. You
  can multiplex data inside any process on your
  deployment diagram using Ticker Core component.

20
Optimization rule

• Do not optimize your code until you can prove by
  profiling that it needs optimization.
• But this does not apply to Design!
• If your design if broken, it might be impossible
  to optimize your implementation without actually
  redesigning and rewriting everything from
  scratch.
• You have to have at least one efficient
  implementation in mind when you are doing design.

21
Ways to spoil high-performance

• Locking (synchronization)
• Locking/unlocking too much
• Locking for too long
• Memory operations
• Allocating too much memory
• Locality of data access
• Ineffective algorithms data structures
• Using inappropriate data structures

22
First design attempt

• Let us try to design method that feeds data into
  Ticker Core.

Why is it wrong? Think about possible
implementations of processEvent method Hint our
data structure must be MT-Safe.
interface Distributor void processEvent(Event
e)
Not good
23
Locking measurement

• You cannot process 1M events per second with
  MT-Safe data structure (even on a single-CPU
  machine).

Performance is measured on Pentium 4M 1.7GHz
laptop with Java 1.5.0_03 (server JVM) under
Windows XP SP2.
24
Locking measurement contd

• On SMP system it becomes even worse.
• And were not even doing anything inside
  synchronized section of code (just k).

Performance is measured on 2-way SPARC Sun Fire
V240 with Java 1.5.0_01 (server JVM) under
Solaris 5.9
25
Locking solution

• Events must be processed in blocks, thus paying
  locking cost per block instead of per event.

interface Distributor void processEvents(Event
e)
Better!
26
Locking conclusion

• Keep in mind the cost of locking when you are
  designing MT-Safe data structures.
• Some data structures can be implemented without
  locking, but not all of them, so do not count on
  work-around unless you know it.
• Contention becomes worse if you keep locks during
  time-consuming operations.

27
Design patterns

• If you use arrays (like Event) in your design,
  then you inherently limit performance and
  constrain implementation
• What if data source keeps events in the hash?
• What if data source constructs events on-the-fly
  (deserializes them from the external source)?
• There are design patterns to help you
• Iterator pattern
• Visitor pattern

28
Iterator pattern

• Keeps flow control on the data receiver side.

interface Distributor void processData(DataIter
ator it) interface DataIterator Event
nextEvent()
29
Iterator pattern contd

• Lock is held inside Ticker for the duration of
  processData.
• DataSource can get data from anywhere (from
  array, hash, deserialize, etc).

30
Visitor pattern

• Keeps flow control on the data provider side.

interface Agent void retrieveData(DataVisitor
vis) interface DataVisitor void
visitEvent(Event e)
31
Visitor pattern contd

• Lock is held inside Ticker for the duration of
  retrieveData.
• DataConsumer can do anything with data (store to
  array, hash, serialize, etc)

32
Patterns conclusion

• Which design is better from the least possible
  constraint point of view?

interface Distributor1 void processEvent(Event
e) ------------------------------------- OR
------------------------------------- interface
Distributor2 void processData(DataIterator
it) interface DataIterator Event
nextEvent()
33
Designing for overload

• What if the load is too high? (too many events)
• The system may fail, but how it fails is quite an
  important design decision.
• What usually happens under the load beyond high
  CPU consumption
• Lock contention
• Buffers overflow
• The problems should not be getting progressively
  worse under the load.

34
Designing for overload attempt

• Let us try to design means to notify data
  consumers on available events.

Interface Agent void setEventListener(EventList
ener l) interface EventListener void
eventsArrived(Event e) // or a version with
visitor
35
Overload scenario

• What would EventListener do if it receives most
  events that it can put into the outgoing TCP
  stream?
• Buffer extra events
• Drop extra events
• Design shall inherently accommodate the
  corresponding buffering and dropping strategies.

36
Design for overload solution

• Let data consumer fetch data from the data source
  only when data consumer has spare CPU cycles to
  do so.
• Automatically increase block size under the load.
  It reduces all kinds of block overheads (lock
  overhead, I/O overhead, etc).

interface Agent void setEventListener(EventList
ener l) void fetchData(DataVisitior
v) interface EventListener void
eventsArrived()
37
Overall conclusion

• Process events in blocks throughout all places in
  your system with high number of events per second
• Save on locking
• Save on I/O cost
• Increase block size under the load
• Use design patterns
• Use appropriate algorithms and data structures

38
The End

• Thank you for your attention.
• Any questions?