INNOV-10 Progress

1
INNOV-10Progress Event EngineTechnical
Overview

• Prashant Thumma
• Principal Software Engineer

2
Agenda Slide

• Event Stream Processing (ESP)
• Progress ESP products
• Progress Event Engine (PSEE)
• PSEE Performance
• PSEE Integration

3
Event Stream Processing (CEP)
What is ESP?

• ESP is about Operational Business Data
• Capture
• Query
• Analysis
• Characteristics
• High Volume Streaming Data
• Time Critical Analysis
• Consistent and Reproducible Results
• Highly Available
• Simple Structured Data
• Data is not Relational or complex Object model

4
Event Stream Processing (ESP)
ESP Platform Requirements

• Event Processing Language
• Data Driven
• Flexible
• Complete
• Event Persistence Engine
• High Performance
• Analytical Capabilities
• Temporal Data Analysis
• Inbound and Outbound Event Processors
• High Availability
• Modeling, Deployment and Development Tools

5
Event Stream Processing (CEP)
ESP Applications

• Financial Markets
• Program Trading
• Testing Trading Strategies
• RFID
• Inventory Logistics
• Asset Tracking
• Theft Detection
• Manufacturing
• Capture and analysis of data from PLCs
• Telecommunications
• Cell Phone Call Tracking
• Multimedia Services Billing
• Network Management
• Scientific
• Sensor Data

6
Agenda Slide

• Event Stream Processing (ESP)
• Progress ESP products
• Progress Event Engine (PSEE)
• PSEE Performance
• PSEE Integration

7
Progress ESP Products
C API
XML Configuration Files
Progress Event Engine (PSEE)
8
Agenda Slide

• Event Stream Processing (ESP)
• Progress ESP products
• Progress Event Engine (PSEE)
• PSEE Performance
• PSEE Integration

9
Progress Event Engine
What is it for?

• Its all about Event Data
• Collection, storage, management and query
• What is Event Data?
• Streaming, constantly flowing information
• Stock Trades, RFID, IP Packets, Mfg PLCs
• Categorized
• Stock tick symbol, RFID tag ID, IP address
• Typically Time Ordered
• Trade time, RFID scan time

10
Progress Event Engine
Data Flow
Event Data Feed
1. Collect
store
Convert external events to PSEE events
sort/merge
2. Organize
Structures data for fast retrieval
11
Progress Event Engine
Why PSEE?

• High Performance, Scalable
• Collection Engine
• Store tens of thousands events per second
• 20 TB database size
• Index data is 1 of database size
• Query Engine
• High Performance
• Pipelined Architecture
• Extensible Functionality
• Simultaneous Query and Collection
• Writes do not block reads!

12
Progress Event Engine
Why PSEE?

• Transparent Data Management
• Event Persistence
• Requires no coding to persist events
• Transactional
• Consistent Data
• Hidden and Managed by PSEE
• Tunable Latency
• Real-time defined by you!

13
Progress Event Engine
Why PSEE?

• Simple API
• Supports Multiple Languages
• Java and C language support
• Multiple Platform Support
• Window, Solaris, Linux
• Heterogeneous deployment

14
Progress Event Engine
Components

• Functor A unit of work in PSEE
• 3 Main types (Source, Process, Sink)
• Built in library of Functors
• Customer coded
• Pipeline A sequence of Functors
• Collect, process, transform event streams
• Task Manager
• Manages pipeline execution
• Events Input and Output of Pipelines

15
Progress Event Engine
Event Definition

• PSEE Event
• XML schema
• Supports All C Primitive Types

ltEventDefinitions bundleNameTABundle gt
ltEvent nameTrade typeTimedCategorize
dEventgt ltMember nameprice
typeFloat32/gt ltMember namevolume
typeUInt32/gt lt/Eventgt lt/EventDefinitionsgt
16
Progress Event Engine
Functor

• Implements Event Stream Processing Logic
• Interface For Accessing Events

class MyCollectorFunctor public Functor
public void initialize(FunctorContext
ctx) void handle(FunctorContext ctx) void
finalize(FunctorContext ctx) private //
functor data variables
17
Progress Event Engine
Collector Pipeline
Real-Time Data Feed
StoreRealtime Functor
MyCollector Functor
Event
Series (PSEE Database)
Stream
Event-Data File for Bulk Loading
18
Progress Event Engine
Query Pipeline

MyXMLwriter Functor
Retrieve Functor
Filter Functor
Sort Functor
Event
Event
Event
Stream
Stream
Stream
Series (PSEE Database)
19
Progress Event Engine
Predefined Functors

• Collector Functors
• StoreRealtime
• StoreBulkload

• Query Functors
• Retrieve
• Filter
• Sort
• Count
• Group Count

20
Progress Event Engine
Scalable Deployment
MessagingScalability
21
Agenda Slide

• Event Stream Processing (ESP)
• Progress ESP products
• Progress Event Engine (PSEE)
• PSEE Performance
• PSEE Integration

22
PSEE Performance
Performance Metrics

• Event storage (events/sec)
• Real-time collector pipeline using
  StoreRealtime functor
• Bulk load collector pipeline using
  StoreBulkload functor
• Event retrieval (events/sec)
• Threaded execution of query pipeline
• Random query
• Query on several (randomly chosen) categories

23
PSEE Performance
Performance Testing Environment

• Database Characteristics
• 160 Million Events
• 20 Byte Event
• 10000 Categories
• System Configuration
• Sun Fire-V440
• Solaris 8
• 4 CPUs (1GHz)
• 8 GB Memory

24
PSEE Performance
Performance Testing Results

• Real-time collector
• 94,062 events / second
• Bulk load collector
• 191,152 events / second
• Query
• 213 Queries / second
• 10000 events / query
• 2,134,560 events / sec

25
Agenda Slide

• Event Stream Processing (ESP)
• Progress ESP products
• Progress Event Engine (PSEE)
• PSEE Performance
• PSEE Integration

26
Under Development

• This talk includes information about potential
  future products and/or product enhancements.
• What I am going to say reflects our current
  thinking, but the information contained herein is
  preliminary and subject to change. Any future
  products we ultimately deliver may be materially
  different from what is described here.

27
PSEE Integration
Integration with Apama

• Provides Event Persistence for Apama
• Initial Implementation for Customer
• Tick Database for Back-Testing Trading Strategies
• Query Interface Using Apama Event Language
• Future
• Extend Apama Event Language to support
  Persistence
• Seamless Configuration via Apama Management
  Interfaces

28
PSEE Integration
Integration with Sonic

• PSEE on the bus using Event Cache
• Event Cache Features
• Native JMS Message Based Query
• Results are Java Objects
• Batched for Optimal Performance

29
PSEE Integration
Integration with 4GL

• PSEE based RFID Accelerator provides 4GL
  Interface
• Web Services based Implementation
• RFID Accelerator as a reference implementation
• Further 4GL Integration

30
In Summary

• Exponential growth of event data requires ESP
  solutions
• PSEE Apama is an ESP solution
• Based on proven technologies
• Simple! Infrastructure problems are handled by
  PSEE
• It can be integrated into existing business
  systems

31
Questions?
32
Thank you for your time!
33
(No Transcript)