Moirae: History-Enhanced Monitoring

1
Moirae History-Enhanced Monitoring

• Magdalena Balazinska, YongChul Kwon, Nathan
  Kuchta, and Dennis Lee
• University of Washington
• and Marchex Inc.

2
Monitoring Applications

• Continuously observe current state
• Produce near real-time information and alerts
• Event (eid, timestamp, a1, ..., an)
• Examples
• Sensor-based environment monitoring
• Computer system monitoring
• Network intrusion detection

load level
time
3
Problem Exploit History

• Monitoring applications accumulate history
• Exploiting history can improve monitoring apps
• Refine event detection
• Explain newly detected events
• What types of queries to support?
• How to support them?

4
Types of Queries

• Standard hybrid queries
• Standard SQL query over the data archive executes
  as part of the continuous query
• For each network intrusion, show historical
  activity of the intruder on the network
• Contextual hybrid queries
• For each newly detected event, produce
  approximate set of k most similar past events
• If a server fails, show similar alerts that
  occurred in the past

5
Standard Hybrid Query
Query model based on Borealis
Other Stream Proc. Ops.
Input Streams
Stream Proc. Operators
Recall
Continuous stream processing (event query)
Archive
Look up historical information (historical query)
6
Contextual Hybrid Query
Similar past events
Event
Input Stream
Event detection
Similarity Recall
Input Stream
Window Join
Past contexts
Context
Input Sream
Window Join
Archive
Find similar past events Use TF-IDF for
similarity computation
7
Framework

• Three key issues
• History size, near real-time, concurrent events
• Three goals
• Responsiveness and fairness
• Incremental processing integrated with stream
  processing
• Retrieve at least some historical data for all
  new events
• Relevance favor recent history over older
  history
• Similarity find similar past events
• Exploit context similarity in other ways as well

8
Moiraes Design
Approximate incrementally improving results
Stop Improving Query
Contextual Standard hybrid queries
Application
MOIRAE
SPE
Stream Processor
Deploy Manager
Based on Borealis
Raw Streams
Recall Manager
RDBMS
Based on PostgreSQL
Storage Manager
Archiver
Materialized Events Context
Other Materialized Views
Raw Stream Archive
Present Chunk
9
Design Components

• Archiver partitioned stream archive
• Archive raw and intermediate streams
• Present chunks in memory
• Recent chunks on disk
• Materialize index necessary streams and
  contexts
• Old chunks on disk
• Recall Manager partitioned, incremental queries
• Execute queries one chunk at the time (present
  -gtpast)
• Schedule concurrent queries to ensure fairness
• Incorporate user feedback to drop events

10
Related Work

• Queries over live data data archives
  chandrasekaran04,chandrasekaran05,
  franklin05
• Log-structured access method muth00
• Multi-level storage manager stonebraker91
• Materialized views (e.g., goldstein01)
• Partial indexes stonebraker89,sartori94,seshadr
  i95
• Online processing hellerstein97,hellerstein00,
  raman02,shanmugasundaram01,tan99
• top-K, kNN, IRRDBMS e.g.,carey97,fagin03,chaud
  huri05,li05

11
Conclusion

• Monitoring applications accumulate history
• How to leverage history ?
• By supporting queries for specific historical
  data
• Through new types of queries
• How to support all these different queries ?
• Can reuse several database/IR techniques
• Need to integrate and extend these techniques
• More information about Moirae
• http//data.cs.washington.edu/moirae/moirae.shtml