MAIDS: Mining Alarming Incidents in Data Streams

1
MAIDS Mining Alarming Incidents in Data Streams

• Automated Learning Group, NCSA
• Department of Computer Science
• University of Illinois at Urbana-Champaign

2
Characteristics of Stream Data

• Different from finite, static data stored in flat
  files and database systems
• Transient data that pass through a system
  continuously
• You get only one look
• In huge amount, potentially infinite
• At low level and multi-dimensional in natural
• Dynamically change over time

3
Stream Data Mining Applications

• Telecommunication calling records
• Business credit card transaction flows
• Network monitoring and traffic engineering
• Financial market stock exchange
• Engineering industrial processes manufacturing
• Sensor, monitoring surveillance video streams
• Security monitoring
• Web logs and Web page click streams
• Tracking information from IntelliBadge

4
Stream Data Mining Tasks

• Perform multi-dimensional statistical analysis
• Cluster data streams
• Classify data streams
• Build prediction models
• Find frequent patterns
• Find sequential patterns
• Mine outliers and unusual patterns

5
Challenges in Stream Data Analysis

• Huge volume
• Multi-Dimensional Data
• Need to remember recent and historical data
• Multiple analysis at the same time
• Support user interaction
• Require instance response
• Most existing algorithms and prototype systems
  are memory and CPU bound, and can only perform a
  single data mining function

6
Mining Alarming Incidents in Data Streams - MAIDS

• MAIDS is aimed to
• Discover changes, trends and evolution
  characteristics in data streams
• Construct clusters and classification models
  from data streams
• Explore frequent patterns and similarities among
  data streams
• MAIDS can be applied to
• Network intrusion detection
• Telecommunication data flow analysis
• Credit card flaw prevention
• Web click streams analysis
• Financial data trend prediction
• Remote sensing data analysis

7
Features of MAIDS

• General-purpose tool for data stream analysis
• Process high rate and multi-dimensional data
• Adopt a flexible Tilted Time Window frame
• work
• Facilitate multi-dimensional analysis using a
• stream cube architecture
• Integrate multiple data mining functions
• Support user-friendly interface automatic
• analysis and on-demand analysis

8
Models of Tilted Time Window
Up to 7 days
Up to a year
Logarithmic (exponential) scale
2t
1t
4t
8t
16t
Time
Now
9
Why Use Tilted Time Window?

• It is impossible to save the history in a full
  scale due to
• the limited memory available in todays
  computers
• Recent data are usually more important than the
• historical data
• People are often interested in the recent
  changes in a
• fine scale, but the long term changes in a
  coarse scale
• Compress data without loosing information
• Time granularities can be configured based on
  the
• application requirement

10
Implementation of Tilted Time Window

• User definable
• Data structure Circular Queues
• Each queue is for a specific time granularity
• Self-maintained
• Aggregate and propagate to a coarser level
  when
• reaching the boundary

15 Minute 4 Quarter 24 Hour
11
Stream Query Engine

• Implemented based on H-Tree Cubing algorithm
• Use Tilted Time Window to track time-related info
• Support many query options

12
Query Options

• One time query continuous query
• Single query multiple parallel queries
• Drill-down roll-up queries
• Exact query approximate query
• Point time query duration time query
• Single dimension query multiple dimension query
• Visualization report

13
Stream Data Classifier

• Use Naïve Bayesian classification with boosting
• Use Tilted Time Window to track time-related
  info
• Can build model and predict alarming events

14
Features of Stream Classifier

• Use an efficient data structure AVC-list
• Classification models can be built automatically
  on requested time horizons and at specified time
  intervals
• Use multi-model evaluation and boosting to
  improve the model accuracy
• Constructed model can be immediately applied to
  predict events

15
Stream Pattern Finder

• Mine frequent patterns and sequential patterns
• Adopt extended FP-Growth algorithm
• Keep precise/compressed history in tilted time
  window.

16
Stream Cluster Analyzer

• Implemented based on Micro-Clustering algorithm
• Use Tilted Time Window to track time-related info
• Can find clusters in the stream data
• Can detect the evolution and changes