The Challenges of Evaluating Traffic Anomaly Detectors

1
The Challenges of Evaluating Traffic Anomaly
Detectors

• WebClass Reliable Labeling of Traffic Traces

Haakon Ringberg, Princeton University
2
Goals of PresentationEvaluating Anomaly
Detectors

• Highlight the Problem
• Background
• Wish list
• Status quo
• Solicit input
• Your thoughts on how to adress problem

3
A network in the Internet
4
Network anomalies
5
The anomaly detector

• Monitor health of network
• Real-time reporting of anomalies

6
Need an evaluation framework
Change detection using sketches
PCA
Reversible Sketches
Wavelets
NetworkAnomography
Kalmanfilters
Hierarchical Heavy Hitter Detectors
AutoFocus
7
Evaluator Wish List

• Obtained results are predictive of real-world
  performance
• Realistic background anomalous traffic
• Varied types, spatial locations, magnitude, etc
• Performance evaluation of detectors

8
Evaluator status quo

• Just as hard as anomaly detection!
• Anomalies are poorly defined
• Varying goals IDS versus worms?
• People dont share software/traces
• Proprietary/privacy
• Infeasible due to size
• No standard evaluation framework

9
Manual Labeling

• Domain experts certify the real anomalies
• Strengths
• Experts are the ultimate evaluators
• Detailed anomaly types
• Weaknesses
• Scalability
• Reproducibility
• No false negatives

10
Suite of detectors

• Real anomalies are those found by detectors ?1,
  ?2, ..., ?n
• Strengths
• Easy to set up
• With good detectors...
• Weaknesses
• No real false-negatives
• Even more circular evaluator evaluee

11
Anomaly signatures

• Anomalies have certain measurable properties
• Strengths
• Easy to set up
• Understandable metric
• Used by operators
• Weaknesses
• Rigid/arbitrary anomaly definitions
• Narrow false negatives

• any source which generated SYNs but no FINs for
  more than n destinations is posited as being a
  true port scan.

12
Synthetic ModelsAnomaly Injection

• Synthetic anomalies injected into collected
  traces
• Strengths
• Real traces
• Vary anomaly magnitude
• Weaknesses
• No false negatives
• Anomaly-background traffic interactions

13
Synthetic ModelsSimulation or Emulation

• Both anomalies and background traffic are
  simulated
• Strengths
• Fully controlled environment
• Complete false negative rates
• Weaknesses
• Scalability
• Realism

14
So whats the conclusion?

• No single evaluation technique is complete
• Essential to specify goal of evaluation
• An incomplete evaluation technique can be
  appropriate for a narrow purpose
• Simulation has been underused
• Can we build an evaluation framework that
  combines several of the techniques?

15
Where Does WebClass Fit In?

• Manual labeling is unavoidable
• Domain experts will be the first to identify new
  anomalies
• Manual labeling is not inherently evil
• Used in many other fields (parts-of-speech
  classification, image classification, etc.)
• But needs a more rigorous methodology
• Standard interface for labeling
• Ability to evaluate others labeled traces

16
WebClass Features

• Labeling Framework
• Upload traces
• Visually inspect timeseries traces
• See heavy hitter information
• Label anomalous events

• Repository
• Upload traces and associated labels
• Inspect and comment on others labeled traces
• Report statistics for individual labels

17
WebClass GUI
18
Thank you! Questions?

• Augustin Soule, Fernando Silveira, Christophe
  Diot, Jennifer Rexford