Distributed Network Monitoring in the Wisconsin Advanced Internet Lab

1
Distributed Network Monitoring in the Wisconsin
Advanced Internet Lab

• Paul Barford
• Computer Science Department
• University of Wisconsin Madison
• Spring, 2002

2
Motivation

• Many applications that run over the Internet have
  minimum performance requirements
• The network is one of the two possible sources of
  poor performance
• Wide area network behavior is unpredictable
• IP networks are best effort
• Constant change is normal
• Quality of service capability is not widely
  deployed
• Will it ever be available?

3
Monitoring is a First Step

• Accurate monitoring of network state can enable
  application adaptivity and improved network
  management
• Data provides basis for improved models and
  protocols
• There are many challenges in network monitoring
• All features of the Internet make monitoring
  difficult
• When, where, what, how
• Todays focus
• Network monitoring efforts at Wisconsin
• Combining monitoring and analysis to understand
  network traffic anomalies

4
The Wisconsin Advanced Internet Lab

• Next generation environment for network research
• Our focus performance, management, security
• Platform for testbeds storage, grid computing ,
  
• Internal environment
• Instances of end-to-end-through-core Internet
  paths
• External environment
• Measurement nodes deployed across the Internet

5
WAILs External Environment

• Existing infrastructure
• WAWM systems (10)
• Surveyor systems (60)
• Partnership with Advanced Systems
• NIMI systems (45)
• Partnership with PCS and ICIR
• Condor/Grid Infrastructures
• Prototype system is under development
• Passive flow measurements
• FlowScan data from UW, Internet2, others(?)

6
WAILs Internal Environment

• Complement to external facilities
• Hands-on test bed which creates paths identical
  to those in the Internet from end-to-end-through-c
  ore
• Variety of highly configurable equipment
• Why do we need an internal lab?
• Enables instrumentation and measurement of entire
  end-to-end system
• Enables new systems and protocols to be
  implemented in places where access is not
  possible in wide area
• Vision of internal lab New means for doing
  network research
• Status Significant commitment from industry
  partners (Cisco, EMC, Fujitsu) and the university
  rev. 1.0 by 5/1/02

7
Distributed Anomaly Detection

• Motivation Anomaly detection and identification
  is an important task for network operators
• Operators typically monitor by eye using SNMP or
  IP flows
• Simple thresholding is ineffective
• Some anomalies are obvious, other are not
• Focus Characterize and develop distributed
  means for detecting classes of anomalies
• Network outages, Flash crowds, Attacks,
  Measurement failures
• Approach Use statistical and wavelet techniques
  to analyze anomalies from IP flow and SNMP data
  from UW and other sites
• Implications Tools and infrastructure which
  quickly and accurately identify and adapt to
  traffic anomalies

8
Characteristics of Normal traffic
9
Our Approach to Analysis

• Analyze examples of each type of anomaly via
  statistics, time series and wavelets (our initial
  focus)
• Wavelets provide a means for describing time
  series data that considers both frequency and
  scale
• Particularly useful for characterizing data with
  sharp spikes and discontinuities
• More robust than Fourier analysis which only
  shows what frequencies exist in a signal
• Tricky to determine which wavelets provide best
  resolution of signals in data
• We use tools developed at UW Wavelet IDR center
• First step Identify which filters isolate
  anomalies

10
Analysis of Normal Traffic

• Wavelets easily localize familiar daily/weekly
  signals

11
Example Anomaly Attacks

• DoS sharp increase in flows and/or packets in
  one direction
• Linear splines seem to be a good filter to
  distinguish DoS attacks

12
Characteristics of Flash Crowds

• Sharp increase in packets/bytes/flows followed by
  slow return to normal behavior eg. Linux releases
• Leading edge not significantly different from DoS
  signal so next step is to look within the spikes

13
Characteristics of Network Anomalies

• Typically a steep drop off in packets/bytes/flows
  followed a short time later by restoration

14
Summary and Conclusion

• Accurate network monitoring is essential for
  improving application performance and network
  management
• The Wisconsin Advanced Internet Lab provides a
  unique environment for network monitoring
• Wavelets are an effective means for identifying
  anomalous behavior in data gathered from IP flow
  and SNMP interface monitors
• Details on distributed and coordinated monitoring
  and analysis available this spring