Experiences with tools for network anomaly detection in the G

1
Experiences with tools for network anomaly
detection in the GÉANT2 core

• Maurizio Molina, DANTE
• COST TMA tech. Seminar
• Samos, 23rd Sep 2008

2
The GÉANT Network

• DANTE operates GÉANT2
• Backbone network for National Research and
  Education Networks in Europe
• 30 NRENs, 2 global connectivity providers (Telia
  and GCrossing), peerings with other research
  networks (Abilene, Canarie, Clara, TEIN2, SINET)

3
The GÉANT Network (IP layer)

• 20 Juniper routers
• tenths of GBit/s of aggregated traffic
• Main accesses and the backbone 10Gbit/s

Pls see www.dante.net
4
The Services

• So. Just a big pipe? No!
• Services
• Dedicated L1-L2 circuits via multiple
  technologies
• Performance Monitoring services (perfSONAR)
• Support for federation of National AA
  Infrastructures (eduGAIN) and wireless roaming
  (eduROAM)
• Security Service

NEW!
Very NEW!
5
The visionenhance NRENs security

• NRENs have their ( - evolved) CERTs to deal
  with security
• and DANTE can filter traffic on GÉANT upon NRENs
  request.
• ! BUT !
• Can we be more proactive to NREN CERTs exploiting
  the visibility of the GN2 core?

6
The vision (cont.)enhance NRENs security

• Approach NetFlow ( Routing data) good
  processing tools

• Netflow collected on all peering interfaces
• 1 / 1,000 Sampling
• 3k flows/s

7
Proof of concept Can we identify anomalies in
the core?

• Anomalies are often hidden

• Requirements
• High detection rate
• Low false positives
• Anomaly classification
• Evidence collection

NfSen
8
From volume to IP feature entropies

• IP features entropies
• Simple linear filter

9
Drilling down on peaks

• IRC server in Slovenia, receiving a lot of 60
  bytes syn pkts on port 6667, mainly from a /16
  Subnetwork of an University in the Netherlands.
• Likely a BotNet war?

• -Concentration of DST IPs and DST ports
  receiving flows
• -Dispersion of SRC IPs and SRC ports

10
Drilling down on peaks (cont.)

• - Concentration of SRC and DST IPs and SRC ports
• - Dispersion of DST ports

• Portscan of host in CARNET, from 4 hosts, 29
  bytes packets

11
Open source tools

• Results
• anomalies are observable in the GÉANT2 core
• Novel methodologies (IP Features entropy) for
  their classifications are applicable
• Limits
• NfSen does not fuse NetFlow and Routing data
• Extensions would need to be run (and tuned) on
  all ingress/egress points
• No support, no guaranteed development

12
Commercial tools

• Test started Jun 08 (3 tools)
• Tool 1
• PCA, entropy
• Tool 2
• Large scale DDoS and Worm spread
• Tool 3
• Per host behaviour

13
Tool 1 (as a security tool)

• Two main novel elements
• Principal Component Analysis (PCA)
• Both Volume and IP features Entropy anomaly
  detection
• Address what makes anomaly detection a complex
  task
• PCA single parameter to control detection
  sensitivity, even if anomalies are attributed to
  specific OD pairs
• Entropy Detection of both low volume (scans) and
  high volume (DoS) anomalies

14
Demo.

• . Or Screenshots.

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
Tool 2

• Well-established (and expensive!) solution for
  detecting large events
• Originally based on large volume shifts only
• Now enhanced to give alerts on fingerprints
  (e.g. communication with CC servers)
• Shared by (part) of the user community (50 out of
  120)
• No usage of routing data
• though zones can be manually created via BGP
  prefixes lists
• Traditional threshold based detection (although
  adaptive)

24
Tool 3

• Per host behavioural analysis
• rather complex scoring system to distinguish
  normal from abnormal behaviour. Proprietary
  algorithms
• Doesnt use routing info
• though zones can be manually created via BGP
  prefixes lists
• Potentially attractive methodology
• Concerns on scalability and accuracy with 1,000
  sampling

25
lessons learnt and directions for research

• Manual validation is required to confirm/correct
  anomalies
• More automatic intelligence to help this process
• Fusion with other data sources (router logs?
  Honeynets?)
• Detection space of 3 tools often disjoint
• (Standard) anomaly injection
• Operations need supported tools to support
  services
• If choice is among published but not a tool or
  secret but supported and (claiming to) work gt
  risk to stick to those!
• Fill the gap towards TOOLS!

26
Thank you!

• maurizio.molina_at_dante.org.uk