Toward Selfdirected Intrusion Detection

1
Toward Self-directed Intrusion Detection
Summer, 2006

• Paul Barford
• Assistant Professor
• Computer Science
• University of Wisconsin

2
Problem space - the good

• Network security analysts have many tasks
• Abuse monitoring
• Audit and forensic analysis
• NIDS/Firewall/ACL configuration
• Vulnerability testing
• Policy maintenance
• Liaison activities
• Network management
• End host management

3
Problem space - the bad

• Adversaries are smart and getting organized
• Vulnerabilities and threats are significant
• Botnets, worms, scans, viruses, DDoS
• 64K addresses gt 5M scans/day
• Risks to infrastructure are significant
• Intranet systems, data
• Internet power grid, air traffic, etc.
• Recovery costs are significant
• Eg. Slammer worm recovery 1B (Wired)

4
Problem space - the ugly

• Network intrusion detection systems (NIDS)
• Static signatures - hard to tune and maintain
• Lots of alarms
• Scalability problems
• Firewalls and intrusion prevention systems
• Limited capability
• Limited resilience
• Bulletin boards and commercial services
• Often are not timely

5
Our objective

• Network situational awareness and defense based
  on self-directed, collaborative systems
• Situational awareness An accurate set of
  information about ones environment scaled to a
  specific level of interest
• Self-directed Resilient systems that can adapt
  to changing conditions in (near) real time

6
Architectural components

• Monitoring unused address space
• Eg. iSink (Yegneswaran et al., RAID 04)
• Analyzing data from unused address space
• Eg. BroSA (Yegneswaran et al. HotNets05)
• Automatic generation of resilient signatures
• Eg. Nemean (Yegneswaran et al., USENIX Security
  05)

7
Scalable attack monitoring

• Internet Sinks (iSinks) are systems designed to
  monitor unused address space
• Augment network telescopes
• Offers opportunity to significantly improve
  perspective (types, diversity, volume) on abuse
• Packets are (nearly) all malicious
• Enables active responses
• Increases opportunity to share data
• Lots of unused space around its just hard to
  get access to it!
• We monitor four class Bs and one class A
• Other monitors we have access to (UCB, PlanetLab,
  UMich/Predict)

8
iSink architecture

• Objectives highly scalable system with packet
  capture and active response capabilities
• iSink Implementation
• Passive component Argus
• libpcap-based monitoring tool
• Active component Click modular router
• Library of stateless responders to collect
  details of intrusions
• NAT filter
• Balances reduction in network traffic with
  richness of data collected

9
Active responders
10
Case study deployment

• Campus iSink
• Unused space on four class Bs
• Peak rate of 1500 connect requests/sec.
• TCP dominates since campus border filters port
  1434
• Service provider iSink
• Entire class A
• Peak rate of 30K connect requests/sec.
• UDP dominates with NetBIOS probes
• High level observation perspectives provided by
  individual iSinks are different
• Characteristics of Internet Background
  Radiation
• R. Pang, V. Yegneswaran, P. Barford, V. Paxson
  and L. Peterson - IMC 04

11
Campus iSink - typical week
12
Provider iSink - typical week
13
Activities on ports (port 135)

• Distribution of exploits varies with network
• 170 byte requests on Class A
• Blaster, RPC-X1 all 3 networks
• Welchia LBL
• Empty connections
• UW Networks

14
A few other interesting tidbits

• SMTP hot spot
• Large amount of SMTP traffic on the provider
  iSink
• 4.5M scans from 14K hosts on one day
• The SMTP responder revealed the sources were
  misconfigured wireless-routers/firewalls from
  D-link
• Emails were firewall logs
• D-link has been notified and is looking into
  this
• Differentiated responses
• Question Does active response traffic affect
  probe traffic
• Answer Yes!
• Response to connection attempts results in an
  increase in probe traffic

15
Network situational awareness

• Honeynet analysis via iSink is off-line
• Useful for forensic investigation
• Real time monitoring is a critical requirement
  for network situational awareness
• Goal 1 to enable security analysts to tune the
  perspective and timeliness of their monitor
• Goal 2 to accomplish goal 1 efficiently,
  scalably, robustly
• While NIDS provide important perspective, can
  iSinks be adapted to enhance network situational
  awareness?

16
Real-time honeynet reports

• Bro plug-in for situational summary generation
• Periodic reports
• New events (based on Bro policy database)
• High variance events (relative volume)
• Top profiles (overall volume)
• Adaptive - generates new policies on the fly
• NetSA in depth
• Goal identify and classify large events quickly
• On-going

17
What about IDS/IPS?
You
Signaturedatabase
IDS
Packettraces
IPS
18
Typical approach
Signaturerepository
You
Signaturedatabase
Not you
IDS
Packettraces
IPS
19
Reality of typical approach
Signaturerepository
You
Signaturedatabase
Not you
IDS
Packettraces
IPS
20
Wouldnt it be nice if
SignatureGenerator
Signaturedatabase
IDS
Packettraces
IPS
21
Signature generation challenge

• Specific signatures
• Identify only characteristics of attack profiles

• General signatures
• Match variants of known attack profiles

Balance specificityand generality
22
Protocol-aware signatures
23
Nemean architecture
iSink packets
Protocolsemantics
ConnectionClustering
Signature Generalizer
FlowAggregator
ServiceNormalizer
DataCollector
SessionClustering
IDS/IPSsignatures
24
Flow Aggregation HTTP semantics
Session
Source IP attacker Destination IP honeynet
Connection
Source port 2492 Destination port 80
Response
Request
weight 1 weight 0
weight 1000 weight 1 weight 50
Method URL Headers
Code Reason
25
Clustering and generation

• Applied to both connection and session data
• Star clustering algorithm Aslam et al. 1999
• Robust to data ordering
• Algorithm determines number of clusters
• We experimented with several similarity metrics
• Automata learning via PFSA generalization
• Build automaton that accepts all instances in a
  cluster
• Compute probability that each edge is traversed
• Merge states when probabilistically
  indistinguishable
• Add transitions representing reordering
  repetition

26
Connection signature generation
C1C2
GET
/scripts
/root.exe?
/cdir
404
27
Session signatures
Welchia
C3
C3
C4
GET /
200
C4
C5
C3
C5
C5
C5
Connection-level
Session-level
28
Experiments

• Built a simple IDS-like system to test signatures
  against packet traces off line
• Trained on iSink campus data
• HTTP 2 days 25,587 connections
• NetBIOS 2 days 38,722 connections
• 22 connection and 7 session signatures
• Detection effectiveness 99.9
• Test period 7 days 2,846,783 connections
• False alarms and misdiagnoses 0
• U.Wisc. HTTP production network packet trace
• 19,000 clients 4,400 servers
• Test period 8 hours 194,001 connections

29
Session level detection - HTTP
30
Connection level detection - NetBIOS
31
False alarm summary details

• Nemean 0
• Patched Linux/Apache server
• Snort
• About 88,000 with full signature set
• About 1,500 with tuned signature set where
  noisy signatures were manually disabled

32
Summary

• Intrusion activity persists on a massive scale
• Extreme diversity and growing volume
• Our objective Network situational awareness
• Components of our NetSA infrastructure
• iSinks offer an important perspective on
  intrusion activity
• BroSA enables iSinks to be used for network
  situational awareness
• Nemean generates resilient NIDS signatures
• Lower false alarm rate
• More precise classification

33
Current activities

• Real-time Nemean
• iSink Nemean in a box
• Includes regular expression matcher
• Includes web-based front end
• IDS implementation
• Tunable signature generation
• Running on 2.6 Linux kernel
• Live network test and evaluation
• Fall 06

34
Acknowledgements

• Vinod Yegneswaran
• Jon Giffin
• Somesh Jha
• Vern Paxson
• David Plonka
• Michael Hare
• Bill Jensen

35
Signature construction time - HTTP
36
Snort false alarms - HTTP
37
Nemean false alarms - NetBIOS
38
Nemean false alarms - HTTP