Intrusion Detection

1
Intrusion Detection

• Somesh Jha
• University of Wisconsin

2
Intrusion Detection

• Goal Discover attempts to maliciously gain
  access to a system

3
Network Intrusion Detection Systems (NIDS)

• Inspects packets at certain vantage points
• For example, behind the routers
• Look for malicious or anomalous behavior
• Much more fine-grained than firewalls
• Example drop a packet whose payload matches a
  certain string
• Called deep packet inspection

4
Network Intrusion Prevention System (IPS)

• NIDS are generally passive
• Raise alerts if something suspicious happens
• IPS are active
• Drop suspicious looking packages
• Route certain packets for further inspection
• Main challenge have to work at line speeds

5
Classification of NIDS

• Signature-based
• Also called misuse detection
• Establish a database of malicious patterns
• If a sequence of packets matches one of the
  patterns, raise an alarm
• Positives
• Good attack libraries
• Easy to understand the results
• Negatives
• Unable to detect new attacks or variants of old
  attacks
• Example
• Cisco, Snort, Bro, Tippingpoint, NFR,

6
Classification of NIDS

• Anomaly-based
• Establish a statistical profile of normal traffic
• If monitored traffic deviates sufficiently from
  the established profile, raise an alarm
• Positives
• Can detect new attacks
• Negatives
• High false alarm rate
• High variability in normal traffic
• Intruder can go under the radar
• Examples
• Mostly research systems

7
Classification of NIDS

• Stateless
• Need to keep no state
• Example raise an alarm if you see a packet that
  contains the pattern mellissa
• Positives
• Very fast
• Negatives
• For some attacks need to keep state

8
Classification of NIDS

• Stateful
• Keeps state
• Sometime need to do reassembly
• Reassemble packets that belong to the same
  connection, e.g., packets that belong to the same
  ssh session
• Quite hard! (out-of-order delivery)
• Positives
• Can detect more attacks
• Negatives
• Requires too much memory

9
Snort
logs, alerts, ...
malicious patterns
Filtered packet stream
libpcap
10
libpcap

• Takes the raw packet stream
• Parses the packets and presents them as a
• Filtered packet stream
• Library for packet capture
• Website for more details
• http//www-nrg.ee.lbl.gov/.

11
Malicious Pattern Example
alert tcp any any -gt 10.1.1.0/24 80 (content
/cgi-bin/phf msg PHF probe!)
action pass log alert
destination address destination port
source address source port
protocol
12
Malicious Patterns Example

• content /cgi-bin/phf
• Matches any packet whose payload contains the
  string /cgi-bin/phf
• Look at http//www.cert.org/advisories/CA-1996-06.
  html
• msg PHF probe!
• Generate this message if a match happens

13
More Examples
alert tcp any any -gt 10.1.1.0/24 60006010 (msg
X traffic)
alert tcp !10.1.1.0/24 any -gt 10.1.1.0/24
60006010 (msg X traffic)
14
How to generate new patterns?

• Buffer overrun found in Internet Message Access
  Protocol (IMAP)
• http//www.cert.org/advisories/CA-1997-09.html
• Run exploit in a test network and record all
  traffic
• Examine the content of the attack packet

15
Notional "IMAP buffer overflow" packet
052499-222758.403313 192.168.1.41034 -gt
192.168.1.3143 TCP TTL64 TOS0x0 DF PA Seq
0x5295B44E Ack 0x1B4F8970 Win 0x7D78 90 90
90 90 90 90 90 90 90 90 90 90 90 90 EB 3B
............... 5E 89 76 08 31 ED 31 C9 31 C0 88
6E 07 89 6E 0C .v.1.1.1..n..n. B0 0B 89 F3 8D 6E
08 89 E9 8D 6E 0C 89 EA CD 80 .....n....n..... 31
DB 89 D8 40 CD 80 90 90 90 90 90 90 90 90 90
1..._at_........... 90 90 90 90 90 90 90 90 90 90 90
E8 C0 FF FF FF ................ 2F 62 69 6E 2F 73
68 90 90 90 90 90 90 90 90 90 /bin/sh.........
16
Alert rule for the new buffer overflow
alert tcp any any -gt 192.168.1.0/24 143
(content"E8C0 FFFF FF/bin/sh" msg"New IMAP
Buffer Overflow detected!")
Can mix hex formatted bytecode and text
17
Advantages of Snort

• Lightweight
• Small footprint
• Focused monitoring highly tuned Snort for the
  SMTP server
• Malicious patterns easy to develop
• Large user community
• Consider the IRDP denial-of-service attack
• Rule for this attack available on the same day
  the attack was announced
• Commercial company (Sourcefire) behind it

18
Disadvantages

• Does not perform stream reassembly
• Attackers can use that to fool Snort
• Break one attack packet into a stream
• Pattern matching is expensive
• Matching patterns in payloads is expensive (avoid
  it!)
• Rule development methodology is adhoc

19
Host-based ID

• Monitor interaction between a specific program
  and OS
• Raise an alarm if suspicious system calls are
  observed
• Unlike NIDS monitoring happens at the end hosts
• Need to model
• Unusual behavior
• Normal behavior

20

• Goal Discover attempts to maliciously gain
  access to a system

• Misuse Detection
• Specify patterns ofattack or misuse
• Ensure misuse patternsdo not arise at runtime
• Snort
• Rigid cannot adaptto novel attacks

• Anomaly Detection
• Learn typical behaviorof application
• Variations indicatepotential intrusions
• IDES
• High false alarm rate

• Specification-Based
• Monitoring
• Specify constraints uponprogram behavior
• Ensure execution doesnot violate specification
• Our work Ko, et. al.
• Specifications can becumbersome to create

21
Specification-Based Monitoring

• Two components
• Specification Indicates constraints upon program
  behavior
• Enforcement How the specification is verified at
  runtime or from audit data

22
SpecificationEnforcement
Analyst orAdministrator
TrainingSets
StaticBinary CodeAnalysis
StaticSource CodeAnalysis
ExecutionObeys Static Ruleset
ExecutionMatches Model of Application
23
Representative Work by Ko, et al.

• Specification Programmers or administrators
  specify correct program behavior
• Enforcement At runtime, only allow actions that
  match the specified policy

PROGRAM fingerd read(X) - worldreadable(X) bin
d(79) write(/etc/log) exec(/usr/ucb/finger
) END
24
SpecificationEnforcement
Analyst orAdministrator
TrainingSets
StaticBinary CodeAnalysis
StaticSource CodeAnalysis
ExecutionObeys Static Ruleset
ExecutionMatches Model of Application
25
Representative Work by Forrest, et al

• Specification Learn correct program behavior
  with training
• Record sequences of system calls
• Enforcement Only accept behaviors similar to
  learned patterns
• Example system STIDE

26
Training

• Repeatedly run the program, varying the input
• For some n, record all sequences of n system
  calls observed
• n depends upon the program
• End result database of n-tuples of system calls

27
cat (print file contents)

• geteuid, getuid, getegid, getgid, fstat, open,
  fstat, lseek, mmap, read, memcntl, write, lseek,
  munmap, lseek, close, close, exit

• geteuid, getuid
• getuid, getegid
• getegid, getgid
• getgid, fstat
• fstat, open / lseek
• open, fstat
• lseek, mmap / munmap / close
• mmap, read
• read, memcntl
• memcntl, write
• write, lseek
• munmap, lseek
• close, close / exit

28
Enforcement

• Monitor system calls generated by application
• Ensure that the last n calls match a sequence in
  the database
• Option Allow slight deviation from database
• Training set may have been incomplete

29
cat (print file contents)

• geteuid, getuid, getegid, getgid, fstat, open,
  fstat, lseek, mmap, read, memcntl, write, lseek,
  munmap, lseek, close, close, exit
• Accepts incorrect system call sequences
• geteuid, getuid, getegid, getgid, fstat, lseek,
  close, exit

• geteuid, getuid
• getuid, getegid
• getegid, getgid
• getgid, fstat
• fstat, open / lseek
• open, fstat
• lseek, mmap / munmap / close
• mmap, read
• read, memcntl
• memcntl, write
• write, lseek
• munmap, lseek
• close, close / exit

30
Drawbacks

• Accepts incorrect call sequences
• Due to window-based approach with ambiguity
• Opportunity for attack sequence to go undetected
• Only learn behaviors exercised in training set
• Not all execution paths followed
• Users must construct valid training sets
• Users must determine window size

31
Drawbacks

• Specification may over-fit the data
• If training on real data, training set may
  contain exploits
• Learn exploit pattern as normal

32
SpecificationEnforcement
Analyst or Administrator
TrainingSets
StaticBinary CodeAnalysis
StaticSource CodeAnalysis
ExecutionObeys Static Ruleset
ExecutionMatches Model of Application
33
Our Approach

• Specification Static analysis of binary code
• Specifications are automatically generated
• Not reliant upon programmersto produce accurate
  specifications
• Analyzes all execution paths
• Source code may be unavailable

function save sp, 0x96, sp cmp i0, 0 bge
L1 mov 15, o1 call read mov 0, o0 call
line nop b L2 nop L1 call read mov i0,
o0 call close mov i0, o0 L2 ret restore
34
Our Approach

• Enforcement Operate an automaton modeling
  correct system call sequences
• Dynamic ruleset
• More expressive than static ruleset of Ko, et. al.

35
Non-Deterministic Finite Automaton (NFA)

• Structure
• States
• Labeled edges between states
• Edge labels are input symbols call names
• Path to any accepting state defines valid
  sequence of calls

36
Our Approach

• Enforcement Operate an automaton modeling
  correct system call sequences
• Dynamic ruleset
• More expressive than static ruleset of Ko, et. al.

37
The Binary View (SPARC)

• function
• save sp, 0x96, sp
• cmp i0, 0
• bge L1
• mov 15, o1
• call read
• mov 0, o0
• call line
• nop
• b L2
• nop
• L1
• call read
• mov i0, o0
• call close
• mov i0, o0
• L2
• ret
• restore

• function (int a)
• if (a lt 0)
• read(0, 15)
• line()
• else
• read(a, 15)
• close(a)

38
Control Flow Graph Generation

• function
• save sp, 0x96, sp
• cmp i0, 0
• bge L1
• mov 15, o1
• call read
• mov 0, o0
• call line
• nop
• b L2
• nop
• L1
• call read
• mov i0, o0
• call close
• mov i0, o0
• L2
• ret
• restore

39
Control Flow GraphTranslation
40
Control Flow GraphTranslation
41
Interprocedural ModelGeneration
A
read
read
close
line
42
Interprocedural ModelGeneration
A
read
read
line
write
close
line
43
Interprocedural ModelGeneration
B
A
line
read
read
line
write
close
close
line
44
Interprocedural ModelGeneration
B
A
line
read
read
line
write
close
close
45
Interprocedural ModelGeneration
B
A
read
read
line
write
close
close
46
PossiblePaths
B
A
read
read
line
write
close
close
47
PossiblePaths
B
A
read
read
line
write
close
close
48
ImpossiblePaths
B
A
read
read
line
write
close
close
49
ImpossiblePaths
B
A
read
read
line
write
close
close
50
Adding ContextSensitivity
B
A
read
Y
read
line
X
write
close
close
Y
X
51
PDA State Explosion

• e-edge identifiers maintained on a stack
• Stack may grow to be unbounded
• Solution
• Bound the maximum size of the runtime stack
• A regular language overapproximation of the
  context-free language of the PDA

X
52
Data Flow Analysis

• Argument recovery
• Statically known arguments constrain remote calls
• Reduces opportunity given to attackers

function save sp, 0x96, sp cmp i0, 0 bge
L1 mov 15, o1 call read mov 0, o0 call
line nop b L2 nop L1 call read mov i0,
o0 call close mov i0, o0 L2 ret restore
53
Call Site Renaming

• Give each monitored call site a unique name
• Associates arguments with call sites
• Obfuscation
• Reduces nondeterminism

function save sp, 0x96, sp cmp i0, 0 bge
L1 mov 15, o1 call read mov 0, o0 call
line nop b L2 nop L1 call read mov i0,
o0 call close mov i0, o0 L2 ret restore
54
Call Site Renaming

• Give each monitored call site a unique name
• Associates arguments with call sites
• Obfuscation
• Reduces nondeterminism

function save sp, 0x96, sp cmp i0, 0 bge
L1 mov 15, o1 call _638 mov 0, o0 call
line nop b L2 nop L1 call read mov i0,
o0 call close mov i0, o0 L2 ret restore
55
Call Site Renaming

• Give each monitored call site a unique name
• Associates arguments with call sites
• Obfuscation
• Reduces nondeterminism

function save sp, 0x96, sp cmp i0, 0 bge
L1 mov 15, o1 call _638 mov 0, o0 call
line nop b L2 nop L1 call _83 mov i0,
o0 call close mov i0, o0 L2 ret restore
56
Call Site Renaming

• Give each monitored call site a unique name
• Associates arguments with call sites
• Obfuscation
• Reduces nondeterminism

function save sp, 0x96, sp cmp i0, 0 bge
L1 mov 15, o1 call _638 mov 0, o0 call
line nop b L2 nop L1 call _83 mov i0,
o0 call _1920 mov i0, o0 L2 ret restore
57
Call Site Renaming

• Give each monitored call site a unique name
• Associates arguments with call sites
• Obfuscation
• Reduces nondeterminism

read
read
close
line
58
Call Site Renaming

• Give each monitored call site a unique name
• Associates arguments with call sites
• Obfuscation
• Reduces nondeterminism

_638
_83
_1920
line
59
Technical Challenges

• Integrating other specification sources
• Optimal null call insertion
• C vtable analysis