Scalable Network-based Buffer Overflow Attack Detection - PowerPoint PPT Presentation

1 / 31
About This Presentation
Title:

Scalable Network-based Buffer Overflow Attack Detection

Description:

Solution: automatically signature-generating approaches. Anomaly ... Generalized Signature. ANCS 2006. Two Factors for a Successful Buffer ... Signature ... – PowerPoint PPT presentation

Number of Views:142
Avg rating:3.0/5.0
Slides: 32
Provided by: yanl9
Category:

less

Transcript and Presenter's Notes

Title: Scalable Network-based Buffer Overflow Attack Detection


1
  • Scalable Network-based Buffer Overflow Attack
    Detection

Tzi-cker Chiueh Computer Science Department Stony
Brook University Stony Brook, NY,
U.S.A. chiueh_at_cs.sunysb.edu
Fu-Hau Hsu Department of Computer Science and
Information Engineering National Central
University Taoyuan, Taiwan, R.O.C. hsufh_at_csie.ncu.
edu.tw
Fanglu Guo Symantec Research Laboratory Cupertino,
CA, U.S.A. fanglu_guo_at_symantec.com
2
Virulence of Buffer Overflow Attacks
  • Buffer overflow attack is arguably the most
    widely used and thus most dangerous attack method
    used today.
  • Most Internet Worms use it to proliferate
    themselves.
  • It accounts for more than 50 of all the security
    vulnerabilities recorded by CERT.

3
Proposed Solutions
  • Compiler Transformation
  • Stack Guard, RAD, Address Obfuscation
  • Library Rewriting
  • OS
  • Non-executable Stack
  • Instruction Set
  • Hardware
  • AMD Athlon-64

4
Discrepancy between Theory and Practice
  • In theory, these efforts have largely solved the
    buffer overflow attack problem.
  • In practice, however, new buffer overflow
    vulnerabilities are still discovered and reported
    on a routine basis.

5
  • substantial modification
  • substantial resistance
  • Why?

6
A Solution to the above Dilemma --Nebula
  • Nebula
  • A network-based buffer overflow attack detection
    mechanism
  • Observe the network traffic only to detect BOAs
  • Currently version is developed for Linux
    paltforms.

7
Existing Network-based Intrusion Detection System
(NIDS)
  • Misuse intrusion detection
  • Zero-day BOAs
  • Labor-Intensive
  • Solution automatically signature-generating
    approaches
  • Anomaly intrusion detection
  • False Positive

8
  • Generalized Signature

9
Two Factors for a Successful Buffer
Overflow-style Attack
  • A successful buffer overflow-style attack should
    be able to overflow the right place (e.g. the
    place to hold a return address with the correct
    value (e.g. the address of injected code entry
    point)).

10
Non-predicable Offset and Entry Point Address
return address
buffer where the overflow start
injected code
address of injected code entry point.
offset between the beginning of the overflowed
buffer and the overflow target.
The offset and the entry point address are
non-predicable. They can not decided by just
looking the source code or local binary code.
11
Non-predicable Offset
  • For performance concerns, most compilers dont
    allocate memory for local variables in the order
    they appear in the source code, sometimes some
    space may be inserted between them. (Source Code
    doesnt help)
  • Different compiler/OS uses different allocation
    strategy. (Local binaries dont help)
  • Address obfuscation insert random number of space
    between local variables and return address.
    (Super good luck may help)

12
Non-predicable Entry Point Address
webserver a b security
fhsu_at_ecsl
system data
0xbfffffff
environment variables
argument strings
command line arguments and environment variables
env pointers
argv pointers
argc
13
Strategies Used by Attackers to Increase Their
Success Chance
  • Repeat address patterns.
  • Insert NOP (0x90) operations before the entry
    point of injected code.

14
Indispensable Elements of BO-style Attacks
  • The Address
  • For buffer overflow attacks, it is the address of
    the entry point of injected code.

15
Linux Process Memory Layout
0xffffffff
kernel address space
0xc0000000
address space of addresses of
injected code and frame pointers
(Stack Address Zone)
user stack
esp
8M
for Shared libraries, including libc functions
0x40000000
brk
run-time heap
data and code
16
Size of Stack Address Zone
  • The default maximum size of a processs user
    space stack is 8 Mbytes.
  • However, according to Ditzel et al., the average
    function frame size is 28 bytes.
  • Therefore, the majority of program are not
    supposed to use a 2Mbyte stack.
  • In our test, a 8k stack is enough to identify all
    10 remote exploit strings.

17
Repeating Times and Values of Return Addresses
2k stack --- 0xbffffffff 0xbfffe000
18
A Property of Stack Addresses
  • The leading byte of any words that contain a
    stack address corresponds to a non-printable
    ASCII character.

19
Generalized Signature
  • Signature of a stack smashing buffer overflow
    attack
  • If a sub-string of a traffic string could be
    interpreted as a stack address that repeats 3 or
    more times, it is alarmed as a buffer overflow
    attack string.

20
  • Contextual Analysis

21
Bypassing Detection
  • Patient attackers could bypass detection based on
    repeating address signature by repeating
    addresses no more than 2 times.
  • PS All the 10 remote exploit code we tested
    repeat at least 4 times.
  • Attackers repeat the addresses to increase their
    chance to success. In other words, it is very
    likely that without the repeat, attackers will
    fail many times before getting a successful one.

22
Unsuccessful Attacks
  • Buffer overflow-style attacks will destroy
    targeted processs address space which in turn
    usually will crash the attacked process.
  • In order to recycle valuable system resources, OS
    will close the sockets opened by crash processes
    automatically.
  • On both Linux and Windows, when a program is
    crashed, the OS will terminate all the programs
    pending socket connections by sending out an RST
    packet to the communicating hosts on its behalf.

23
Server Termination Signature
  • After forwarding a sub-string which could be
    interpreted as a single stack address, Nebula
    detects that the server closes the TCP connection
    without sending any data, then the traffic string
    is deemed as a buffer overflow attack string.
  • Future traffic coming from the same hosts will be
    blocked or examined thoroughly.

24
Will Normal Traffic Behavior the Same Way?
  • HTTP Protocol (RFC 2616) works in the
    request-reply way. (After the request, there will
    be a reply before the server close the
    connection)
  • SMTP protocol (RFC 2821), for e-mail, and FTP
    protocol(RFC 959) use QUIT command to close a
    connection. (QUIT can not be interpreted as a
    stack address.)

25
  • Payload Bypassing

26
Payload Bypassing
  • Payload bypassing tries to avoid packet analysis
    for as much traffic as possible.
  • Because most buffer overflow attacks take place
    during the exchange of control messages, it is
    safe to ignore the bulk of data that is
    downloaded as uninterpreted bytes.
  • For example, in an FTP session, data transferred
    over the data connection can never be used to
    mount a buffer overflow attack against the FTP
    program because the FTP program does not
    interpret them.

27
Internet Traffic Statistic
  • From CacheLogics measurement on USA, Europe, and
    Asia backbone in June 2004, HTTP and P2P packets
    accounted for more than 70 of the total traffic.

28
Percentage of Payload
  • Percentage of payload in the traffic when each of
    the four protocols that Nebula can recognize is
    used to transfer files of a total size of 1.22
    Gbytes.

29
Number of False Positives without Payload
Bypassing
  • Number of false positives under the our sample as
    reported by Nebula.
  • The minimal number of times the attack pattern is
    repeated is assumed to be 1, 2, 3 or 10, and the
    stack size tested is 2Mbytes, 16Kbytes, or
    8Kbytes. In each entry the left is the number of
    false positives for RTL attacks, whereas the
    right is the number of false positives for CI
    attacks.
  • The sample includes 134966 TCP connections and
    about 1.582 Gbytes of data.

30
Number of False Positives with Payload Bypassing
  • The number of false positives in the test traffic
    associated with different protocols after
    applying payload bypassing is negligible even
    when the attack pattern repetition count is 1.

31
Throughput Comparison
  • The throughput of Nebula under a test HTTP
    connection when different options are turned on.
    With payload bypassing, Nebula can perform buffer
    overflow attack detection and still achieve a
    throughput higher than a generic Linux router.
Write a Comment
User Comments (0)
About PowerShow.com