Polymorphic Buffer Overflow

1
Polymorphic Buffer Overflow

• Pallavi Garg

2

• If the data size is not checked, return pointer
  can be overwritten by user data
• Attacker places exploit machine code in the
  buffer and overwrites the return pointer
• When function returns, attackers code is
  executed

. . .
Buffer 2 (Local Variable 2)
Machine Code execve(/bin/sh)
New Pointer to exec code
Function Call Arguments
. . .
Smashed Stack
3

• Include NOPs in front of the executable code
• Execution will continue down the stack until it
  gets to the exploit

NOP NOP NOP NOP NOP Machine Code execve(/bin/sh)
Buffer 1 Space is overwritten
Return Pointer is overwritten
New Pointer to exec code
Function Call Arguments
. . .
Top of Memory
4

• IDS use signature matching techniques to detect
• A sequence of NOP statements
• Common shellcode commands
• /bin/sh

5

• LPR exploit against a HP JetDirect printer

6
Polymorphic Buffer Overflow

• Aim is to avoid detection NOP sled and the
  shellcode
• Code is morphed in order to defeat signature
  analysis
• In April, 2001, ADMutate released by K2
• ADMutate designed to defeat IDS signature
  checking by altering the appearance of buffer
  overflow exploit
• Uses techniques borrowed from virus writers

7
Mutation Engine

• NOP substituted is with operationally inert
  commands
• Intel Architecture has 55 NOP equivalent commands
• Shell code is encoded by XORing with a randomly
  generated key
• Return address is modulated
• Least significant byte altered to jump into
  different parts of NOPs

8
Decode Engine

• Need to decode the XORed shellcode
• Engine is also polymorphic
• Varying the assembly instructions to accomplish
  the same results in different ways
• Out of order decoding to vary the signature even
  more

9
ADMutate Tool

• An API that allows an exploit writer to integrate
  polymorphism directly into the exploit code.   
• include "ADMmutapi.h
• struct morphctl mctlp
• struct morphctl mut
• mut.upper 0 mut.lower 0
• mut.banned 0 mctlp mut
• mut.arch IA32
• init_mutate(mctlp)
• apply_key(buff, strlen(shellcode), nops-1,
  mctlp)
• apply_jnops(buff, nops-1, mut)
• apply_engine(buff, strlen(shellcode), nops-1,
  mut)

10
Some detection techniques been discussed by IDS
vendors

• Shellcode payload decrypt and detection with old
  shellcode payload signatures
• Same technique as used by antivirus software
• Open Issues
• How do you detect that a payload is encrypted?
• Which encoding technique has been used?
• Signatures to detect the decrypter engine
• Since this engine also mutates, IDS would have to
  check too many signatures
• Too many CPU cycles needed
• Decrypter engine emulation
• IDS emulates the code. If it finds code that
  seems to decrypt something in memory it raises a
  shellcode alarm
• Too many CPU cycles needed

11

• Best approach seems to be to go after the NOP
  sled and the decode engine
• K2s response
• Some will say that in a 1K overflow where
  approx 700bytes represents a 55/256 selection of
  possible codes, this could be a means for
  detection. However, I do not believe an IDS
  could do this very effectively, IA32 instructions
  are variable length (1-gt16bytes) and it is not a
  simple matter of doing byte-per-byte analysis of
  the code they will have to decode the stream
  (bitwise), and thanks to CISC encoding there are
  many branch's in this logic, leading to increased
  processing overhead in the detection process.

12

• Typically IDSs have about 25 success in
  detecting polymorphic shellcode
• Next Generation Security Technologies has a free
  NIDS
• http//www.ngsec.com/ngresearch/ngtools/
• ISS RealSecure
• Focuses on analyzing protocols to recognize
  context and then employs pattern-matching
  techniques where appropriate to positively
  identify malicious content.

13
References

• www.phrack.org
• www. ktwo.ca/security.html
• www.sans.org
• www.ngsec.com/docs/
• www.predictive.com