J. A. Drew Hamilton, Jr., Ph.D.

1
Auburn Information Assurance Laboratory
J. A. Drew Hamilton, Jr., Ph.D. Director,
Information Assurance Laboratoryand Associate
Professor Computer Science Software Engineering
FAX [phone] hamilton_at_eng.auburn.edu
drew_at_drew-hamilton.com
107 Dunstan Hall Auburn University, AL
36849-5347 http//www.eng.auburn.edu/users/hamilto
n
2
Software Vulnerability Analysis

• Homeland Security Red Teaming
• Simulations as Software
• Operating Systems
• Security by Obscurity
• Many Eyes Phenomenon
• Buffer Overflows
• Reverse Engineering
• Code Obfuscation

3
Homeland Security Red Teaming

• Directs intergovernmental coordination
• Applies to DOD
• Specifies Red Teaming
• Viewing systems from the perspective of a
  potential adversary
• Target hardening
• Looking for weakness in existing security measures

4
Typical DoD Security ModelNuclear Weapons
Training
Weapons Effects Calculations
notional weapons data
unclassified results
actual weapons data
classified results

• Assumptions
• Calculating nuclear and chemical weapons effects
  are already well known, only the actual weapons
  capabilities are classified
• The calculations themselves do not reveal
  sensitive information about training, tactics and
  procedures used in nuclear/chemical targeting

Is this model appropriate for missile defense
simulations?
5
Attacking Simulations as Software

• OBJECTIVES
• Look for the underlying models the simulation is
  constructed from
• Compromise training, tactics and procedures used
  in missile defense
• Compromise weapons and systems performance data

SimulationSoftwareProgram
SimulationOutputs
SimulationInputs (files orinteractive)

• Sensitivity Analysis of output based oninput
  changes
• One off test cases to examine relationships

• Exploitation of operatingsystem vulnerabilities
• Analysis of installed files
• Decompilation and disassembly of targeted
  executables

• Experimentation w/open source system data
• Privilege escalationvia buffer overflows
• Analysis of bounds checking if implemented

6
Operating Systems

• Distributed System vulnerabilites
• User Space versus Kernel Space
• Process space protection
• A single process is not allowed to access any of
  the memory allocated to other processes directly
• Additionally, no process can directly access the
  memory marked as in use by the operating system
• Windows NT/2000 and ALL UNIX systems provide
  process space protection
• Windows 95/98/ME do NOT provide process space
  protection

7
Handheld OSs

• Palm Pilot example
• any file stored on an internet-enabled PalmOS
  device is accessible by any application running
  on the Palm
• Generally, in an advanced OS, user-level
  processes need to use kernel services to access
  devices

8
Security by Obscurity

• Hide the source code and only release the
  executable.
• False belief that code compiled into binary
  remains secret just because the source is not
  available.
• Java byte code is particularly vulnerable
• Netscape POP (post office protocol) 1999
• password with weak cryptography
• stored in windows registry
• experimentation with XOR on password strings
• pattern detected
• encryption algorithm reverse engineered

9
Many Eyes Phenomenon

• In general, letting more developers scrutinize
  source code makes it more likely that
  security-related bugs are found and repaired in a
  timely manner.
• Code is generally reviewed for economic motives
• Many developers are not security experts
• C Traps and Pitfalls
• while (c \t c c \n )
• c getc(f)
• ACM Computing Surveys
• circa 1988 On Program Correctness

10
Open Source vs Closed Source

• SANS Critical. Vulnerability Analysis
• 7 Oct 02
• (1) HIGH Microsoft FrontPage Server Extensions
  (FPSE) Buffer Overflow
• (2) HIGH Microsoft Windows 2000/XP PPTP Buffer
  Overflow
• (3) MODERATE Opasoft network worm
• (4) MODERATE Fetchmail Buffer Overflow
• (5) MODERATE Winamp Skin File Buffer Overflow
• (6) MODERATE WN HTTP Server Buffer Overflow
• (7) MODERATE GV/GhostView Buffer Overflow

11
Buffer Overflow Problems as a Percentage of CERT
Alerts
12
C, an average programming language

• C is inherently unsafe programs may overflow
  buffers at will.
• No runtime checks that prevent writing past the
  end of a buffer.
• Reading or writing past the end of a buffer can
  cause a number of diverse behaviors
• Programs may act in strange ways
• Programs may fail completely
• Programs may proceed without any noticeable
  difference in execution.

13
Security versus Poor Programming

• Consider a boolean flag in main memory
• The flag determines whether the user running the
  program can access private files
• The overflow overwrites the boolean flag
• Illegal access to the files provided to attacker
• Stack Smashing
• Careless use of data buffers allocated on a
  programs runtime stack (i.e. local variables and
  function arguments)
• Attacker can usually run arbitrary code
• Place attack code somewhere (i.e.code to invoke a
  shell)
• Overwrite the stack in such a way that control
  gets passed to the attack code

14
Notes from the Cult of the Dead Cow

• To get this to happen, I fed a string of 0x80
  bytes into a popular conference package called
  'Microsoft Netmeeting' through the address field
  of a 'speeddial' shortcut.
• EIP happens to be 0x80808080.
• Guess what?
• That's good!
• I found a stack overflow!
• Now all I have to do is craft my exploit string
  to have some fun code inside, and tweak four of
  those 0x80 bytes to point to my exploit string.

http//www.cultdeadcow.com/cDc_files/cDc-351
Warning Foul language on this site
15
Buffer Overrun

• MS Visual C Output
• Overrun using the tab key

16
Attacking the Executable

• Overflowing Buffers
• Tab key
• Is there any security versus proprietary
  information to be protected in the executable?
• Disassembling Executables
• Win32 console application
• Generate complete assembly level source code from
  executable
• Decompiling
• Attempted with
• dcc (1 Australian PhD student)
• Turbo C decompiler
• Note a serious adversary could develop a custom
  decompiler

17
Reverse Engineering

• Machine code analysis
• Core Dumps
• Reverse Engineering Tools
• Debuggers usually have disassemblers
• takes machine code and translates into assembly
  language
• C code versus assembly
• loops versus counters and jumps
• Decompilers are not as mature as dissassemblers
• attempt to convert machine language into
  high-level language constructs
• JVM programs much easier to reconstruct than
  hand coded assembly language
• decompilation performance can be enhanced if the
  program is compiled with debugging options on
• Assume that binary code can be reconstructed

18
Copy Protection

• Tradeoff protection of intellectual property
  versus hassling legitimate users
• OPNET Example
• License Keys A psychological deterrent
• Encryption Keys
• use 36 character set less 1 l 0 O 32
  characters
• Use CBC and say Blowfish and produce valid keys
• Each key is a counter concatenated with a fixed
  binary string, encrypted and converted base 32
• Checking the license key for validity
• decode the base 32 string, decrypt the binary
  with the stored encryption key and to see that
  the last 12 bytes are equal to our stored binary
  string
• Force software to run off of distribution CDs
• Theoretically, no media is copy proof.

19
Code Obfuscation

• Anti-tampering
• Checksums
• Check for debuggers
• Running debuggers reset the instruction cache on
  every operation
• Check for this condition and jump your code to
  crash the program
• Obfuscation
• Rename all variables in code to arbitrary names
• Automated code obfuscation still an open research
  area
• JVM retains much more data than other HLLs
• Makes programs harder to maintain

20
Obfuscation Techniques

• Add code that never executes or that does nothing
• Make calculations more complex
• Move code around
• Spread related functions as far apart as possible
• Fake encapsulation
• Combine multiple unrelated functions into a
  single function
• Encode your code oddly
• Picking strings directly out of memory is easy
• Convert strings to odd character sets, only make
  strings printable when necessary
• Encrypt program parts
• Generally low grade because of performance
  considerations
• Data versus operation encryption
• Hex editor for manual encryption
• Encryption of padding

21
High Assurance Vulnerability Assessment

• Line-by-Line verification of source code
• Professional and/or contract decompilation of
  executables
• Complete review of published documentation
• Analysis of simulation runs to evaluate training,
  tactics and procedures
• Open source review of weapons and systems data
• Analysis of degree of parameterization

A Software Engineering Approach to VA
22
Med. Assurance Vulnerability Assessment

• Desk check of selected source code
• String search on disassembled code
• Professional and/or contract decompilation of
  executables
• Targeted review of published documentation
• Analysis of simulation runs to evaluate training,
  tactics and procedures
• Analysis of degree of parameterization

A Software Engineering Approach to VA
23
Low Assurance Vulnerability Assessment

• Desk check of selected source code
• String search on disassembled code
• Targeted review of published documentation
• Minimal analysis of simulation runs to evaluate
  training, tactics and procedures
• Analysis of degree of parameterization

A Software Engineering Approach to VA
24
Future Work

• Planned improvements
• Systematic tie between assembly code
  identification of parameters, documented
  parameters and open source database.
• Search for better debuggers and decompilers.
• Expand Simulation Software VA Process for use
  against actual BMC2 software
• Employ network simulation and DoD Architecture
  Framework products to analyze network
  vulnerabilities against distributed denial of
  service attacks.

25
Simulation Software Security Summary

• Best defense on buffer overflows is implicit
  bounds checking.
• Machine language executables cannot be considered
  inherently secure.
• Source code not required to compromise compiled
  software
• Executable software once released cannot be
  controlled.
• Training, tactics and procedures embedded in a
  compiled software simulation are vulnerable to
  compromise if released.
• Reverse engineering techniques have limitations
• Reverse engineering by resource unconstrained
  professional intelligence efforts can over time
  make significant discoveries.