Operating System Protection Through Program Evolution

1
Operating System Protection Through Program
Evolution

• By Dr. Frederick B. Cohen
• Presented by William Lu

2
The Ultimate Attack

• How to defeat defensive measures of a system?

• Gain physical access to the system
• Reverse engineer defenses
• Find weak link and exploit

3
The Ultimate Defense

• How to defend against attackers

• Make attacks extremely complex
• Make costs too high to be worth attacking
• i.e. passwords
• Large space
• Spread out probability density (diffusion)
• Obscuring stored information (confusion)

4
The Ultimate Defense

• Current operating systems
• Space is enormous (all programs that fit in
  memory)
• High probability subspace (very small number of
  versions)
• No confusion

5
The Ultimate Defense

• How to increase operating system defenses?

• Reducing coherence

• How?

• Unique defense for each system

• Feasibility?

• Too many unique defenses to design

• Compromise?

• Implement a fixed number of defenses

6
The Ultimate Defense

• More practical solution?

• Evolutionary defenses

• Goal?

• Produce a large search space
• Provide confusion
• Provide diffusion

7
(some) Techniques for Program Evolution

• Equivalent instruction sequences
• Instruction reordering
• Variable substitution

8
Equivalent Instruction Sequences

• What does it do?

• Replaces instruction sequences with equivalent
  sequences
• i.e. add 17 is equivalent to add 20 and subtract 3

• How does it help defend against attacks?

• Potentially infinite evolutions
• Creates enormous possible executions

9
Instruction Reordering

• What does it do?

• Reorders instructions without altering program
  execution
• Order does not matter

• How does it help defend against attacks?

• Increases complexity of attacks to n! different
  orderings (n of different instructions)

10
Instruction Reordering

• 3 different instructions
• 6 different forms

I3 I3 J5 J5 K8 K8
J5 K8 I3 K8 I3 J5
K8 J5 K8 I3 J5 I3

11
Variable Substitution

• What does it do?

• Alters the location of memory storage areas

• How does it help defend against attacks?

• Prevents static examination and analysis of
  parameters

12

• What to do with these
• (and other) techniques?

13
Providing Evolution in Defenses

• How to evolve?

• Select a mix of evolution techniques
• Increase complexity while minimizing impact on
  end users

14
Providing Evolution in Defenses

• When to evolve?

• At the factory?

• Uniquely identify each disk sent out
• Reduced efficiency

• At installation?

• Crucial to have unique and confidential evolution
• Takes time at end user level

15
Providing Evolution in Defenses

• When to evolve?
• After installation?

• Cannot assure against corruption
• Cannot trust internal checking
• Attack that succeeds on one day may fail the next

16
Attacks on Program Evolution

• Points of Attack
• Tracing Attack

17
Point of Attack

• How?

• Find original entry point and exploit it
• Gain direct access to hardware or operating
  system internals

• How to defend against it?

• Evolve the core of the operating system
• Evolving the calling mechanism
• Calls that bypass protection may be of the wrong
  form

18
Tracing Attacks

• How?

• Trace programs at execution or simulation

• How to defend against it?

• Use redundancy
• Force attacker to use tracing on each attack

19
Conclusion

• Program evolution can increase the complexity for
  an attacker
• Create a large search space to make attacks
  infeasible
• Need more study to reach maturity