Software Security Monitors: Theory

1
Software Security MonitorsTheory Practice

• David Walker
• Princeton University
• (joint work with Lujo Bauer and Jay Ligatti)

2
General-purpose Security Monitors

• A security monitor (program monitor) is a process
  that runs in parallel with an untrusted
  application
• monitors examine application actions
• decide to allow/disallow application actions
• may terminate an application, log application
  actions, etc.
• monitors detect, prevent, and recover from
  erroneous or malicious behavior at run time
• monitors generalize specific enforcement
  mechanisms such as access control lists, etc.

3
What is a security monitor?

• Monitors analyze transform untrusted
  application actions

Monitor
Input Stream
Output Stream
a3
a1
a2
a2
a4
a2
a2
a1


Application generates actions to be input into
monitor
Machine executes actions output by monitor
4
Possible Monitor Actions

• Accept the action
• Halt the application
• Suppress (skip) the operation
• Insert some computation
• Also replace results raise exceptions

5
Formalizing security monitors

• Security monitors gt formal automata that
  transform a stream of program actions
• Given a set of possible program actions A
• Monitors are deterministic state machines (Q,
  q0, T) where
• Q state set
• q0 start state
• T transition function

6
Operational Semantics

• Single step (determined by T)
• (Sin, q) ? (Sin, q)
• Multi-step (reflexive, transitive closure of T)
• (Sin, q) ? (Sin, q)
• Output sequence is observable
• Input sequences are not observable

So
So
7
A Hierarchy of Security Monitors
We classify monitors based on their
transformational abilities (ie based on T).
Insert Suppress OK
Halt Truncation Suppression Insertion Edit
?
?
?
?
?
?
?
?
?
?
?
?
8
An Example E-Banana.com

• Set of application actions A take(n),
  // take n bananas pay(n), // pay for n
  bananas browse, // browse for bananas
  receipt // commit
• Edit Automaton

take(n)
pay(n)
pn
browse
browse
pay(n)
take(n)
start
tpn
init
tn
receipt
pay(n)take(n)receipt
9
Edit Automata

• Definition (Q,q0,T)
• where T (t,e,i)
• State transition function t
• t action x state ? state
• Emission function e
• e action x state ? ,-
• Insertion function i
• i action x state ? action sequence x state

10
Edit Automata

• Operational Semantics
• (S, q) ? (S, q)if SaS and t(a,q)q and
  e(a,q)
• (S, q) ? (S, q)if SaS and t(a,q)q and
  e(a,q) -
• (S, q) ? (S, q)if SaS and i(a,q)(Sins, q)
• (S, q) ? (empty, q)otherwise

a
(E-Accept)
(E-Suppress)
Sins
(E-Insert)
(E-Halt)
11
Security Policies

• A program execution is a sequence of actions
• A Security Property is a predicate over
  executions.
• Example Properties
• P(S) iff bananas taken equal bananas paid for in
  S
• Access control, resource bounds policies are
  policies
• Non-properties
• Relations between different executions of a
  program
• Information-flow policies

12
What does it mean to enforce a policy?

• Principle of Soundness
• All observable outputs obey the policy ?
  sequences Sin . ? state q . ? sequence So
• 1. (Sin, q0) ? (empty, q)
• 2. P(So)
• Principle of Transparency
• Semantics of executions that already obey policy
  must be preserved 3. P(Sin)?? (Sin ??So)

So
13
Some Useful Equivalences

• Remove/Insert unnecessary actions
• fclose(f)fclose(f)?? fclose(f)
• Replace a sequence with equivalent actions
• socket(S)send(S,m)?? socketSend(S,m)
• Permute independent actions
• fopen(f)fopen(g)?? fopen(g)fopen(f)
• Necessary properties
• reflexive, symmetic transitive
• S?? S ?? P(S)?? P(S)

14
E-Banana.com

• Equivalence Rules

1) (browse S) ? S 2) (S1 take(n) pay(n)
S2) ? (S1 pay(n) take(n) S2)
15
Conservative Enforcement

• Enforcer satisfies Soundness but not necessarily
  Transparency
• ? properties P . (? sequence S . P(S)) ? P can
  be conservatively enforced

Conservative
16
Effective Enforcement

• Enforcer satisfies Soundness and Transparency
• provides some flexibility for the enforcer to
  edit the execution sequence
• guarantees the final results of running the
  application with the monitor are semantically
  equivalent to running the application without the
  monitor

Effective
Conservative
17
Precise Enforcement

• Definition
• Enforcer satisfies Soundness and Transparency
• Enforcer must output actions in lock-step with
  application
• Motivation
• In some scenarios, operations cannot be delayed
  without disrupting application semantics

Precise
Effective
Conservative
18
What properties can be enforced?

• The enforceable properties depend upon
• the definition of enforcement (conservative,
  effective, precise)
• the class of automaton (truncation, suppression,
  insertion, edit)
• the space of possible input programs
• if the monitor can assume certain bad
  executions do not occur, it can enforce more
  properties
• static program analysis (type systems
  proof-carrying code) can constrain program
  execution in ways useful to run-time monitors

19
Effective Enforcement

• An E-Banana.com policy
• Our edit automaton is an effective enforcer
• It satisfies Soundness
• It satisfies Transparency
• Proofs are by induction over the possible inputs
• Less powerful automata (truncation, suppression
  and insertion) cannot enforce the E-Banana
  property
• Proof by contradiction shows either Soundness or
  Transparency will be violated

browse ((take(n)pay(n) pay(n)take(n))
receipt)
20
A Simple Theorem

• Theorem Any decideable predicate P on
  executions is a property that can be effectively
  enforced by some edit automaton
• Proof construct a transactional edit automaton
  that suppresses and logs program actions when
  P(S) and commits (outputs) when P(S), for every
  initial sequence of actions S in a program
  execution

21
Effectively Enforceable Properties
Editing Properties
Insertion Properties
Suppression Properties
Trunc. Prop.
22
Summary of theoretical results

• We have developed the following rigorous
  methodology for reasoning about run-time
  security
• Define the computational framework using formal
  operational semantics
• Define what it means to enforce a policy
• Prove results about enforceable policies
  mechanisms from definitions 1 2

23
Future Work/Research Ideas

• Proper definitions of enforcement for infinite
  execution sequences
• Understanding edit automata on infinite sequences
• Understand transactional policies develop
  transaction automata
• what can they enforce?
• Incorporate more practical elements into the
  model
• security environment cryptographic secrets
• replacement of results, exceptions and program
  state

24
Polymer, the Language

• Polymer
• A domain-specific language for programming
  security monitors (ie edit automata)
• Java a couple of simple extensions
• atomic policy definitions encapsulating
• a set of security-relevant actions
• security state
• decision procedure that produces security
  suggestions (halt, suppress action, insert
  action, etc)
• compositional policy definitions involving
• higher-order policy combinators

25
Securing Untrusted Applications
untrusted code
describes security-relevant program points
Java application
policy interface
instrumented application
separately compiled from policy
contains hooks to call monitor
26
Securing Untrusted Applications
Java application
implements dynamic security policy
policy interface
policy implementation
instrumented application
combines application and policy
secure application
27
Atomic Polymer Policy
new policy definition extends policy class
class limitFiles extends Policy private int
openFiles 0 private int maxOpen 0
limitFiles(int max) maxOpen max
....
private policy state
policy constructor
28
Atomic Polymer Policy Continued
class limitFiles extends Policy private int
openFiles ... private int maxOpen ...
public ActionPattern actions new
ActionPattern ltFile
fileOpen(String)gt, ltvoid fileClose(File)gt
....
set of policy- relevant methods
29
Atomic Polymer Policy Continued
class limitFiles extends Policy private int
openFiles ... private int maxOpen ...
public ActionPattern actions ...
Suggestion before(Action a) aswitch (a)
case fileOpen(String s) if
(openFiles lt maxOpen) return
Suggestion.OK() else
return Suggestion.Halt() case
fileClose(File f) ...
policy behavior
30
Atomic Polymer Policy Continued
class limitFiles extends Policy private int
openFiles ... private int maxOpen ...
public ActionPattern actions ...
Suggestion before(Action a) aswitch (a)
case fileOpen(String s) if
(openFiles lt maxOpen) return
Suggestion.OK() else
return Suggestion.Halt() case
fileClose(File f) ...
31
Atomic Polymer Policy Continued
class limitFiles extends Policy public
ActionPattern actions ... private int
openFiles ... private int maxOpen ...
Suggestion before(Action a) aswitch (a)
case fileOpen(String s) if
(openFiles lt maxOpen) return
Suggestion.OK() else
return Suggestion.Halt() case
fileClose(File f) ...
32
Complex Monitors

• Combine atomic policies defined over a variety of
  different resources
• eg sample applet policy
• file system access control
• number of files opened
• restricted network access
• no network access after local file is read
• communication with applet source only

33
Policy Combinators

• Programmers may write parameterized policy
  combinators
• And, Or, Forall, Exists, Chinese wall,...

P1
P2
AndPolicy
?
s2
s1
s
34
Policy Combinators

• class AndPolicy extends Policy
• private Policy p1
• private Policy p2
• AndPolicy(Policy pol1, Policy pol2)
• p1 pol1
• p2 pol2
• ...

first-class policies
35
Policy Combinators

• class AndPolicy extends Policy
• ...
• Suggestion before(Action a)
• Suggestion s1 p1.before(a)
• Suggestion s2 p2.before(a)
• if (s1.isOK() s2.isOK())
• return Suggestion.OK()
• else ...

using suggestions
In reality, writing combinators is very tricky
36
Summary of Language Design

• Polymer facilitates the implementation of program
  monitors by
• encapsulating all elements (relevant actions,
  state, decision procedure) of atomic policies in
  a single place
• providing mechanisms to compose policies in a
  well-defined manner
• coming equipped with a formal semantics
• were working on it

37
Conclusions

• Technology for securing extensible systems is in
  high demand
• Software security monitors are one part of the
  solution
• For more information, see
• Edit Automata Enforcement Mechanisms for
  Run-time Security Policies. IJIS 2003.
• Types and effects for non-interfering program
  monitors.  ISSS 2002 LNCS 2609.
• More Enforceable Security Policies. FCS 2002.
• www.cs.princeton.edu/sip/projects/polymer/

38
End
39
Realistic Monitors

• Protect complex system interfaces
• interfaces replicate functionality in many
  different places
• method parameters communicate information in
  different forms
• eg Java file system interface
• 9 different methods to open files
• 4 different methods to close files
• filename strings, file objects, self used to
  identify files

40
Abstract Action Definitions
java.lang.io
FileReader(String fileName) FileReader(File
file) RandomAccessFile(...) ... FileReader.clos
e() RandomAccessFile.close() ...
fileOpen(String n) fileClose()
41
Abstract Action Definitions
class fileOpen extends ActionSig boolean
canMatch(Action a) aswitch (a)
case FileReader(_) return true case
RandomAccessFile () return true ...
String parameter1(Action a) ....

42
Abstract Action Pattern Matching
class limitFiles extends Policy ...
Suggestion step(Action a) aswitch (a)
case fileOpen(String s) ...
case fileClose() ...
fileOpen.parameter1(a)
fileOpen.canMatch(a)
43
Taxonomy of Precisely Enforceable Properties
44
Secure Application
Untrusted application
Host System (Java)
Program Monitor Definition
Polymer language extensions
Java core
45
Policy Architecture Simple Policies
system interface
Simple Policy Def.
Host System (Java)
Polymer language extensions
Java core
46
Policy Architecture Abstract Actions
abstract system interface
Host System (Java)
Simple Policy Def.
Abstract Action Def.
Polymer language extensions
concrete system interface
Java core
47
Policy ArchitectureComplex Policies
Complex, System-specific Policy
abstract system interface
Simple Policy Def.
Policy Comb. Def.
Abstract Action Def.
Host System (Java)
Polymer language extensions
concrete system interface
Java core
48
Securing Extensible Systems

• Many questions
• Our application requires property X. Can we
  enforce it precisely or will we have to get by
  with an approximation?
• How do we write down our policy succinctly and
  unambiguously?
• What specific mechanism will we need to enforce
  our policy?
• How do we implement the mechanism?

49
Summary

• A general framework for formal reasoning about
  security monitors
• defined a hierarchy of security monitors
• gave meaning to the word enforceable
• developed rigorous proofs concerning enforceable
  properties
• Polymer A programming language for composing
  security monitors
• techniques for modular monitor design
  composition
• formal semantics as an extension of FeatherWeight
  Java