Formal reasoning about detection rules of specificationbased IDS

1
Formal reasoning about detection rules of
specification-based IDS

• Tao Song, Jim Alves-Foss, Karl Levitt

2
Motivation

• A formal framework to analyze the detection rules
  of Intrusion Detection Systems
• To verify security policies with detection rules
• To analyze and improve detection rules

3
Block Diagram of the Approach
Security Requirement Policy / attacks (SR)
Assumption (H)
Auditing Model (L)
Abstract System Model (S)
SLHR ? SR
4
Abstract System Model

• Security-critical components
• Processes,Users,Files,Environment variables
• Discretionary Access Control(DAC)
• Subject users
• Object files and processes
• Setuid programs/ privileged programs

5
Abstract System Model Cont.

• system (proglist calllist filelist userlist
  envlist)
• proglist((pname pdir)...)
• calllist((callname)...)
• filelist((path ouid ogid pmode inodeid)...)
• pmode ((r w x)(r w x)(r w x)(dir reg socket
  pipe))
• userlist((uid uname gid homedir)...)
• envlist((envname envvalue)...)

6
Auditing Model

• Logging mechanisms are required for the security
  of systems
• Auditing mechanisms are used in most IDS
• A subset of system calls will be audited

7
Auditing Model cont.

• log record (procobj fileobj syscall newprop)
• Procobj (prog ruid pid euid egid)
• Fileobj(name ouid ogid pmode nodeid)
• Syscall(syscall flags)
• Newprop (newowner, newmode, newpath, chpid)
• pmode ((r w x)(r w x)(r w x)(dir reg socket
  pipe))

8
Assumptions

• System assumption
• System kernel is not vulnerable to attack
• DAC of the system is correctly implemented
• Integrity of log data and IDS
• Behavior of root
• Verification assumption
• Properties of trusted files
• Values of environment variables

9
Security Requirement

• Security policy
• An example Trusted file access policy
• The Trusted file can only be access by specific
  user or specific programs
• (trusted file, user, program, access)
• For example
• (/etc/passwd, root , passwd, (open-wr,create,
  chmod, chown, rename))

10
Security Requirement cont.

• Formalization of the Policy
• (defun access-passwd (logrec)
• (if (and (equal '(/ etc passwd) (getfilename
  (logrec-fobj logrec)) )
• (or (equal (getprocruid logrec) 0)
• (equal 'passwd (getprogname (logrec-fobj
  logrec)) ) )
• (or (equal 'open-wr (getcallname logrec))
• (equal 'chmod (getcallname logrec))
• (equal 'chown (getcallname logrec))
• (equal 'rename (getcallname logrec))
• (equal 'delete (getcallname logrec)) )
• ) )
• t nil
• ) )

11
Verification

• IDS satisfies some security policies
• Detection rulesassumption implies a security
  policy
• Detection of unknown attacks
• Violation of security policy imply violation of
  detection rules
• Improvement of security
• Detection rules protect assumptions from breaking

12
Intrusion Detection Systems(IDS)

• Anomaly detection systems
• The characteristics of attacks are significantly
  different form common behavior.
• Misuse detection systems
• Match against signatures of known attacks
• Specification-based IDS
• Specification of security-critical entities
• System Health and Intrusion Monitoring (SHIM)

13
Formal methods

• Proof-based verification
• Finds a set of intermediate formulas that allow
  verifier to reach conclusion starting from
  premise
• Theorem proverACL2

14
Verification of SHIM

• SHIM Spec-based IDS
• Certain privileged programs may be subject to
  attack, resulting in unauthorized behavior
• SHIM monitors audited behavior of privileged
  programs and raises a flag is the process make an
  unauthorized system call or accesses an
  unspecified file. This effectively constrains the
  behavior of the privileged programs.

15
Verification of SHIM cont.
Audit data
Audit Filter
Audit data of one process
Spec of the process
16
Spec of privileged programs

• Spec of ftp
• (ltvalidopgt-gt (OPEN_RD, WorldReadable(F.mode))
• (OPEN_RD, F.ouid S.uid)
• (UNLINK, CreatedByProc(P.pid, F))
• (CHMOD, CreatedByProc(P.pid, F))
• (CHOWN, CreatedByProc(P.pid, F))
• (forkvfork)
• (OPEN_RD, InDir(F.path, getHomeDir(S.uid)))
• )

17
Spec of privileged programs cont.

• Formalization of ftpd spec
• (defun spec_ftpd_rec (sys logrec filelist)
• (or (and (operate 'openrd logrec)
  (WorldReadable (getfile logrec)))
• (and (operate 'openrd logrec) (OwnerofFile
  logrec))
• (and (operate 'unlink logrec) (CreatedByProc
  (getfile logrec) filelist))
• (and (operate 'chmod logrec) (CreatedByProc
  (getfile logrec) filelist))
• (and (operate 'chown logrec) (CreatedByProc
  (getfile logrec) filelist))
• (or (operate fork logrec) (operate vfork
  logrec))
• )

18
An Example of Verification

• The normal behavior of ftpd will not violate
  trusted file policy
• (defthm passwd-ftp
• (implies
• (not (member '(/ etc passwd) created))
• (implies
• (and (logp log)(sys-p sys)
• (passwdsafe log)
• (homedirsafe sys)
• (validuser sys log)
• (spec_ftpd sys log created))
• (not (access-passwd log) )
• ) )
• )

19
An Example of Verification

• Concurrent execution of different programs will
  not violate trusted file policy
• (defthm passwd-specs
• (implies
• (and (logp log) (sysp sys) (procsafe log)
• (passwdsafe log) (homedirsafe sys) (validuser
  sys log)
• (validenv sys 'printerspool).//other
  assumptions
• (spec_ftpd sys (filter 'ftpd log) created)
• (spec_lpr sys (filter 'lpr log) created))
• (spec_spec_crontab sys (filter 'crontab log)
  created))
• //other specs
• (access-passwd log))
• ) )

20
Future work

• Using SHIM to enforce Clark-Wilson policy
• IDSIVP(Integrity Verification Procedure)
• Privileged programTP (Transformation Procedure)
• FileCDI(Constrained Data Item)
• SpecAccess Triple(User, TP, CDI)
• (defun accessrules( user process file)
  (spec_process user file))
• Certification rules and enforcement rules

21
Future work cont.

• Verification of network-based IDS
• Formalized network-related entities
• Detection rules of network-based IDS
• Security invariants of protocol
• Verification between security invariants and
  detection rules

22
Conclusion

• A formal framework to reason detection rules of
  IDS
• Methods to formalize detection rules and security
  policis
• Formalization and analysis of specs of SHIM
• Verification of a trusted policy

23
Thanks