NETWORK TRAFFIC ANALYSIS OPTIMIZATION FOR SIGNATURE-BASED INTRUSION DETECTION SYSTEMS

1
NETWORK TRAFFIC ANALYSIS OPTIMIZATION FOR
SIGNATURE-BASED INTRUSION DETECTION SYSTEMS

• Dmitry S. Kazachkin
• student, Computational systems lab at CMC MSU
• Dennis Y. Gamayunov
• scientific advisor, PhD, Computational systems
  lab at CMC MSU

2
INTRODUCTION

• High-performance traffic analysis at gigabit
  channels
• Optimization of a given signature set matching
• Packet the unit of analysis
• Operations number minimization for a single
  packet

3
SIGNATURE FORMAL DEFINITION (1)

• Packet pair H, P
• Header (H) H1,H2,...,Hn, Hi belongs to finite
  space
• Payload (P) text line of unrestricted length
• State vector (C) ?1,?2,...,?k, Ci belongs to
  finite space

4
SIGNATURE FORMAL DEFINITION (2)

• Header condition logical predicate Hi
  arguments
• Payload condition set of functions
• CondPi(P,C) logical predicate payload and
  state vector are arguments
• Effectsi Effecti,j(P,C) represents side
  effect on variable ?i, performed on CondPi(P,C)
  success
• Reaction element of event classes finite set

5
SIGNATURE FORMAL DEFINITION (3)

• Signature triplet ltSH,SP,Rgt, where
• SH set of header conditions
• SP ordered set of payload conditions
• R reaction.
• RES(ltSH,SP,Rgt,H,P,C)
• R, if all the header conditions return true
  and all the payload conditions successfully
  evaluate consecutive.
• Æ, else

6
SNORT SIGNATURESSTRUCTURE

• Action action performed on rule activation
  (usually, alert)
• Header context-independent conditions on packet
  header
• Options context-dependent conditions on packet
  payload
• Info rule info and message, generated at rule
  activation

7
SNORT SIGNATURES FORMALIZATION (1)

• Header
• H1 protocol type (TCP / UDP / ICMP / other IP)
• H2 and H3 IP and port of packet source
• H4 packet direction (from server/to server)
• H5 and H6 IP and port of packet destination
• Payload packet payload byte string
• State vector C C1, C1 position of text
  marker

8
SNORT SIGNATURES FORMALIZATION (2)

• CondHk(H) logical expression, operating Hi
  only
• ltCondPk(P,C), Effectk(P,C)gt pair of functions
  representing pattern matching performed after
  marker
• Reaction finite set of identifiers, which
  represent action and info sections

9
R-LANG LANGUAGEFORMAL SIDE

• Automaton - (K, PS, T, PT, s0, I, g q)
• S states set
• PS state predicates set
• T transitions set
• PT transition conditions set
• s0 initial state
• I automaton exemplars set
• g global environment
• q global timer queue
• automaton exemplar series of slices NK x L x Q
  x E
• NK current state
• L local environment
• Q local timer queue
• E local events queue

10
R-LANG LANGUAGEDECLARATIVE SIDE

• Scenario
• Events subscription
• Variables declaration scenario fields
• Functions declaration scenario methods
• States declaration
• One is marked as initial
• Transitions declaration
• Transition type consuming/nonconsuming/unwinding
  
• Condition logical function
• Body code block

11
SIGNATURE CONVERSIONTO R-LANG (1)

• S.T. Eckmann, "Translating Snort rules to STATL
  scenarios
• Typified events on every packet
• SrcIP, SrcPort, Direction, DstIP, DstPort
• Payload
• Scenario field variable Current
• Header conditions logical expressions on Hi
• Payload conditions boolean function calls, with
  side-effect on Current
• Reactions alert sending code blocks

12
SIGNATURE CONVERSIONTO R-LANG (2)

• scenario sc(ltevent corresponding to packet typegt
  ev)?
• ltSecondary variables definitiongt
• initial state st0
• consuming transition st0-gtst0
• event ltevent corresponding to packet typegt (
• ltHeader condition 1gt ltHeader condition
  Ngt
• )
• ltRestoring state vectorgt
• if(ltPayload condition 1gt)?
• if(ltPayload condition Mgt)?
• ltReactiongt

13
HEADER-BASED OPTIMIZATION (1)

• Conditions alternative a set of pairs ltSPi,Rigt,
  where
• SPi ordered set of payload conditions
• Ri reaction
• Alternative-containing signature a pair ltSH,
  SAgt, where
• SH set header conditions
• SA condition alternative.
• Set of signatures with the same header are
  converted to alternative-containing signature
  this way
• Alternative-containing signature evaluation
  result is defined as

14
HEADER-BASED OPTIMIZATION (2)

• R-lang transition body of the alternative-containi
  ng signature model
• ltRestoring state vectorgt
• if(ltSeries of payload conditions 1gt)? ltReaction
  1gt
• ltRestoring state vectorgt
• if(ltSeries of payload conditions 2gt)? ltReaction
  2gt
• ltRestoring state vectorgt
• if(ltSeries of payload conditions Ngt)? ltReaction
  Ngt
• We get the same RES set with less operation by
  grouping signatures with the same header
  conditions set.
• 519 different header condition sets for Snort
  base of 6372 signatures grants a good performance
  speed-up

15
HEADER-BASED OPTIMIZATION (3)

• Tree-style order provides even more economy.
• consuming transition st0-gtst0
• event ltevent corresponding to packet typegt(true)
• if(ltHeader condition 1gt)
• if(ltHeader condition 2gt)
• ltpredicate alternative for header
  granted by 1,2gt
• if(ltHeader condition 3gt)
• if(ltHeader condition 4gt)
• ltpredicate alternative for header
  granted by 1,3,4gt
• Snort analysis engine use that header
  optimization only, that does not allow achieving
  further speed-up on a fixed signature set.

16
PREDICATE TREE

• Fact Condition chains contained in conditions
  alternative can have the same beginnings.
• Predicate tree ST tree
• the edges are marked with a payload condition
• the nodes are marked with a reaction set,
  possibly empty
• the leafs are marked with non-empty reaction sets

17
PREDICATE TREE EVALUATION

• Predicate tree evaluation
• recursive traversal of tree from root node
• conditions control sub-trees evaluation
• reaction performing if current node is marked
• context restoration after sub-tree evaluation
• RES(ST, P,C) all the reaction achieved
• Statement Let predicate tree ST is converted
  from condition alternative SA. Then
• RES(ST,P,C)RES(SA,P,C).

18
ADDITIONAL OPTIMIZATIONS

• Synonymic bunch set of leafs
• the same R
• the same edges parent
• PCRE union
• Aho-Corasick algorithm

19
STATIC RESULT ANALYSIS

• Tree profit edges number in the tree - total
  number of payload conditions
• Relational tree profit tree profit / total
  number of payload conditions
• For 2 signature groups (Samba)
• 2000 signatures
• tree profit gt 11000
• relational tree profit gt 80
• Overall relational tree profit 62.
• 40 pairs of identical signatures

20
DYNAMIC RESULT ANALYSIS

• N seconds of flood with legal packets, a bad
  packet after them
• Test 1 legal traffic is ignored by most rules
• FTP HELP command
• Test 2 legal traffic is checked by lots of
  rules
• specially constructed HTTP request

Seconds of alert delay Seconds of alert delay
Before Optimization After optimization
Test 1, 5 seconds 123 10
Test 1, 20 seconds 391 24
Test 2, 5 seconds 103 37
Test 2, 20 seconds 203 133
21
CONCLUSION

• Proposed methods provides significant speed-up of
  packet analysis at experimental IDS.
• Though speed-up is significant, it is still not
  enough. There is a need for other optimizations,
  e.g. parallel signature matching.

22
REFERENCES

• Snort IDS, http//www.snort.org
• S.T. Eckmann, "Translating Snort rules to STATL
  scenarios" presented at the 4th International
  Symposium on Recent Advances in Intrusion
  Detection (RAID 2001), Davis, CA, October 2001,
  LNCS 2212, pp. 69-84.
• M. Norton, "Optimizing Pattern Matching for
  Intrusion Detection," white paper, Sourcefire
  Inc., 2004 Online Avaliable http//
  docs.idsresearch.org/OptimizingPatternMatching/For
  IDS.pdf.
• D.U. Gamayunov, Network objects behavior
  analysis for detecting computer attacks PhD
  thesis, Faculty of Computational Math and
  Cybernetics, Moscow State University, Moscow,
  2007.
• S.T. Eckmann, G. Vigna, and R. A. Kemmerer.
  STATL An Attack Language for State-based
  Intrusion Detection Dept. of Computer Science,
  University of California, Santa Barbara, 2000.
• G. Vigna, R. Kemmerer, "NetSTAT A Network-based
  Intrusion Detection Approach." Proceedings of the
  14th Annual Computer Security Application
  Conference, Scottsdale, Arizona, December 1998.
• M. Roesch. "Writing Snort Rules How To write
  Snort rules and keep your sanity" Online
  Avaliable http//www.snort.org.

23
Thank you for attention