On the Difficulty of Scalably Detecting Network Attacks

1
On the Difficulty of Scalably Detecting Network
Attacks
Kirill Levchenko with Ramamohan Paturi and George
Varghese
2
Detecting Attacks

• Which Intrusion Detection problems are hard
  (require per-flow state)?
• Counting distinct flows? Approximately?
• Detecting SYN Flooding? Port Scans?
• Are there efficient algorithms we dont know yet?
• How to tell?

3
Theoretical Techniques

• Abstract problem formulation
• Communication Complexity
• Source for NIDS problem reductions
• Establishes worst-case lower bounds
• Easy and fun to use

4
Outline

• Problem Motivation and Setting
• The Set Disjointness problem
• Application to SYN Flooding
• Implications
• Conclusion

5
Problem Setting
6
Problem Setting

• Protect more hosts
• Single device to administer
• More informed position

7
Problem Setting

• Challenges
• More hosts to protect
• Higher bandwidth links
• How do resource requirements scale?
• Are there fundamental limits?

8
Problem Setting
Outbound
Inbound

• Two packet sequences traffic entering and
  leaving the network
• Goal detect attack using packet sequences

9
Algorithms Reductions

• Formalize the algorithmic problem
• An algorithm on packet sequences
• Reduce one problem to another
• Show solving one problem solves the other

10
Our Reduction
Set Disjointness ? NIDS Problem

• Solving the NIDS problem solves the Set
  Disjointness problem
• Use strong bounds on Set Disjointness
• Establishes space lower bound for the NIDS problem

11
Set Disjointness
Alice
Bob
12
Set Disjointness
Communication
Alice
Bob
(even randomized)
13
Lets Try It!(on SYN Flooding)
14
SYN Flooding
Normal
Attack
Goal detect unclosed connections.
15
SYN Flooding

• Creates packet sequence corresponding to her set
  with SYN flag
• Runs NIDS algorithm on input sequence
• Suspends it after reading the last item
• Sends the state of the algorithm to Bob

16
SYN Flooding
Yes/No

• Creates packet sequence corresponding to elements
  not in his set with FIN flag
• Resumes NIDS algorithm on input sequence (using
  state sent by Alice)
• Result indicates if sets intersect or not

17
SYN Flooding
Algorithm sees

• If A and B intersect, there is a SYN packet not
  followed by a FIN
• If A and B are disjoint, every SYN packet has a
  matching FIN

18
SYN Flooding

• Alice sent Bob the state of the NIDS algorithm
• Set Disjointness requires O(n) bits
• The state of the algorithm is O(n) bits

no. of flows
19
Other NIDS Problems

• Port Scans
• per-host state required
• TCP connection hijacking
• per-flow state required
• Evasion by fragmentation
• re-assembly required

20
Implications

• Sometimes per-flow state is required, but
• Examples are artificial (may not occur in
  practice)
• Problem semantics may be a great help
• System can fail gracefully when out of memory
• Additional information may be available!

21
Implications

• Additional information can be useful
• Count outgoing SYNACK and FIN packets
• Works if protected network can be trusted
• Hop-count filtering or other fingerprinting
• Small UDP fragments are unusual

22
Conclusion

• Set Disjointness is a useful reduction source
• Even hard under randomization
• Permits relaxations (see paper)
• Exposes and formalizes the hardness of some NIDS
  problems
• Guides practical algorithm design to consider
  hard cases and failure modes