Detecting Backdoors

1
Detecting Backdoors

• Yin Zhang, Vern Paxson
• 2000 USENIX Security Symposium

2
Various tradeoffs for developing

• Detecting backdoors based on interactive traffic
• Several features for backdoor detection
• the size and transmission rate of packets
• timing structure
• Using passive monitoring
• Using timing-based approach

3
Filtering

• Important factor for the success of real-time
  backdoor detection
• Tradeoff between reduced system load and lost
  information
• Filtering makes it difficult to determine whether
  an attack or not.
• The attackers can abuse filtering criteria.
• Three possible filtering criteria
• Packet size
• Directionality
• Packet contents

4
Accuracy Responsivness

• The problem of false positive and negative
• It is desirable to detect backdoors as quickly as
  possible, but waiting longer allows the monitor
  to detect more accurately.
• Tradeoff of responsiveness versus accuracy

5
General Algorithms for Detecting Backdoors

• Incorporating three types of characteristics
• Directionality
• Packet size
• - using 20 bytes as cutoff for small packets
• - using the ratio of the number of small
  packets over the total number of packets to
  determine whether interactive traffic or not
• Packet interarrival time
• Making the algorithm run in real-time
• Filtering out all packets with more than 20 bytes
  of payload

6
Special-Purpose Detection Algorithm

• Finding servers for those protocols running on
  ports other than their standard ports is the
  evidence of a backdoor.
• Comparison with the general-purpose algorithm
• More accurate and efficient
• Making the algorithm susceptible to evasion
• Two classes
• The algorithms to identify backdoors as
  accurately as possible, without worrying about
  efficiency
• The algorithms to incorporate filtering
  mechanisms into the optimal algorithms

7
SSH

• ssh-sig
• Using the SSH version string as the signature for
  SSH
• SSH-1., SSH-2.
• ssh-len
• The most packets for SSH have either length 8k4
  or 8k.
• If 75 of all packets have more than either
  length 8k4 or 8k, classify the connection as SSH.

8
Telnet

• Most Telnet sessions begin with a series of
  option negotiations. Those are one of the
  following four 3-byte formats.
• telnet-sig
• If a connection involves any option negotiation,
  we classify it as a Telnet connection.

9
Performance

• SSH algorithms
• False negative ratio
• - running ssh-sig on trace ssh.trace
• - False negative ratio was extremely low.
• False positive ratio
• - running on lbnl.mix1.trace, lbnl.mix2.trace
• - false positive ratio was nearly 0.
• telnet algorithms
• False negative ratio
• - 18 among 1526 Telnet connections are missed
  by telnet-sig. 17 out of 18 was the types that
  didnt involved any option negotiation.
• False positive ratio
• - none of the 12708 non-Telnet connections is
  mis-classified as Telnet connection.

10
Performance(Contd)

• General detection algorithm
• False negative ratio
• - 22 among all 1450 Telnet connections are
  missed by the timing-based algorithm.
• - The algorithm detects less accurately than
  protocol-specific algorithm.
• False positive ratio
• - 57 backdoors among 12000 connections
• - 45 are IMAP and POP mail servers used
  interactively, so are not in fact false positives.