Best Known Methods in Security Events Correlation

1
Best Known Methods in Security Events Correlation

• Mohammed Fadzil Haron
• GSEC GCIA
• April 12, 2005

2
Agenda

• Correlation overview
• Knowledge requirements
• Methodology
• Data representation
• Reaction

3
Correlation defined

• A relation existing between phenomena or things
  or between mathematical or statistical variables
  which tend to vary, be associated, or occur
  together in a way not expected on the basis of
  chance alone1
• 1 http//www.webster.com

4
Overview

• Correlation is the next security big thing in
  importance
• An important tool in the security analysts
  toolbox for monitoring security events
• To be most effective, most if not all events
  should be examined
• Defense in depth means more data from different
  technologies, vendors, and products
• Huge amount of data to analyze terabytes in size
  and growing
• Reduce false-positive and false-negative findings
  compared to use of a single product/technology
• Expensive manned 24x7 monitoring capabilities

5
Ultimate goal

• Et Dt Rt
• Exposure time (Et) The time the resource,
  information, or organization is susceptible to
  attack or compromise.
• Detection time (Dt) The time it takes for the
  vulnerability or the threat to be detected.
• Reaction time (Rt) The time it takes for the
  individual, group, or organization to respond and
  eliminate or mediate the vulnerability or
  risk.
• Time Based Security by Winn Schwartau

6
Security events flow
7
Axiom on correlation

• You only see the tip of the iceberg
• Know the environment and perimeter of defense
  well
• Dont trust the tool trust your judgment
• Automate whenever possible 1
• Use the simplest data representation possible
• Balance between over-correlated and
  under-correlated
• Get the big picture
• The truth is in the packet 1
• 1 Toby Kohlenberg, Intel Corp.

8
Knowledge requirements

• Know your environment
• Know your perimeter of defense
• Automate tasks
• Simplify data representation

9
Know your environment

• Knowing the ins and outs of your network is a
  necessity
• External network, DMZ and internal network
  architecture
• Other networks, such as VPN and dial-up
• Logistical and geographical locations of servers
  and users
• Different operation systems, applications and
  functionality of servers and client machines
• Network switches and routers in use
• Logistical and geographical locations of critical
  servers (DNS, WINS, DHCP) as well as high-valued
  servers (web servers, servers containing
  intellectual properties)
• You cannot know everything yourself, so know the
  individual experts on each piece of the network
  puzzle

10
Example of environment knowledge usage

• Can isolate IP addresses of Internet, DMZ and
  internal network for different categorization
• Potential detection of external attack versus
  inside job
• VPN and dial-up services introduce other threats
  and need to be given separate consideration
• Allows assignment of customized severity levels
  for different services, such as DNS and servers
  housing intellectual property, for upgraded
  security needs

11
Source of events

• Host level Syslog, HIDS/HIPS, eventlog, log
  files, apps logs, anti-virus signature level
• Network level NIDS/NIPS, NBAD, firewall,
  network routers and switch logs, active directory
  logs, VPN logs, third-party authentication logs
• Audit Vulnerability scanning, OS and patch
  level
• Knowledgebase Software vulnerabilities and
  exploits

12
Know your perimeter of defense

• Firewall
• IDS
• IPS
• Audit capabilities
• Host level defenses
• PENS
• Vulnerability scanning data
• And so on.

13
Know your firewalls

• Location Outer-facing, inner-facing, DMZ,
  internal, internal isolated network
• Type Packet filter, stateful, application
  firewall/proxy
• Whats allowed versus denied
• Capabilities versus shortcomings

14
Know your IDS/IPS

• Which product deployed? NIDS, HIDS/HIPS, NIPS
• Where were they deployed? What kind of traffic is
  being monitored?
• What product/vendor deployed?
• Capabilities versus shortcomings

15
Know your audit capabilities

• Where are logs being kept? Syslog server or logs
  on host?
• How long have logs being kept? Rotated?
• Know your syslog servers

16
Host level defenses

• Anti-virus logs
• Minimum security specification compliance
  enforcement software logs
• OS, service packs, patches-level information

17
Automate tasks as much as possible

• Daunting tasks to detect intrusion due to
• Amount of data involved reaching terabyte range
• Complexity of network environment architecture
  with Internet presence, DMZ, WAN, MAN, PAN, LAN,
  VOIP, VPN, Dial-up
• Complexity of perimeter of defense
• Large IP address ranges used internally, that is,
  using Class A 10.x.x.x
• Multiple internally isolated networks with
  different type of policies, and access controls

18
What and where to automate

• Data aggregation at data source and event
  manager
• Manual, repetitive tasks at event manager and
  reaction
• Data correlation event manager
• Simplify data representation event manager
  console
• Incident notification event manager

19
Group your assets

• Break down IP addresses into groups, such as
  internal, DMZ and others for Internet
• Determine and group all critical servers, such as
  DNS, WINS, and DHCP
• Determine and group all high valued servers, such
  as file shares, web servers, and FTP servers, and
  encrypted content servers for intellectual
  properties

20
Types of correlation

• Sets
• String a group of events together to generate a
  trigger
• Sequences
• String a group of events together in sequence or
  particular order to generate a trigger
• Statistical
• Deviation of normal behavior, such as mean or
  normal curve

21
Methods of correlation

• Rule
• Manually constructed, easy to create/update.
  Usually explicit in nature and can be applied to
  set, sequence and threshold types. Contains three
  elements condition, time interval, and
  response.
• Heuristic
• Similar to anti-virus signature. One signature
  can detect multiple variations. More implicit
  than explicit in nature, thus potential for
  higher false positives/negatives.
• Fuzzy Logic / Artificial Intelligence
• Model approach to correlation that can
  dynamically adapt to changing environment.
  Difficult to produce and still immature very
  cutting-edge.
• Hybrid
• No one doing them all yet. Commonly used are
  heuristic and rule.

22
Correlation constraint

• Time
• Time should be considered when creating time box
  correlation
• Correct time is critical in correlation
• Time synchronization is crucial
• Context
• Order of events sequence is important
• Context can be necessary in correlation rules

23
Sample of correlation flow
24
Graphical representation

• Seeing is believing
• Pros
• Can represent huge data in simple and easy to
  understand graphs
• Cons
• Not many tools (commercial/open source) with this
  capability
• If exist, limited capabilities

25
Effective graphics should

• Show the data
• Avoid distorting data
• Present a large volume of data in small space
• Make large data sets coherent
• Show several levels of detail
• Provide clear purpose of data presentation
• Represent the data and not the underlying
  technology, methodology, and design

26
Forms of data representation

• Graphs
• Link graph
• Charts
• Data maps
• Time series
• Narrative graphics (space and time)
• Animation
• Visualization
• Virtual reality

27
Scanning graph (One source to many target
relationship)
Mar 14 083320 66.34.244.122827 -gt
xxx.yyy.1.118905 SYN S Mar 14 083320
66.34.244.122830 -gt xxx.yyy.1.218905 SYN
S Mar 14 083320 66.34.244.122833 -gt
xxx.yyy.1.318905 SYN S Mar 14 083322
66.34.244.122836 -gt xxx.yyy.1.418905 SYN
S Mar 14 083322 66.34.244.122839 -gt
xxx.yyy.1.518905 SYN S Mar 14 083322
66.34.244.122842 -gt xxx.yyy.1.618905 SYN
S Mar 14 083322 66.34.244.122845 -gt
xxx.yyy.1.718905 SYN S Mar 14 083320
66.34.244.122848 -gt xxx.yyy.1.818905 SYN
S Harder to internalize
Scan activity easily recognized
28
Link graph
Stage 1 of worm propagation
29
Link graph
Stage 2 of worm propagation
30
Link graph
Stage 3 of worm propagation
31
Moving average (Simple network anomaly detection)
Example Monitoring port 445
Increase in moving average, showing an increase
in activities
32
Animation movie

• Inbound connection attempts to San Diego State
  University (SDSU) from external source
  (unauthorized)
• Representing 332 GB of raw data, 3.4 billion raw
  syslog records, and 1 million events
• Period of 1996-2002 (6 years)
• Available at http//security.sdsc.edu/probes-anima
  tions/index.shtml

33
Animation movie
34
Reaction to correlated data

• Enforcement for malware cleaning
• Blocking to minimize malware propagation and
  attack
• Investigation for malicious non-worm activities
• Learning mode for improving data (reducing
  false-positives and false-negatives)

35
Conclusion

• Correlation is a must tool for information
  security professionals
• Time saved in detection will allow faster
  response time
• Faster response time will minimize damages to
  your assets

36
Questions?
37
References

• Event correlation http//www.computerworld.com/ne
  tworkingtopics/networking/management/story/0,10801
  ,83396,00.html
• Protecting the Enterprise with Scalable Security
  Event Management, Part II - Intelligent Event
  Correlation Michael Mychalczuk
  https//www.sans.org/webcasts/show.php?webcastid9
  0468
• Thinking about Security Monitoring and Event
  Correlation http//www.securityfocus.com/infocus
  /1231