Building ICAS with Hadoop and HBase

1
Building ICAS with Hadoop and HBase
2
Outline

• Background Issues
• Motivation Proposal
• ICAS Architecture
• ICAS Procedure
• Experiment Results
• Pros and Cons
• Conclusions

3
IDS Introduction

• Intrusion Detection System (IDS)
• Detecting unwanted attempts at accessing,
  manipulating or disabling of computer systems
  through Internet.
• Alert
• Be produced when IDS detect something as
  malicious.

4
HIDS v.s. NIDS

• Host IDS
• Easy to control and maintain
• Network IDS (NIDS)
• Monitoring network traffic both incoming and
  outgoing.
• Alerts are much more and more complex.

5

• Host IDS is easily to maintain

6
Network IDS can only show many alerts!
7
Whats problem about Alerts of NIDS ?

• Difficult to realize the overall accidents
• Huge Data ? less Efficient
• Ignoring the crucial information easily !!!
• Got Nothing if the database were crash

8
Our Motivation

• To resolve above problems come with huge amount
  of anomaly information generated by NIDS
• So, we need
• Reducing redundancy
• Merge relation
• Higher capability
• Fault tolerance

9
Our IDEA - ICAS

• ICAS, IDS Cloud Analysis System
• Applying Cloud Computing technique
• Higher capability
• Fault tolerance
• Making alerts algorism to generate manifest
  report
• Reducing redundancy
• Merge relation

10
System Architecture
ICAS Component Overview
11
System Architecture
Snort

• SNORT is an open source network intrusion
  prevention and detection system
• The most widely deployed intrusion detection

12
System Architecture
Why is Snort

• The most popular (over 1m downloads and 200k
  registered users)
• Open source network IDS
• Support Windows and Linux
• Light weight and easy to extend
• High accuracy and performance

13
System Architecture
Hadoop

• Apache Hadoop Core is a software platform that
  lets one easily write and run applications that
  process vast amounts of data.
• Inspired by Google's MapReduce and Google File
  System (GFS) papers
• Implements MapReduce and Hadoop Distributed File
  System (HDFS)
• Operates ltkey, valuegt pairs

14
System Architecture
Why is Hadoop

• The most popular open source Cloud platform
• Support its API for developments
• Scalable, economical, efficient, and reliable
• Scaling Hadoop to 4000 nodes at Yahoo! (2008-09)
• Hadoop Sorts a Petabyte in 16.25 Hours and a
  Terabyte in 62 Seconds (2009-05)

15
System Architecture
HBase

• HBase is the Hadoop database
• An open-source, distributed, column-oriented
  store modeled after the Google paper, BigTable

16
System Architecture
Why is HBase

• The Hadoop database
• Output can installed into HBase directly
• Support its API for development

17
System Architecture
Four Components

• Regular Parser
• Parsing original snort log and transfer to HDFS
  (hadoop file system)
• Analysis Procedure
• Dispatch job if pool is not empty and insert the
  result into database
• Data Mapper
• ltkey, valuegt mapping
• Data Reducer
• ltkey1, value1valueNgt
• ltkey2, value1valueNgt

18
Program Procedure
19
Alert Integration Procedure
20
Key - Values
The victim IP addresses
A unique ID used to identify attack method in
Snort rules
The time when the attack was launghed
TCP/IP protocol
Victim ports
The IP address where malicious one launghed attack
Attack was lunched from this port
21
Alert Merge Example
22
Experiment Environment

• Machine ( 6 nodes)
• CPU Intel quad-core, Memory 2G
• OS Linux Ubuntu 8.04 server
• Software
• Hadoop core 0.16.4
• Hbase 0.1.3
• Java 6
• Alerts Data Sets
• MIT Lincoln Laboratory, Lincoln Lab Data Sets
• Computer Security group at UCDavis, tcpdump file

23
Experimental Result
The Consuming Time of Each Number of Data Sets
24
Experimental Result
Throughput Data Overall
25
Pros Cons

• Legible
• Efficient
• Scalable
• Economical
• Reliable

• Non-realtime
• Latency
• Immature

26
Hadoop Development Issues

• Absolute dependency of Hadoop Version ( Neither
  backward nor upward compatibility)
• ICAS can work on Hadoop 0.16.4.
• ICAS has 8 errors and 8 deprecations on Hadoop
  0.18.3
• ICAS has 26 errors and 22 deprecations on hadoop
  0.20.0
• A word-count sample code on hadoop 0.20 cant
  work for hadoop 0.18
• HBases A version is only correspond to
  Hadoops A version (upper or lower not work)
• Sample codes are hardly to find

27
Conclusions v.s. Future Works

• ICAS supplies a efficient way to analyze and
  merge huge number of alerts based on cloud
  platform.
• Support more types of IDS
• Visualize the final results
• Prepare more large-scale and complete experiment

28
Thank You ! Question ?