ENAVis: Enterprise Network Activities Visualization

1
ENAVis Enterprise NetworkActivities
Visualization

• Qi Liao, Andrew Blaich, Aaron Striegel, and
  Douglas Thain
• Department of Computer Science Engineering
• University of Notre Dame

cse.nd.edu
2
Problem

• Complex systems are hard to understand and
  visualize.
• Plenty of micro-level tools
• Host level (syslog, ssh log, etc)
• Network level (MRTG, Netflow, etc)
• Need macro-level picture of network
• Not just in raw network connectivity
• Need to know
• Who (users) are responsible
• What (applications) are running on the network.

3
Context vs. Content

• Packet content
• (protocolIP addressport number)
• local context
• (Network connection, user, application,
  arguments, file accesses)

Who? What?
NetFlow sFlow Analyzer
4
ENAVis
Host
Users
Applications
5
IP/Port ? User/App

• Logging usually in form of
• Network addresses? User identity
• Port numbers ?Application identity
• Network addresses and port numbers are NOT good
  identifier for network activities
• Two problems
• Lack of a mechanism to collect this missing Local
  Context.
• Lack of a tool to correlate the huge amount of
  local context data.
• Visually and interactively explore the data.
• Visualization is the key.

6
Highlights

• Local Context Data Collection
• Light-weight, easy-to-deploy,
• monitoring agent
• Scalable central data processing
• Heterogeneous Graph
• Hierarchical graph representation of data
• HUA Hosts, Users, Applications
• Local-context aware connection chaining
• Visualization
• Statistics report and chart plotting
• Visualize HUA graphs and perform queries
• Interactive exploration

7
Data Collection

• Need to know 4W for each network connection
• who (users)
• what (applications)
• when (time)
• where (hosts)
• Proof of concept
• An easy to deploy and lightweight agent written
  in bash script
• Only calls commonly available system tools
• KISS
• A hierarchy of local context gathering
• Tier one netstat
• Tier two ps
• Tier three lsof (optional)

8
Built-in System Tools

• netstat
• Displays network connections and configurations
• Whois for network connectivity
• Proto, src/dst IP/port, State
• -e ? UID, -p ? PID
• ps
• Currently running processes.
• PID ? GID, PPID, argument list
• lsof
• All open files
• Location (full path) of application, libs, files
• diff
• Difference of two consecutive outputs
• gt start of a new record
• lt end of an existing record

9
Directories/files structure
days
hosts
10
Agent Performance

• 300 machines on our campus since April 2007
• Over 400 GB data
• Mix of CSE faculty / students, scientific grid,
  engineering lab.
• Linux, Solaris, Mac OS X, (Windows)
• CPU
• lt100 ms CPU time every 5 sec (2)
• Bandwidth
• Total data size sent to the server lt 3 MB / day
• 1000 hosts 240 Kbps

11
Server Performance

• Disk
• Sun Fire X2100, AMD Opteron dual core (2.2GHz), 2
  GB SDRAM, 1TB SATA disk.
• 1000 hosts, window size one year 1 TB disk
• 300 hosts, window size past month 30 GB
• Processing
• Time
• Up to 4500
• hosts

12
HUA Graph Model

• Heterogeneous graph
• 4D space
• Hosts, Users, Applications (HUA), Time
• A meta-graph illustrating states

Host-to-Host (similar to Netflow)
Host-to-User
Host-to-App
User-to-User
Application-to-Application
User-to-App
13
Example HU graph

• A HUA graph
• uses User nodes to glue Host and
  Application nodes
• use Application nodes to glue local and remote
  parties.

14
Identities Linking

• Perform connection chaining and bipartite
  matching.
• Mapping src/dst identifiers in O(n) time.
• Allow explicit identity linking between any pair
  of nodes.
• User and Application identities is no longer
  inferred from host addresses or port numbers.

15
Implementation

• Tool developed using Java, JFreeChart, and
  Prefuse.

• Load n days connection data whose state
  established.

16
Graph View
Users
Monitored hosts
Graph controls
hops
Apps
External Domains
Node selection
17
Applications
Time window
Enterprise users
Clusters/subdomains
Local users
Hosts
18
Applications
User IDs
Top Users
User Info
19
Applications
Top Apps
Application Names
20
Applications
Finance System
Trusted Host
Violation 2
Violation 1
21
Summary
ENAVis approach
Traditional approach

• Centralized correlation and visualization make
  life easier for admins ?
• Augmented local-context data (Users and
  Applications), which are not available in
  existing schemes.

22
Conclusion

• It is important to know who is responsible and
  what is running on an enterprise network.
• Local-context (users and applications) is useful.
• Network management, security policy auditing,
  fault localization, forensic, etc.
• ENAVis
• Collects, fuses, and visualizes the missing
  local-context data.
• Interesting HUA network connectivity graph.
• Interactive exploration tool.
• Future works
• Windows agent as a service
• Data mining modules built into the tool.

23
Acknowledgements

• This work was supported in part by
• the National Science Foundation (CNS-03-47392,
  CNS-05-49087), as well as
• Sun Academic Excellence Grant (AEG)
  (EDUD-7824-080234-US).

24
Visit http//netscale.cse.nd.edu/Lockdown/
Thank You !
25
Demo