NetFlows

1
NetFlows Network Forensics

• E. Larry Lidz
• ellidz_at_pobox.com

2
NetFlows

• Logs right from routers and switches
• Supported by most network vendors
• Takes some CPU time on the routers
• Might be turned on for network performance
  reasons
• Quick to search through
• Multiple ways to save on a system multitude of
  available commercial tools

3
General Intro to Flows

• Logs flows, not connections!
• This is good and bad.
• Flows are uni-directional
• Youll see two flows for each connection one in
  each direction.
• Tools can reconnect them using heuristics

4
Introduction to Network Auditing

• Obligatory "audit" definition
• 1. trans. To make an official systematic
  examination of (accounts), so as to ascertain
  their accuracy.
• So, what is "Network" auditing?
• Making a systematic examination of network
  traffic to understand the traffic and verify its
  legitimacy.

5
Introduction to Network Forensics

• Obligatory "forensic" definition
• A. adj. Pertaining to, connected with, or used in
  courts of law suitable or analogous to pleadings
  in court. forensic medicine medicine in its
  relations to law medical jurisprudence.
• So, what is "Network" forensics?
• Looking at traffic on a computer network in such
  a way that it could be used for evidence in a
  court of law.

6
So, what am I talking about?

• Basically, the ability to log information about
  your network traffic in order to help with an
  investigation.
• Generally, a compromise.
• Also useful for trying to explain unexplained
  network problems.

7
A Quick Story

• Machine on network compromised
• Launched an evil DoS attack
• Took out an ISP for a weekend
• ISP thought they knew who did it
• We were contacted by Alaska State Police and
  Secret Service
• Suspect is in Anchorage, AK
• Suspect is suspected of Credit Card Fraud
• Compromise is a big, big problem
• Lots of damage caused to ISP
• Credit Card fraud via our network

8
A Quick Story, continued

• The network audit logs show that the compromise
  occurred from .ak.us.ibm.net
• (IBM was a large ISP at the time)
• We turn logs over to Alaska State Police
• Note none of our sensitive data was turned over
  much, much easier to convince the powers that be
  to turn over
• Kid denies it, confesses when shown the logs
  turns on friends
• (You can tell this is an old story these days
  itd be a professional crook)
• Parents offer to pay restitution
• (Do you think that ever happened?)
• Network logs save the day!

9
Some Terminology

• Network Flow
• A group of packets related by having attributes
  in common.
• Distinct from a network connection.
• AS
• Autonomous System Number
• Every ISP has one, used in routing.
• VLAN
• Virtual LAN
• A single broadcast segment of a network (a
  subnet).
• These definitions are slightly simplified as the
  complexities arent relevant for this
  conversation. Im sure youll hear more details
  on these later on if you havent already.

10
TCP/IP

• When looking at netflows, the more you know about
  TCP/IP, the better.
• 3 main protocols TCP, UDP, ICMP
• UDP/ICMP are conectionless.
• No way to easily tell if traffic is the start of
  a "connection" or part way through.
• ICMP Internet Control Message Protocol.
• Ping, traceroute, etc.
• Type destination unreachable (3), echo req (8),
  echo rep (0).
• Code 3.0 network unreachable, 3.3 port
  unreachable.
• TCP has connections.
• Controlled by TCP Flags FIN, SYN, RST, PSH, ACK,
  URG
• FIN Sender is finished sending data.
• SYN Synchronize sequence numbers to initiate
  connection.
• RST Reset the connection (closing it).
• PSH Push data to the application as soon as
  possible.
• ACK Acknowledge
• URG Urgent flag set.

11
A TCP Connection

• Client sends a SYN packet to server.
• If the server is listening, it responds with a
  SYN-ACK packet.
• Client then sends back an ACK packet.
• Communication takes place with ACK (and often
  PSH) set.
• Push flag is generally set when the application
  is waiting on input from a user or otherwise
  expects to be idle.
• Side closing the connection sends a FIN.
• Other side sends an FIN-ACK.
• Other side sends a FIN.
• Closing side sends an FIN-ACK.
• If the server isn't listening, it sends a RST.

12
IP Addresses (v4)

• Netmasking
• 128.135.0.0/255.255.0.0
• 128.135.0.255 is broadcast for 128.135.0 subnet.
• 127.0.0.1/255.0.0.0
• local net. 127.0.0.1 is local host
• 10.0.0.0/255.0.0.0, 192.168.0.0/255.255.0.0,
  172.16.0.0/255.240.0.0
• private IP space

13
TCP/UDP Ports

• Useful to know a variety of ports.
• In particular the ones that are used a lot on
  your networks.
• General
• 21 ftp
• 22 ssh
• 25 smtp
• 53 dns
• 80 http
• 135, 137, 139, 445 windows ports
• Details beyond the scope of this conversation
• 443 https
• 6667 irc

14
NetFlows vs. other security

• vs. Intrusion Detection System
• We have an IDS, what else do we need?
• vs. System Forensics
• "We can always look at the system to figure out
  what happened.
• vs. Sniffers
• "We can always install a sniffer when we expect
  something is wrong.
• vs. Firewalls
• "We have firewall logs, who needs anything more?"
• "When I've already got X, why do I need Network
  Audit Logs?"

15
NetFlow vs.

• IDS
• Generally do not log traffic unless it is
  suspicious
• System Forensics
• With systems, youre always looking at a system
  that has been tampered with
• Sniffers
• Disk consumption privacy concerns
• Firewalls
• Often only at the network border not optimized
  for searching logs

16
Defense in Depth

• Use it all.
• (Based on available budget, resources, etc.)

17
Network Design Considerations

• Decide what you want to log.
• Which machines? Which networks?
• Choose places that handle interesting traffic.
• Some examples
• Gateways
• Peering points
• DMZs
• Important machines
• Keep logging server close to router/switch
• Theoretically possible to inject fake packets
  into stream
• Difficult
• Network traffic requirements for logs fairly
  small.
• About a 1.5Mb/sec per 100 Mb/sec of traffic.

18
Prepare

• Prepare ahead of time!
• That way, when you you need it, you'll have logs
  to consult.
• Too late if the incident has already happened.
• Allocate hardware
• Start small if necessary, build a case for the
  usefulness of the tools.
• Install software
• Test! See if you can avoid the logs.
• If you can, it's probably misconfigured.
• We noticed that we were missing logs from one
  segment to external networks.
• Turns out we were missing an interface on our
  switch.

19
What can you see in a NetFlow?

• Think of it like phone call records who called
  whom, when.
• Without the conversation.
• Source/Destination IP
• Input/Output Router Interface
• Protcol
• Type of Service
• NextHop
• Packet Count
• Octet Count
• Start/End Time
• TCP Flags
• Source/Destination AS
• Source/Dest Network Mask
• Input Output Interface encapsulation Size
• IP Address of next hop within the peer
• Router IP of cache shortcut in supervisor

20
Reading NetFlows

• Depends a bit on the tool but
• 04/5/2011 115946 -gt 04/5/2011 115946 6 01
  65.219.38.62 3844 lt-gt 11 128.135.12.7 http 8
  480 00 FS-PA-
• 04/5/2011 115946 -gt 04/5/2011 115946 6 11
  128.135.12.7 http lt-gt 01 65.219.38.62
  3844 11 11636 00 FS-PA-
• Flow start time, flow stop time
• Flows sorted by stop time
• Protocol number
• Source interface, source hostname, source port
• lt-gt
• Source and destination are the source and
  destination of the flow, not of the connection.
• Should be two flows for each connection (assuming
  traffic is sent in both directions).
• Destination interface, destination hostname,
  destination port
• Packets in flow, Bytes in flow, Type of Service
• TCP Flags All TCP flags set in that flow.

21
An Incident

• Received a report of a compromised system from a
  sysadmin
• Well of ten systems.
• Systems had a file which listed the other
  machines in the same BotNet 35 on our network
• Each system we looked at had a different list of
  systems in that file.
• The Challenge find all compromised machines.

22
Incident, continued

• We chose a system that wasnt doing much
  (network-wise).
• Why?
• Pulled all the NetFlows for that system.
• By reading the Flows, we were able to find the
  signs of a compromise (attack against an RPC
  service).
• Also saw another host (38.31.107.236) connecting
  to a backdoor on a port (1524)
• Then connections out from the machine to download
  a rootkit (to 152.3.127.99).
• Then the BotNet traffic starts up.
• Note Im skipping some of the analysis of how we
  figure out that it was downloading a rootkit,
  etc.

23
Incident, continued

• Looked for all connections from our network to
  152.3.127.99
• Sorted the output and got the list of our IP
  addresses in it.
• We were able to easily pull together the list of
  compromised machines by looking at the network
  trail of the one machine.
• In practice, we hand verified a lot of this.
• Its always good to double check.
• Compared the machines to those listed in the file
  on the infected machines.
• Did other analysis of the systems to ensure that
  they were compromised. (Forensics in depth?)

24
NetForensics when?

• When was the first traffic to a back door?
• When did traffic first seem to successfully
  connect to ports that previous had no traffic?
• When did BotNet traffic start up?
• Did the machine start transferring lots more
  traffic at a certain point in time?

25
NetForensics how?

• What type of machine is it?
• When you look at it, what vulnerable services, if
  any, do you see?
• Did unexpected traffic start up shortly after a
  connection to a service running on the machine?
• Connections to a port's service that isn't used.
• ftp/http out to a third site
• Particularly if it is grabbing a rootkit.

26
NetForensics who?

• Who is connecting to it that you don't expect to
  connect?
• Which IP caused the compromise?
• If you know when/how it was compromised.
• Who connected to the backdoors?
• Look for weird connections.
• May be a backdoor or a covert channel.

27
NetForensics what?

• How much data was transferred?
• Useful if you fear that information was stolen.
• If they only transferred 500k of data, they
  didn't take the 20 Gig master plans for the new
  virtual-hyper-cyber-widget of doom.
• What was on the machine in the first place?

28
NetForensics why?

• In my experience, NetFlows arent so useful for
  determining motive.
• Sorry.

29
When in Doubt

• Read through the traffic logs for the compromised
  machine.
• Extract logs for the specific host.
• Very Time Consuming
• You won't miss anything...
• Unless there's too much data to look at.
• Start ignoring stuff that you suspect to be
  legitimate (ntp, dns, etc.)
• Don't worry about stuff before the compromise.
• Build a timeline.
• Keep notes as to the connections that you see.
• Who connected out to where at what time?
• Which are legitimate?
• Interview system administrators and users.

30
Things to look for

• Intruders often script attacks
• Machines will have similar traffic patterns
• Same source IPs.
• Same backdoor ports.
• Connect to same external site.
• Known bad traffic
• IPs/hosts list on bad traffic reports
• Known malware ports
• Example finding SQL Slammer
• flow-extract -nd ft-v06.2003-01-25.030000 -e
  'proto 17 port 1434 pkts 1 octets
  404 ((srcnet 10.0.0.0/255.255.0.0
  dstport 1434 ) (dstnet 10.0.0.0/255.255.0.0
  srcport 1434)) print

31
Argus

• Logs off of a spanning port
• No need to have router support
• From QoSient.
• Free for most purposes. Read the license
  agreement.
• Can log bytes of application data, too. packet
  contents, smtp exchange, ftp username/password
• A bit slower than NetFlows
• I use it at home

32
Summary

• NetFlows are a powerful tool.
• Extremely useful for Security, but also for
  network management.
• If I can find machines compromised by a
  malefactor trying to remain anonymous, I can
  easily find the bandwidth hogs.
• Heavily adopted behind the scenes in Security and
  Network tools.
• Very useful at making network traffic tangible as
  youre learning more about it.
• Theyre a little bit higher level than a Sniffer
  would show you, and therefore easier to consume.

33
Questions?

• Questions?
• Comments?
• Thoughts?
• Anything else youd like to learn about in the
  time remaining?