Title: Collecting and Preserving Evidence after a System Compromise
1Collecting and Preserving Evidence after a
System Compromise
- Byron Collie
- Federal Agent
- Technical Operations
2Why Bother?
- Determine how they broke in
- Determine what damage was done
- Determine who did it (attribution)
- Support broader incident analysis (ie. not the
only organisation afflicted or affected but
pulling together with other agencies affected) - Meet legal requirements/liability
- Support prosecution
3Computer Forensics
- "Process of identifying, preserving, analysing
and presenting digital evidence in a manner that
is legally acceptable in any legal proceedings
(i.e. a court of law). - Sen. Sgt Rodney McKemmish QLD Police, 1998
Donald Mackay Churchill Fellowship to Study
Overseas Developments in Forensic Computing
4Similar But Different . . .
- Computer crime investigations only differ in two
major areas with other more common investigations
- Different types of records will be searched
- Electronic media, equipment, and communications
will have to be dealt with
5Lack of Preparation!
- Organisations are not adequately prepared to deal
with intrusions from policy or operational
perspectives. - Organisations only address the need to prepare
AFTER a network security breach occurs. - Result is, when first intrusion is detected
- there is no appropriate decision chain in place,
- many decisions are made in haste, and
- much evidence is lost!
6Lack of Preparation can prevent
- determination of the source and extent of an
intrusion, - protection of sensitive data contained on
systems, - protection of the systems, the networks, and
their ability to continue operating, - recovery of systems,
- collection of information in a manner consistent
with legal evidentiary requirements, and - provision of support to law enforcement (LE)
investigations.
7Preparation
- Security Policy, Incident Response Plan (IRP)
with associated Forensic Plan - Implement standing procedures for intrusion
detection - Appoint staff with authority and resources to act
- Implement procedures to react to different
situations and threats - Regularly review all policies/procedures
8Preparation
- PROTECT
- Your system and information resources, and
- Contain the intrusion
- PRESERVE
- The system, appropriate logs, artifacts etc. in a
legally acceptable manner - NOTIFY
- Your Incident Response Team, Management, AusCERT,
and as necessary AFP.
9Evidence
10We need to know
- Where the evidence is
- What the evidence means
- How to put it together
11Aim
- Collect and analyse evidence to form one or more
chronological sequences of events that fit the
evidence - There may be more than one!
- Cant always be conclusive as system/network
evidence is circumstancial in nature - It is a feedback loop
- Analysis leads to more evidence which feeds
analysis
12Sources of Evidence
- Three basic sources
- Users
- Systems (including backups)
- Networks/communications
- Basically we rely on logs and remnants recovered
from a compromised system
13Sources of Evidence
- Users
- First hand observations
- Systems
- Log files!
- Intruder remnants (processes, files etc)
- Networks/communications
- NetFlow Logs
- Firewall logs
- Modem banks/telephone logs
- Network transaction auditing (tcpwrappers, argus
etc)
14Mutable Evidence
- Computer evidence is endlessly mutable
- An intruder might add/remove/modify log entries
- They might compromise system components that
maintain the logs - You might modify something during your
investigation
15Chain Of Custody
- Who has had access to the evidence?
- What procedures did they follow in working with
the evidence? - How can we show that our analysis is based on
copies that are identical to the original
evidence? - Answer documentation, checksums, timestamps
16Four Principles
- From the definition of Computer Forensics, the
four steps that need to be undertaken are - Identify
- Preserve
- Analyse
- Present
17The Four Steps
- Identify the evidence
- Must identify the type of information that is
available - Determine how to best retrieve it
- Preserve the evidence
- With the least amount of change possible
- You must be able to account for any changes
18The Four Steps
- Analyse the evidence
- Extract, process, interpret
- Extract - may produce binary 'gunk' that isn't
human readable - Process - make it humanly readable
- Interpret - requires a deeper understanding of
how things fit together
19The Four Steps
- Present the evidence
- To management, LE, attorneys, in court, etc.
- Acceptance will depend on
- Manner of presentation (did you make it
understandable, convincing?) - The qualifications of the presenter
- The credibility of the processes used to preserve
and analyse the evidence - Credibility enhanced if you can duplicate the
process - Especially important when presenting evidence in
court
20Best Evidence
- In decreasing order of desirability ?
- The original disk
- A binary copy of the original disk
- A log file from the disk (e.g. UNIX wtmp)
- Output of the 'last' command run against that log
file. - Output of 'last collie
- Notes I recorded when I ran 'last collie'
21Best Evidence
- Be clear about what you are copying
- Records from within a file (e.g. grep)
- Contents of a file (e.g. cat)
- A collection of files (e.g. tar)
- A collection of files with attributes intact
- A partition/file system (e.g. a dd image)
- A disk (e.g. a dd image of the whole disk)
- Magnetic domains from the disk
22Best Evidence
- A byte stream image of the disk is good for most
purposes, but not all purposes - You can recover data from files that have been
overwritten, or from damaged hard drives or
floppies - But not from copies!
- This is, of course, expensive
23Evidence
- Remember the four steps identify, preserve,
analyse and present. - Presentation may be a LE issue
- Analysis can be a complex procedure that I will
touch on very briefly - Well concentrate on identification and
preservation - Starting with preserve
24Evidence Preservation
- Remember our principles of best evidence and
chain of custody - Procedures described here are designed to follow
those principles and help preserve evidence - You wont always be able follow these procedures
to the letter - But you should until youre sure that you dont
need to
25Evidence Preservation
- Make a copy
- Generate checksums of original, copy
- Verify accuracy of copy through checksum
- Document
- Who collected it from where, how, when, maybe why
- Give a copy (or the original) to the custodian
- Custodian gives copies to others
- Document chain of custody
- Fewer custodians is better fewer to testify
that way
26Evidence Preservation
- Digital Time stamping
- Basic idea create a checksum and publish the
checksum - Shows that the data matching the checksum existed
in that state on that date - Commercial, free services
- http//www.itconsult.co.uk/stamper.htm - free
service - http//www.surety.com - commercial
27Collect Volatile Evidence
- Volatile evidence is evidence that will
disappear soon, such as information about active
network connections, or the current contents of
volatile memory. - Contrast this with the contents of a disk or tape.
28Collect Volatile Evidence
- From Farmer Venema http//www.porcupine.org
- Registers, peripheral memory, caches...
- Memory (virtual, physical)
- Network state
- Running processes
- Disks, floppies, tapes
- CD/ROM, printouts.
29Collecting Volatile Evidence
- What you do on the system will affect the
remaining evidence - Running ps will overwrite parts of memory
- Your shell may overwrite the history file
- You may affect file access times
- Theres always the risk of trojans! (e.g. gcore)
30Collecting Volatile Evidence
- Rootkits
- Old School replaces system programs (ls, find,
login, netstat, ps) with trojaned versions that
dont report/log certain things - Some versions replaced dynamic libraries
- New School loadable kernel modules that cause
the kernel to not report certain things
31Collecting Volatile Evidence
- You need to use known, safe tools to examine a
system - Statically linked
- Or include your own libraries
- Mount from floppy or CD, through net, or download
through net - This wont help with kernel rootkits
32Collecting Volatile Evidence
- Toolkit might include
- UNIX sh, ls, find, tar/cpio, ps, netstat,
ifconfig, cat, less/more, grep, lsmod, The
Coroners Toolkit, lsof, script, chkwtmp, chkact,
chklast - NT GNU file utils, handleex, filemon, netmon,
dumpevt, dumpreg, dumpacl
33Collecting Volatile Evidence
- If you are collecting volatiles
- Login
- Download your safe tools (net, floppy, cd)
- Memory, swap, /tmp
- Copy whole, or run strings
- Network state
- Connections, promiscuous interfaces
- Processes
- Get list, memory for interesting ones, possibly
list of open files (grave-robber from TCT)
34Document The Scene
Collect Volatiles?
Record Volatiles
YES
Being Safe?
NO
YES
Power Off
NO
Investigate online, run last, etc.
What Was The point? ?
Image Drives?
NO
YES
Make Images
Back _at_ lab, Analyse Copies, Reconstruct Computers,
Analyse artifacts
35Should you
- When you examine a computer, should you
- Turn it off? Switch vs. battery/cord?
- CTRL-ALT-DELETE, L1-A?
- Reboot?
- Unplug it from the net?
- Filter it at the router?
- Leave it running and examine it quickly?
36Shutdown
- Shutdown/halt/sync would leave file systems clean
- But these routines might be rigged for
destruction - Dont reboot!
- Worse than doing a shutdown!
- Wiping /tmp on reboot (if it isn't a ram-disk)
- Is it rigged to restart "bad stuff" (backdoors,
destructive things) at reboot? Or later, through
cron?
37Unplug from the Network
- If you unplug from the network or filter it...
- What about "dead man switches" that detect when
they're off the net and wipe evidence? - Marcus Ranum wrote about this in the CSI Alert,
September 1999, 198
38Leave it Running
- Without unplugging from the network
- Until you power it off
- This is probably safe in the short term
- Risk increases with time, though
- They might use it to do nasty business -
liability? - They might wipe evidence, especially if they see
you poking around
39Power Off
- When you turn it off...
- You lose volatile evidence processes, network
connections, mounted network file systems,
contents of memory... - This is critical evidence in many cases crackers
increasingly store tools, logs on remotely
mounted file systems - On the other hand, if you investigate on running
system, you risk modifying the system (especially
the disk)
40Imaging Disks
- Some partitions might not be mounted or otherwise
visible when system is running (e.g.
multi-booting) - Image of drives is best
- dd if/dev/wd0s2a of/incident/1999-10-11-001/host
a/wd0s2a.dd - It is easier to work with images of the
partitions - You can pull an image of the partition out with
dd later, though - Fdist gives you the partition table
41Imaging Disks
- Save partition info especially if you don't image
the whole drive - fdisk /dev/wd0 gt /incidents/1999-10-11-001/hosta/w
d0.fdisk
42Imaging Disks
- Free space, slack space
- UNIX - The Coroner's Toolkit, Farmer Venema
(available soon) - unrm - pull out free space as a list of blocks
- lazarus - point-n-click to see contents of blocks
- DOS, Windows
- NTI tools getslack, getfree, filter-i,
textsearch plus - Sydex safeback to image drive
43Importance of knowing
- Where the logs might be wrong
- Syslog, NetFlow exports are sent via UDP
- Authentication logs from parallel authentication
servers - NetFlow logs and asymmetric routes
- Spoofed IP addresses
- Writable logs (wtmp, utmp on old UNIX systems)
- Logs modified by the cracker
44Correlating Logs
- You can build stronger case if you can show
multiple sources that are in agreement - Relating log entries to each other
- Matching log entries by value - e.g. IP address
- Matching entries by time
45Time Related Issues
- We often use timestamps to correlate entries from
different logs on different systems - Problems include
- time synchronization
- time zone
- event lag
- chronological order of events
- event bounding
46Time Synchronization
- We can sometimes infer clock offset from the logs
- shell history on computer A shows telnet B at T1
- tcp wrapper on computer B shows telnet from A at
T2 - offset is (probably T2-T1)
- assuming same time zone
- We can't always do this
- if we don't have enough other info to do the work
- event lag
47Date Time
- Record offset from "real" time
- DON'T CHANGE THE CLOCK!
- You can get the CMOS time after powering off
48Time Zone
- You can't compare apples to oranges
- Send, request time zone for all logs
- GMT offsets are best
- This is amazingly painful to deal with in bulk
- Make sure you do the maths right
49Event Lag
- Event lag is the difference in time between
related events in different types of logs - connect from computer A to computer B using
telnet and login - NetFlow log shows telnet starting at 130512
- TCP wrapper on computer B shows telnet at
130512 - wtmp shows actual login at 130558
- Lag can be very variable
50Chronological Order of Events
- Some logs are created in chronological order by
the ending time of the session - process accounting records on Unix
- Cisco NetFlow logs
- TACACS session summary entries
51Chronological Order of Events
- This can be very confusing
- look through flow log, see traffic from computer,
but not telnet traffic to computer - might not
appear until 30 minutes later in the log - look through process accounting logs, see
sub-processes, but not shell process - We often need to reorder by the starting time of
the session
52Reliability
- Logs vary in reliability
- How are the logs protected?
- Some wtmp, utmp are world writable
- Shell history are writable by their owners
- Depends on the integrity of software that creates
log entries - Crackers replace these with versions that don't
log, or which log false entries - rootkit
53Reliability
- Is subject to the security of transmission over
the network - syslog, NetFlow both use UDP
- subject to data loss
- subject to possible spoofing
- Guard against problems by correlating from as
many sources as possible
54Reliability
- We will need to adjust theories to account for
anomalies - see telnet session to computer, but there's no
login session - this might indicate rootkit installation
- doesn't call into question validity of the theory
that someone broke into the system - supports it
55IP Address and Host Name Problems
- IP addresses can be spoofed
- need to recognise cases where this is
likely/unlikely - common in flooding
- uncommon in telnet
- Domain stealing, cache poisoning, etc
- IP address is "better" than the name it resolves
to - really want to log both
- if you have to choose one, choose the IP address
56Recognise What's Missing
- Sometimes the stuff that's missing is what's
interesting - see long telnet in NetFlow to target
- but there's no login session
- raises suspicion that there's a rootkit
- We found a ... directory but it doesn't contain
anything - might be empty
- might be a rootkit
57Users
58Documentation
- Observations of users and administrators are
critical in determining what has occurred - Who, what, why, where, when, and how
- Document everything carefully, consistently,
neatly - You want to demonstrate attention to procedural
details - You want to avoid having to weed through tons of
evidence again later - You can't have too much documentation?
59Documentation
- Paper is good
- Bound, numbered, write in ink
- Date and initial pages as you go
- Electronic can be good but needs to be protected
- The UNIX 'script' program is helpful.
- Easily modified this is both good and bad!
- Checksum and timestamp at intervals
- Use revision control (e.g. CVS)
60Documentation
- Chronological reconstruction of events
- 12/02 1204-1323 attacker.org ping floods
foo.com (flows) - 12/03 1057-1123 attacker.org stealth probes
foo.com (flows) - 12/03 1124 attacker.org telnets to foo.com
(flows) - 12/03 1124 foo.com shows 4 invalid logins
(syslog) - 12/03 1124 attacker.org successful login to
romig_at_foo.com - 12/03 1124-1223 romig account runs exploit
shell script, which invokes ls, ps and rm.
61Documentation
D of S Sources
Denial of Service Victims
Attack Sources
Probe Sources
3
2
Victims
1
Compromised Targets
Probed Targets
4
62PC
PC
PC
Victim Computer
DHCP
Dialup NAT
Routed Network
TelCo
Launch Site
Modem Pool
Terminal Server
Auth Server
63File Integrity Assessment
64FIA
- Detects changes in a systems file system through
the use of a cryptographic checksumming baseline
database. - This provides the ability to
- Detect attacks, and
- Assess damage to system, and selectively recover
from an attack
65FIA
- FIA creates a baseline database, and monitors
files for unauthorized changes. - System files should not change.
- Log files should only grow.
- FIA is probably the most reliable form of
intrusion detection as it does not rely on a
database of attack signatures. - Is however host-based.
66Integrity assessment Theory
- Detects modified files or files that should not
change in certain ways. - System binaries should not change (unless
updated/patched). - Log files should not shrink.
- Security settings should not change unexpectedly.
67Network and CommunicationsSources
68Cisco NetFlow
- Flows represent unidirectional collection of
similar packets - Source, destination Autonomous System Number, IP,
TCP/UDP port - Start, end time
- TCP flags
- IP protocol type
- packets, octets
69Cisco NetFlow
- Multiple flows for a session
- Flow lifetime, timeouts, etc.
- One for each direction, at least
- Problems
- Sorted by end time
- Spoofed IP
- Asymmetric routing
70Etherboy - Real-time Connectivity
71RealSecure - Intrusion Detection
72Sessionwall - Intrusion Detection
73Network Monitoring
74Monitoring of Network Traffic
- Intruder suspected of having compromised hosts on
network. - Explicit agreement with users and/or warning
banners on systems.
75Monitoring of Network Traffic
- Tcpdump/Ethereal
- Requires decoding programs
- must handle retransmissions
- must handle out of order packets
- Monitoring network usage plus traffic
- source/destination addresses
- protocols
- traffic amounts
- traffic timings - relate to expected protocol
76Windows Network Monitor
77Be Careful . . .
- Australian Telecommunications (Interception) Act
- prohibits unlawful interception of any form of
communications except broadcast radio. - US Electronic Communications Privacy Act (ECPA)
- prohibits all monitoring of wire, oral, and
electronic communications unless specific
statutory exceptions apply -
78 Netmap - Network Traffic Analysis
79What the Evidence Means
- Requires a deeper understanding
- How evidence is created
- Where it might be missing
- Or wrong
- Get an expert, ask questions
- Cant cover this today
80What the Evidence means
- A ratbert login entry in a UNIX wtmp file means
- Someone used the ratbert account to login
- Or inserted a fake entry
- NOT necessarily that Rat W. Bert logged in
- A DHCP lease means...
- A computer was assigned the lease
- NOT that that computer was the one using that IP
address during the lease time
81Forensic Tools
82Tool distribution
- CD, floppy, ftp or web site with tools
- Safe copies of standard tools (dd, cp, strings,
ls, ps, netstat, ifconfig, lsof...) for many
platforms
83Other Tools
- The Coroners Toolkit forensic suite from
Venema/Farmer - Chkwtmp - looks for overwritten wtmp entries.
- Chklastlog - looks for overwritten
/var/log/lastlog entries. - MD5 - for validating suspect files.
- COPS - Security profile checker.
- Nessus Vulnerability Scanner.
- Tripwire - File integrity checker.
- Lsof - lists open files used on the system.
- Tcpwrappers - simple connection/access control
mechanism. - Tcpdump - packet capturing utility for monitoring
intruders.
84TCT
- Grave-robber
- Collects info from a computer
- Among other things open deleted files, process
binaries, process memory, system config, MAC
times - Ils, icat
- List info about inodes, contents
- Mactime
- List files whose MAC times have changed
85TCT
- Unrm, lazarus
- Unrm copies the free space from a partition to a
file - Lazarus interprets the contents of that file and
makes it easy to browse
86Tools
- http//www.fish.com/security/lazarus.html - The
Coroners Toolkit. - http//www.net.ohio-state.edu/software OSU
flow-tools, review packages. - http//all.net/ - ForensiX software for Linux.
- http//forensics-intl.com/ - Windows, DOS, NT
Disk imaging, free/slack space searching, etc. - http//www.ntobjectives.com/ - NTLast, Forensics
Toolkit, others.
87Tools
- http//www.somarsoft.com/ - dumpevt, dumpacl,
dumpreg (for NT) - http//www.surety.com - commercial time stamping
service - http//www.itconsult.co.uk/stamper.htm - free
time stamping service - http//packetstorm.securify.com/NT/audit/,
http//packetstorm.securify.com/UNIX/audit
auditing tools for NT and UNIX
88Tools
- http//www.ethereal.com - Ethereal protocol
analyzer
89Summary
90Investigative Process
- Be patient. The investigative process can be
long and frustrating. - Accumulate info about incident.
- Replacement value of proprietary information.
- To establish preventative measures.
- To investigate and litigate.
- DO NOT make legal assumptions.
- DO NOT correspond via E-mail with others about
the incident.
91Summary
- Education, especially before an incident
- use of log book
- network time synchronisation
- effective logging - append-only files
- Policy and IRP
- intruder drills - test procedures
92Documentation
- Document what is happening at all times
- Write down everything that occurred
- Document what steps are being taken during the
crisis - Document what people around you are doing
Do NOT document your information on the system
that has been compromised.
93Be Prepared
- Backups!
- Synchronize your clocks (include log servers)
- Turn up log levels
- Network traffic (probably not contents, though)
- Connections to services on computers
- Authentication decisions
- Address assignments (e.g. DHCP, NAT)
- Events
- Process accounting
- C2 auditing
94Be Prepared
- Copy logs to "secure" log server
- Secure your computers!
- Determine policies, procedures for investigations
- Who decides whether to investigate or fix?
- Who contacts law enforcement?
- Other decisions - wiretap, etc?
- Get critical contact information
95Be Informed
- About recent exploit scripts, newly discovered
vulnerabilities, evidence hiding techniques - Might help identify the means of attack, guess
where evidence might have been hidden (or how it
might have been modified) - Many sources see References
96Evidence
- Needs to be
- complete
- Accurate
- As detailed as possible!
- Identify clock drift problems
- Preparation prevents disappointment!
- Bottom line KNOW YOUR NETWORK AND HAVE A PLAN!
97References
98Forensic Investigation
- Steve Romig, Ohio State University CERT,
http//ftp.net.ohio-state.edu/users/romig - Dan Farmer and Wietse Venemas Forensic Computing
Workshop http//www.porcupine.org great stuff! - Digital Evidence and Computer Crime, Eoghan
Casey, Academic Press, 2000, ISBN 0-12-162885-X.
99Forensic Investigation
- Fighting Computer Crime, Donn B. Parker, John
Wiley and Sons, Inc. 1998 ISBN 0-471-16378-3. - http//staff.washington.edu/dittrich/talks/blackha
t/ - Dominique Brezinski and David Dittrich,
Black Hat Intruder Discovery - http//staff.washington.edu/dittrich/misc/forensic
s/ - David Dittrich, Basic Steps in Forensic
Analysis of Unix Systems
100Mostly for Law Enforcement
- http//www.fbi.gov/programs/lab/fsc/current/comput
er.htm - Recovering and Examining Computer
Evidence - http//www.dcfl.gov/ - U.S. DOD Computer Crime
Lab - http//www.usdoj.gov/ - U.S DOJ
101Articles
- ftp//ftp.net.ohio-state.edu/users/romig/other-pap
ers/intrusion20investigation.doc - My FIRST
paper about intrusion investigation. - http//www.forensic-computing.com - International
Journal of Forensic Computing - http//www.fbi.gov/programs/lab/fsc/current/index.
htm Forensic Science Communications - http//www.sans.org/y2k/finding.htm Finding
listening processes under NT with Inzider.
102Articles
- http//www.wetstonetech.com/ - several good
forensics papers here. - http//www.infowar.com/ - misc. security related
articles. - http//www.securityfocus.com - search for
forensics. Several very good articles. - http//jya.com/esnoop.htm articles on
electronic surveillance.
103Articles
- http//www.fish.com/security/secure_del.html
Peter Gutmanns paper on magnetic and solid state
memory. - http//www.crazytrain.com/ - couple of good
papers here on using dd and other forensic issues.
104Sites with forensics links
- http//www.washington.edu/People/dad David
Dittrichs page. - http//www.forensic.to/forensic.html Zenos
Forensics Site - http//haven.ios.com/nyrc/homepage.html
Reddys Forensics Home Page - http//www.fsu.edu/crimdo/misc.html FSU
Criminology Page
105Incident response
- "Expectations for Computer Security Incident
Response", RFC 2350, Brownlee Guttman,
http//www.cis.ohio-state.edu/htbin/rfc/rfc2350.ht
ml - "Handbook for Computer Security Incident Response
Teams (CSIRTs)", CERT/SEI, West-Brown, Stikvoort
Kossakowski, http//www.cert.org/nav/reports.htm
l
106Information about cracking, exploits, etc.
- NIPC Cybernotes digest - http//www.nipc.gov
- The bugtraq, ntbugtraq, focus- mailing lists
from security focus http//www.securityfocus.com
(plus many other resources) - SANS mailing lists and resources
http//www.sans.org
107Information about cracking, continued
- Packetstorm http//packetstorm.securify.com
- Rootshell archive http//www.rootshell.com
- Hacking Exposed Stuart McClure, Joel Scambray,
George Kurtz McGraw-Hill 1999 ISBN
0-07-212127-0.
108Miscellania
- Decompiling programs http//archive.csee.uq.edu.a
u/csmweb/decompilation/
109Acknowledgments
- Steve Romig from Ohio State University!
110Contact Details
- Federal Agent Byron COLLIE
- Technical Operations
- Australian Federal Police
- GPO Box 401 Canberra ACT 2601
- Ph (02) 6275 7324
- Fax (02) 6275 7330
- Email byron.collie_at_afp.gov.au
111(No Transcript)