Live Memory Analysis Tools and Techniques in Linux Environment-TechForing

1
Title Live Memory Analysis Tools and Techniques
in Linux Environment Abstract Live memory
analysis on the Linux based system has never been
easier in the past. Analysts always had to take
traditional approach which involved multiple
stages before even begin the investigation.
Taking the snapshot of the whole disk or the
memory, shutting down the compromised machine and
then investigate the memory in the lab were few
steps an analyst must had to follow in a
traditional approach, which has a high risk of
losing potential data as killing the machine not
only end all the processes but
also alter many programs behavior which can
mislead the investigator. On the other hand,
capturing a memorys live state can be very
crucial for forensic investigators as it contains
all the running processes, file, directory
information, recent activity details, session
details and the user activity related
information. Apart From post investigation issue,
there has been limited work done in Linux memory
forensic analysis compare to windows. Live memory
analysis requires precise understanding about the
complex struct layout information in memory and
how memory store data. It also involves multiple
steps to carry out to analyze and extract the
valuable evidence. Looking for the clues and
tracing the suspicious activity in the memory can
be a slow and steady task if traditional approach
is taken and one doesnt know which tools to use
and what techniques can retrieve information fast
and in a sufficient manner. Though there are many
Linux based forensic investigation tools but only
few of them are focused on memory analysis, among
them many are outdated. Open source ones are
either not properly documented or the documents
mismatch the current available version. In this
paper I actively researched the live Linux memory
analysis related works and took the action to
demonstrate, how to effectively carry out live
Linux memory analysis using different tool set
and showed different approach and techniques to
dissect a live memory on Linux platform. The main
purpose of this paper is to provide a details
hands on demonstration about how to use various
forensic and non-forensic tools for incident
response on live Linux memory analysis. Introducti
on In the era of computerized world and digital
cryptocurrency like bitcoin, cyber criminals have
become more active than ever and they are
expanding their devil eye throughout all kinds of
industry to single individuals. This rise of
cyber criminals activity also have forced the
forensic analysts to think out of the box and
come up with new techniques to response
immediately for any unexpected incident. Memory
analysis is one of the most sophisticated way
that can provide insight details about a
criminals activity if performed properly and in
time. Though memory analysis is been around for
some time but it is not only very hard to perform
but also time consuming and dangerous as there is
a possibility of losing sensitive evidence if not
properly handled. There are many forensic
investigation tools available out there but out
of all, very few can be used to retrieve
sensitive information like username,
password,email, encrypted keys, hashing details,
activity and login information along with many
other data. Memory analysis tools are among them
what can be a real asset for any analyst. However
as the memory focused analysis tools
2

• are expanding their wings and integrating new
  plugins every day, so does the number of new
  analysts are increasing but many of them dont
  know how to use these tools and use them for
  their own benefits.
• At first observation, many can make a mistake to
  think physical memory is randomly constructed
  wide range of data but that is not true at all.
  Physical memory is used to store all programs
  activity and state in a much disciplined way. The
  developers take the following steps to ensure the
  structure of a memory
• Employ logical construct (C struct) to gather
  relevant data into logical units - This represent
  abstract data types
• Struct is laid out in a memory in a consistent
  manner
• Generate codes to access various members of the
  struct
• Analyst must have access to abstract struct
  object to successfully extract vital information
  from the memory. For an open source operating
  system like Linux, it is very difficult to do
  that as struct layout and kernel version changes
  frequently. In a case like this, researchers
  tried to find a solution for the question that
  how memory analysis could be done in Linux in
  more automatic fashion?
• To make analysts life easier few open source
  projects were founded and many forensic
  professionals did intensive research to find a
  way to automate everything which is fast and
  efficient. Though open source foundations like
  volatility memory analysis framework and Rekall
  successfully found a way to use the information
  derived from debugging streams to analyze memory
  images from multiple versions of Linux based OS
  but many of them have limitations to guide an
  examiners to conduct their analysis effortlessly.
  This process not only requires many manual steps
  but also all the memory analysis framework has so
  many plugins and modules that it can be
  overwhelming for an investigator to choose
  between right tool for the right incident
  response that can lead to the right path to
  extract the right evidence.
• This situation has raised questions like
• What is the right approach and methodology to
  perform live memory analysis without losing or
  altering valuable evidence?
• What tools and techniques to use for different
  incident response scenarios?
• In the past academic work, most researchers
  focused on particular issue to carry out Linux
  memory analysis and usage of tools only on disk
  analysis. In this research my prime goal is to
  address the obstacles on performing live Linux
  memory analysis, various tools and techniques
  focused on only live Linux memory analysis and
  shed light on details methodology to carry out an
  incident response.
• I have used an action based method to demonstrate
  various tools and techniques to perform my
  analysis. I started with LiME to capture the
  memory. After capturing the memory and creating
  the profile , I have used Volatile and Rekall
  Memory Analysis Framework to perform extensive
  analysis on the memory image to find out evidence
  of criminal activity .I have also used simple
  tool like cat, grep and photorec to demonstrate
  the techniques to perform quick investigation on
  live memory and retrieve valuable information. A
  cloud based (AWS ) Ubuntu 14.04 server with 1 GB
  memory was used during the practical work.

3
The outcomes of the practical work provides a
high level overview of how to handle the
compromised system, capture a memory of a live
system or running the analysis on the live memory
instead of capturing it, usage of decomposer
library for advanced binary analysis, how to use
system map to generate a profile on any Linux
based machine, how to extract system information,
process memory , network information and perform
rootkit analysis using volatility, how to create
virtual environment to run live mode using Rekall
, details comparison between volatility and
rekall and some more techniques with non-forensic
tools. Literature Review of Previous Work Socala
and Cohen 1 stated, if the investigators want
to extract any sensitive information from a live
memory image, they must get their hands on
abstract struct objects first, which the program
handles from the physical memory. In order to
achieve this, the analysts should have an exact
model of the physical layout of the structs and
their members data types, which is a difficult
process, especially for Linux based OS. In most
cases, investigators perform live triaging by
copying and moving the necessary files to a
remote host, likely configured to the targeted
host, make the profile and then copy the profile
to the target host. To make everyones life
easier, the authors have developed a tool called
Layout Expert which not only can calculate memory
layout of critical kernel structures without any
extra tools but also generate automatic profile.
The generated profile can be used to diagnose the
live memory via /proc/kcore device. The way
Layout expert tool work is as follows First it
predicts the struct layout , after that it
calculates a profile without installing the full
compiler tool chain or contacting any external
server. The Kernal sources are dissected into a
PRE-AST which is later used to get rid of
unnecessary code, macro expansions and struct
definitions. On this stage, the new version of
Pre-AST is being serialized using JSON and stored
on the disk. This file can be shipped to live
memory acquisition tool in the data files group.
The final version of layout is bundled with the
system.map to produce a profile for analysis
framework. Case, Marziale and Richard 2
mentioned, in earlier days running a number of
statically linked binaries and capturing the
output of the commands ( ls,ps,lsof ,lsmod etc
for Linux ) for later usage on the target machine
was required for live forensics. Physical memory
dump was also limited to only string searches.
Linux releases new kernel versions often and it
is quite difficult for analysts to model the
required structure even if they have the complete
source code. Without having a proper
understanding of the data structure and
algorithms of the original OS , it is not
possible for the investigators to analyze
everything on the physical memory dump. The
authors tried to find a solution for this issue
and developed a tool called Ramparser which
allows live memory analysis tool to work with
wide range of kernel versions automatically
without relying on any specific kernel
distributions. At the beginning Ramparser
collects all the required structure offsets (
task_struct,mm_struct,file, dentry, Qstr,
inet_sock, Sock, Socket ) to analyze the memory,
then it gathers information according to the
needs of its features and store them in a SQLite
database. This database contains all the
sensitive information about memory image that
Ramparser can retrieve. Authors had successfully
tested the tool on different distributions of
Linux from 2.6.9 to 2.6.27 kernel versions. Wang
et al., 3 proposed a methodology for live
investigation by breaking it into multiple stages
such as information gathering, analyzing it and
creating a report. This method is applicable for
the memory
4
image. Live examination based on the physical
memory works on the running host. An external
media is linked to the targeted host but this
will alter the behavior of the system and it
could be difficult to confirm the validity of the
evidence. It is very significant to enable the
validation of live evidence. After procuring an
image of memory, the raw data of the toolkit
ought to be accessible for the third party
assessor so that they can come to a conclusions
about the validation of the live evidence. With
this model, it is possible to reduce the impact
of live evidence collection by executing the
memory image collection task. In 4, Aljaedi
et al. depicted the volatile memory strategies,
because of fast increment in memory size, the
analysts firmly prescribe the live reaction
approach for securing of unpredictable proof.
Through this procedure, the specialists can
gather the data about live processes as well as
about the cache and finished processes. Volatile
memory examination turns into an imperative bit
of examination since the physical memory could
have vital information which analysts can't find
on the hard drives. To get a decent hold of the
occurrence, the volatile information
accomplishment is the first step in the digital
examination. Typically the examiner accumulates
the volatile information through live reaction
while the criminal may utilize different
libraries to influence the system call, connect
to the kernel and update the information. To
explore the unstable information of target
system, a memory image examination approach is
also utilized. This approach may fill in as an
alternative to live reaction. With administrative
tools, the inspector can acquire all the
unpredictable data in the memory including the
live and ended processes data. S. Balogh, and M.
Pondelik 5 proposed how to discover encryption
keys from the memory dump, in Linux they utilized
TruCrypt application that has AES encryption
algorithm with XTS mode to procure image of the
memory so they get everything except
password. Investigators examine that the
encryption keys is found in HEAP of the last
process produced from the HEAP procedure by the
TrueCrypt. They discovered PID and with PID last
process, they additionally discovered address of
pages in the memory dump which is the access
process, so by using this technique it is
possible to lessen the span of image since
analysts don't need to take image of the full
memory. All the keys are available in kernel
memory since all the encoding and decoding forms
are executed as kernel factor. So the TrueCrypt
driver appoints the kernel mode memory and store
all the keys as non- paged pool. The first
restriction of this work is finding the location
of the keys. Analysts need to install third party
encryption packages on the target machine for
various encryption packages. Second, if the
memory contain any fraction of decomposition then
the extracted keys might have some blunder and
the final issue about this procedure is, the key
can split through the memory in a consistent way.
To properly configure the virtual machine to
adopt the target OS settings, tools like LiveView
and VFC are commonly used. Mariusz Burdach 6
presented various methods that can be used during
an evidence collection phase from live memory for
Linux based OS. First he started with running a
network sniffer tool (tcpdump) to monitor the
communication flows to and from a compromised
machine. He pointed that it is very much possible
to detect malicious activity by analyzing the
captured data in raw format in the network. He
also advised to build and copy the following
tools to the external media (Such as CD-RW disc)
nc, dd,
datecat, hunter, pcat, NetstatArproute, insmod,
dmesg that would be mounted in the compromised
5
system. In initial step, he suggests to take a
picture of the compromised systems screen before
mounting the media. After that author started to
collect information from cache tables (arp and
routing tables), current / closed connections and
port details, list of modules loaded into kernel
memory, list of active processes and list of
suspicious processes. A very useful method to
detect any rootkit if it loaded into kernel
memory by checking one of its task if it is
hiding any open port. The author used gdb tool
for advance analysis on kcore file. As an example
of usage he compared and verified the system call
address by looking for address of the
sys_call_table from Symbol.map. Project
Methodology a. Basic Idea Before explaining
the methodology, let me provide some basic idea
of memory usage when any program is running. Once
any program starts, the memory module divide into
program segment and data segment part which
consist of heap and stack.
Program Segment Data Segment
Program
Heap Bottom ? Stack Pointer ? Heap
Heap Bottom ? Stack Pointer ?
Heap Bottom ? Stack Pointer ? Stack
Heap Bottom ? Stack Pointer ?

• Methodology
• To obtain all the reliable and sensitive data
  from both segment we have used two main method.
  Both of the methods has sub steps and some
  benefits over each other.
• Live Memory analysis after capturing it
• In this method a memory was captured in lime
  format with .mem extension prior to analyze. The
  following methodology was taken step by step to
  analyze the memory without putting it into risk
  of losing any information.
• Memory acquisition
• Creating profile
• Reconnaissance
• Enumeration of running process
• Recovering memory mapped files and directories
• Analyzing active tasks details
• Investigating for rootkits
• Live memory analysis without capturing the memory
• In this method no captured memory was used and no
  profile was created prior to analyzing the
  memory.
• Automatic profile detection
• Reconnaissance

6

• Enumeration of running process
• Recovering memory mapped files and directories
• Investigating for rootkits
• String search
• Detecting and Recovering hidden data
• Memory acquisition
• In this step the live state memory of the target
  machine was acquired
• Creating profile
• A profile is required to perform the memory
  analysis for volatility and rekall framework. To
  create the profile , Linux distribution, kernel
  version, cpu architecture information was
• retrieved, which was used to generate system map
  file by LiME and module.dwarf file by volatility.
  Later on both system map and module.dwarf file
  was used to create the compatible profile to
  carry out our analysis. However for rekall, we
  didnt have to create any profile manually as
  rekall has a large collection of profiles for
  various operating systems with many kernel
  versions.
• Reconnaissance After creating the profile , we
  started our investigation slow to retrieve
  information like system and network information
  and got a technical understanding about the
  memory and the target machine.
• Enumeration of running process
• On this step we enumerated the running processes,
  This step was very crucial to analyze the
  incident in depth and identify if any malicious
  activity is in progress.
• Recovering memory mapped files and directories
• Retrieved the path of all files and directories.
  With this technique one can find many sensitive
  information along with further clue of what the
  target machine was compromised on the first
  place.
• Investigating for rootkits
• This was the final stage to use all the possible
  techniques to identify any rootkits, infected
  files or any other malicious activity.

Live Memory Analysis with Volatility Live Memory Analysis with Volatility Live Memory Analysis with Volatility
Main Task Sub Tasks Tools/Plugins/ Files
Memory acquisition - LiME
Profile Creation - System Map, module.dwarf
Analysis System Information Linux_cpuinfo
Network Information Linux_arp
7
Linux_enumerate_files
Process Memory Linux_bash
Rootkit Detection Linux_check_creds
Linux_check_fop
Linux_check_modules
Linux_check_syscall
Live Memory Analysis with Rekall Live Memory Analysis with Rekall Live Memory Analysis with Rekall
Memory acquisition LinPmem
Profile Creation Automatic
Analysis pslist
Describe(pslist)
pstree
session
paxview
arp
banner
check_afinfo
check_creds
check_proc_fops
maps
Live Memory Analysis Techniques with Non-forensic tools Live Memory Analysis Techniques with Non-forensic tools Live Memory Analysis Techniques with Non-forensic tools
String Search - Cat , Grep
Detecting and Recovering hidden data - Photorec
d. Procedure I have taken the following
procedure to perform the memory analysis.
8
Victim system is being identified Image is
being acquired
Acquiring Image
Profile is created Memory structures are
parsed Memory segments are identified
Establish Context

• Looked for suspicious activity like unlined
  processes, known malware, hook etc
• Discovered interesting information like username,
  password, email
• Reocvered and reviewed all files including hidden
  files

Analysis
Documentation
Practical work Volatility Volatility is an open
source framework focused in memory analysis for
the purpose of forensic investigation, malware
analysis and incident response. It is developed
in python and supports all kinds of OS and have a
number of plugins to perform normal to deep
memory forensic investigation. It can also
analyze a wide range of file formats including
but not limited to VMware .vmem ,raw dumps,
expert witness (EWF), hibernation files ,crash
dumps, VirtualBox core dumps, VMware saved state
and suspended files (.vmss/.vmsn), direct
physical memory over Firewire and LiME (Linux
Memory Extractor). Volatility works in a fast
and efficient manner regardless of the size of
the memory. Environment Setup To demonstrate the
usage of volatility, I have setup an Ubuntu
14.04.5 server on AWS with a LEMP stack on it.
The server has a live website running that is
configured in a virtualhost with Nginx web
server. It has a memory of 1 gb. Basically we
are going to acquire the memory and dissect it
using volatility.
9
I did ssh to connect to the target server with my
username and private key. I need to have full
power on the server to perform all the tasks
without having access limitation, so I decided to
become root!
sudo su
It is always a good idea to keep everything up to
date before starting any serious work on the
system, so I performed an update and upgrade
sudo apt-get update
sudo apt-get upgrade
10
To make sure our memory acquisition and analysis
tool will work without any issue, we will need to
install some third party dependencies.
sudo apt-get install build-essential
11
sudo apt-get install Linux-headers-uname -r
sudo apt-get install git python dwarfdump zip
python-crypto
I couldnt find the distorm library directly from
the Ubuntu official sources. So I pulled the repo
from github and build that manually. Distorm is
fast and easy to use decomposer library. It can
return a binary structure upon giving an
instruction which is very helpful for advanced
binary code analysis.
git clone https//github.com/gdabah/distorm.git
12
cd distorm python setup.py build
Installing LiME Volatility doesnt provide any
memory acquisition tool but it recommends to use
a LiME format to perform and get a better
result. LiME is a LKM (Loadable Kernal Module) to
acquire our target systems RAM. It is possible
to acquire the memory both to the file system and
over the network. It avoids any kinds of
interruption between user and kernel space
processes during taking the snapshot of the
memory. Thats why it is able to capture the
memory in a live version and produce the exact
copy including all the vital information on the
memory without ending or disrupting any
processes. We will need to build LiME according
to our targets kernel version.
git clone https//github.com/504ensicsLabs/LiME
cd LiME/src/ make
13
On the above screenshot if we have a look at the
last line of the make output we can see the
kernel version that we need to use. In my case
its lime-3.13.0-105-generic.ko. Now I am going
to capture the systems memory and acquire a copy
of it on my current folder with lime format by
loading the kernel module. After capturing the
memory I have moved it to /tmp folder for safety
reason. We will use this snapshot to perform our
forensic analysis.
sudo insmod lime-3.13.0-105-generic.ko
pathubuntu.mem formatlime sudo cp Ubuntu.mem
/tmp
Installing Volatility Now we have are going to
install and configure the volatility tool. Its
better to get the updated version of the
volatility directly from the official site.
git clone https//github.com/volatilityfoundation
/volatility cd volatility/tools/Linux/ make
clean
14
make
It is time to locate our system map which
provides details information to volatility about
how memory analysis snapshot is structured. In
our case(Ubuntu) we can find that in /boot/ so,
ls -al /boot/

• System.map-3.13.0-105-generic is what we will
  need for our next step.
• Creating Volatility Profile
• It is important to create a profile to make
  volatility work properly and perform memory
  analysis. For any kind of memory analysis using
  volatility, we must be careful to build the
  profile for the system that we want to analyze
  but not the system where we want to analyze the
  memory. Volatility comes with a list of profile
  but in our case, we will make one manually as it
  is not available in the pre-defined list.
• We need to keep three things in mind to match the
  profile with our target system, which are
• Linux distribution
• Exact kernel version
• CPU architecture (32-bit, 64-bit, etc).
• We will select System.map-3.13.0-105-generic file
  as it matches our lime-3.13.0-105-generic.ko
  file.
• Our profile will require module.dwarf file made
  by Volatility and our System map, so I will
  simply zip them together.

cd ../../
15
sudo zip volatility/plugins/overlays/Linux/Ubuntu
140405.zip tools/Linux/module.dwarf /boot/System.m
ap-3.13.0-105-generic
Running Volatility Before we run the volatility,
lets find out if our profile has been created
successfully and everything is in place.
python vol.py --info grep Linux
As we can see Volatility is printing the profile
that we have created on earlier steps
LinuxUbuntu140405x64 A Profile for Linux
Ubuntu140405 x64 It indicates everything has been
setup properly and we are good to start our
memory analysis. Lets have a look at the
available plugins from volatility that we can use
to perform our memory analysis.
python ./vol.py --info grep -i Linux_
16

• In this demonstration I am going to perform the
  analysis using various plugins to give an
  overview idea of their usage and outcomes.
• There are total seven main categories of plugins
  in volatility.
• Processes
• Process Memory
• Kernel Memory and Objects
• Rootkit Detection
• Networking
• System Information
• Miscellaneous
• Each of the category has bunch of plugins that
  can be used to perform details analysis on the
  memory. These tool set provides insight details
  of the memory and can help an analyst to navigate
  through the memory , understand its pre and post
  compromised state along with performing very high
  level analysis. It doesnt stop there, it also
  allow an investigator to gather all the required
  digital evidence to put together his case.
• We will run all the plugins against our saved
  memory snapshot located in /tmp. We will also
  need to define our profile before the plugin
  module name.
• Lets start with something simple and get
  ourselves familiar with the plugins.
• System Information
• a. Linux_cpuinfo
• It simply prints the target machine cpu details.

17
sudo python vol.py f /tmp/ubuntu.mem
profileLinuxUbuntu140405x64
Networking b. Linux_arp This module simply
prints the ARP table. It can be useful for an
investigator to get some network related
information.
sudo python vol.py f /tmp/ubuntu.mem
profileLinuxUbuntu140405x64 Linux_arp
c. Linux_enumerate_files This plugin simply
enumerate all the files with their path details.
This list of file details can be helpful to find
any suspicious file on the system.
sudo python vol.py f /tmp/ubuntu.mem
profileLinuxUbuntu140405x64 Linux_enumerate_file
s
Now, lets have a look at some past history on
our target machines memory. Process Memory
18
d. Linux_bash This plugin allows us to see
all the Bash commands that were ran prior to
taking our snapshot. This can help an
investigator to understand what the attacker was
doing, which directories he was visiting, what
files he was opening or if he was installing and
malware remotely and so on.
sudo python vol.py f /tmp/ubuntu.mem
profileLinuxUbuntu140405x64 Linux_bash
It would be also a good idea to look under the
hood and see if we can find something
fishy. Rootkit Detection e.
Linux_check_creds This module recognizes
rootkits that have increased the authority to
root level utilizing DKOM procedures. The
obtaining of cred structures prompts an
irregularity that Volatility can use to discover
if any processes were given root level authority.
In the typical workings of the kernel, each
process gets a special cred structure and they
are never shared or acquired. The
Linux_check_creds module uses this by building a
mapping of processes and their cred structures
and after that reports any processes that is
similar. On our memory, we have not got any pid
that has similarity with any of our processes.
That indicates it is rootkit free.
sudo python vol.py f /tmp/ubuntu.mem
profileLinuxUbuntu140405x64 Linux_check_creds
19
f. Linux_check_fop This plugin prints the lists
of the /proc filesystem and all opened files and
confirms that each part of each file_operations
structure is in the right place and not
hidden. From the screenshot below we can see,
volatility couldnt find any hooks placed by
Average Coder. This confirms again that the
system is not compromised with any rootkits.
sudo python vol.py f /tmp/ubuntu.mem
profileLinuxUbuntu140405x64 Linux_check_fop
g. Linux_check_modules This plugin helps to
discover rootkits that break themselves from the
module list but not sysfs. We have never found a
rootkit that really expels itself from sysfs, so
on a live framework they are covered up from
lsmod and /proc/modules, but can still be found
beneath /sys/modules.
sudo python vol.py f /tmp/Ubuntu.mem
profileLinuxUbuntu140405x64 Linux_check_modules
h. Linux_check_syscall This module also helps
to identify if the system is compromised with
rootkits. It lists the system call tables and
checks for hooked functions. For 64-bit OS, it
prints both the 32-bit and 64-bit table. In case
a hooked function, it prints the "HOOKED" on the
output, if not, it just shows the name of the
system call function. In our case, we havent
found any hooked function.
20
sudo python vol.py f /tmp/Ubuntu.mem
profileLinuxUbuntu140405x64 Linux_check_syscall
21

• Rekall
• It a memory forensic and investigation framework
  consists of a set of tool. It is written in
  python and provides complete memory analysis
  solution to all major os including Windows, Linux
  and Mac. Rekall also provides memory acquisition
  tool for each platform.
• Overview of rekall
• Memory acquisition tool Pmem
• MacPmem For most recent OSX.
• WinPmem For windows memory acquisition
• LinPmem For Linux (ordinarily
  utilize/proc/kcore).

• Live examination ability
• Automatically stack driver and access crude
  memory without taking a picture first.
• Used for triaging and gathering a full image
• Focus on flexible live memory investigation.
• Wide collection for memory investigation
  profiles
• AutoDetect the right profile
• Focus on client encounter.
• Automates everything
• Provide an entire workplace with an easy to
  understand interface.

22
Install/Setup Rekall We will need to install
some third party dependencies before we proceed
with the installation of Rekall.
sudo apt-get install python-pip python-dev
libssl-dev libncurses5-dev -y
Lets install and create our virtual world
pip install virtualenv virtualenv /tmp/MyEnv
Activate it
source /tmp/MyEnv/bin/activate
pip install pandas
23
Once we are done with installing all libraries,
we can go ahead and install rekall along with
rekall-agent.
pip install rekall-agent rekall
Install/Setup Linpmem All the memory
acquisition tools provided by Rekall uses unified
AFF4 imager framework to produce all the images
in a similar way. Rekall can utilize all kernel
components straightforwardly when performing live
examination on all supported systems. Rekall
moreover includes a plugin called aff4acquire
which uses this physical memory access to get a
physical memory image, in a comparable way to the
standalone instruments. Memory procurement
through Rekall is able to incorporate extra
information which can only be reduced from live
investigation (e.g. it too captures all mapped
records) but it requires running Rekall (with
bigger footprint and requires access to the
profile repo). As we are on Linux, we are going
to use Linpmem for our memory acquisition.
However, in this demonstration we dont need to
use the snapshot as Rekall also provides live
memory analysis without dumping it first.
24
wget https//github.com/google/rekall/releases/d
ownload/v1.5.1/linpmem-2.1.post4
Lets give it right permission to work without
any issue.
chmod x linpmem-2.1.post4
./linpmem-2.1.post4 -o mem.aff4r
The above command will take the snapshot and save
it as mem.aff4r file on the current directory.

• We can perform our investigation using Rekall in
  multiple ways.
• Trough Rekall API which performs the analysis
  using OS APIs
• Perform analysis on the memory image after
  acquiring it from the target machine.
• Rekall-Live We can erform our analysis directly
  on the physical memory instead of creating an
  image first.
• As we already have seen how to perform analysis
  on a captured memory with volatility, lets use
  Rekall- Live mode to investigate our target
  machines memory.
• For rekall live memory analysis, we dont need to
  create any profile as it will automatically
  detect the kernel version and load the required
  profile to work with.

25

• Rekall has wide range of plugins supporting all
  major operating systems including Linux, windows
  and OSX. It has six major categories of plugins
• Filesystems
• These plugins are focused on filesystems only. It
  has six major plugins supporting various file
  systems including NTFS, MFT,
• General
• These are common plugins that can be used
  regardless of Operating system. There are 29
  plugins in this category to the date. They can be
  used for various purpose like dumping the image,
  exporting, building index, searching with
  specific pid id, session or process details etc.
• Response
• These set of plugins are particularly for
  incident responders. There are total 15 plugins
  under this category. These plugins use the API to
  fetch live system state in a timely fashion. All
  of these plugins can be used with --live Memory
  or the --live API mode
• Linux
• There are total 38 plugins focused on only Linux
  operating systems.
• OSX
• Plugins focused on OSX based operating system.
• Windows
• Plugins focused on windows operating system.
• In our following demonstration we are going to
  show techniques related to Linux , General and
  Response plugins category.

26
Rekall- Live rekall - -live
a. Pslist
It gathers active tasks by using the
task_struct-gttask list. It does not print any
swapper process. If the DTB column is empty, the
item is most likely a kernel thread.
Structured ouput

• describe(pslist)

In general describe describes the output of any
plugin. In this particular case, it is providing
a details view of the pslist.

• pstree

It prints a parent/child relationship tree by
running the task_struct.sibling and
task_struct.children members.
27

• session

• paxview

• arp

It simply provides the arp table details.
28

• banner

It prints the os version details.

• check_afinfo

The plugin distinguishes the area of each
function pointer of diverse network protocol.
If found inside the kernel or a stacked module,
rekall provides details data as well as its
kernel-space address. If malware powerfully
distributes memory and duplicates code there to
handle these functions, the Module column will
show up as Obscure. We can see we have so many
module column that is unknown.
29

• check_creds

This plugin finds out if any process is sharing
credential structures. Credentials are dealt with
inside through the task_struct-gtcred member.
Likely due to apathy or a poor endeavor at
remaining stealth, a few rootkits basically reuse
the cred member of tasks that have the specified
credentials (most frequently ID 0 root). This
plugin reports the area of the cred part of each
task. When this structure is being reused, it
show more than one line of yield with the same
cred address.

• check_proc_fops

It goes through the file operations pointers of
each open file within the proc filesystem. Many
rootkits hook these operations to hide the
process. To detect if any operation pointer is
hooked, rekall checks that the pointer stays
inside the kernel image or any known module. If a
pointer is found outside of these, then it
reports with details.
30

• maps

It provides process maps.
In this section, I am going to use several Linux
based simple tools to analyze our Linux memory
that we obtained with LiME. Cat stands of
concatenate. Its a very useful and widely used
command in Linux environment. It allows the
system admin to view contain of file, concatenate
files, create single or multiple files, and
redirect output in terminal or files. We are
going to use cat command and tell it to find all
the strings in our memory image file and save
everything in a text file named dictionary.
cat Ubuntu.mem strings gt dictionary.txt
If we open the dictionary.txt file, we can see
full of text, some binary and some raw texts.
Here the question is what we are looking for and
how this can help us to analyze the memory. Lets
imagine a scenario where we dont have any clue
about the target machine and we need to response
to the incident live and fast. With this
techniques, we can quickly find out more about
the memory , relevant information os the OS that
it was running on and what it was being used for.
Not only that, this simple
31
technique can help us to find out many sensitive
information like username, passwords , email,
social security number and so on.
vi dictionary.txt
If we look closely at the dictionary.txt file, we
can see there is a theme name ,WP, wp-content,
phpmyadmin and some plugin directory urls. This
indicates that someone was running a wordpress
site on our target machine. Lets try to find
more relevant information.
Let us use cat command again but this time, we
are also going to use grep tool to find a single
word wordpress within our memory and get a
filtered output on the terminal instead of
getting all the text from the memory.
32
cat ubuntu.mem strings grep wordpress
Now this provides us more details and accurate
information including cookie details, server and
site details, the root directory of the website ,
form structure, server users and group.
Let us use the keyword username and see if we
can find anything interesting.
cat ubuntu.mem strings grep username
From the above screenshot above, if we look
closely on the same line as envato_username, we
cann see rabiulis, this could be interesting.
Lets use the keyword rabiul and examine the
output.
cat ubuntu.mem strings grep rabiul
33
It shows rabiul was used as an username at some
point on our target machine but that is not
probably valid. However, it also show us the
email address ri.sumon_at_hotmail.com . These
techniques can help us to build a dictionary of
username and password along with understanding
the memory better at an initial stage. It is
always a good idea to carry out a dictionary
attack rather than bruteforce to save our time
and work efficiently. Now though the above tools
are techniques are very easy to apply and find
out sensitive information quickly but we can not
look for all the files separately ( unless we
divide them manually ) or look for photos and
other formats of files easily. Photorec is one of
the most frequent used tool to recover lost /
deleted files in Linux. As we are on Ubuntu, we
are going to install testdisk library for
photorec. On my machine, it is already installed.
sudo apt-get install testdisk
Now, we will simply run the following command to
start our analysis on the memory.
photorec ubuntu.mem

• Proceed

34
It will show the name and size of our memory to
confirm our selection. We will select proceed
and click enter.

• File Opt

As our memory image is a single image file and
without any partition, it is printing unknown.
Lets click on File opt to review all the
options.
35

• Quit

We will choose all the options to recover as many
file types as possible. We will also select
Quit to return to the previous menu and proceed
to the next step.

• Other

As our memory file isnt a common used file type,
we will select other option to start our
analysis.

• Directory selection

One last thing we need to do is point the
directory where we want to save all of the output
files.
36
It takes some time to recover all the files. We
can see photorec was able to recover 9294 files
and saved them at our pointed directory. We will
select Quit and navigate to the output
directory to review the recovered files.
All the directories with recup prefix are
created by photorec.
Lets open one of them and have a look what it
found for us.
cd recup_dir.1 ls l
We can see it has found many format of files
including .c,.h,.java,.txt,.jpg,.xml Now this
makes our life lot easier and we can easily go
through all of these files , sort them our and
investigate further for sensitive information,
malwares , machine architecture, used
applications, images and so on.
37
Lets a have a peak of one of the random file.
vi f1275108.txt
38
Outcomes/ Results/ Findings
Features Volatility Rekall cat grep Photorec
API Access Live Memory Analysis
Need to create a profile Yes No No No No
Can work without profile No Yes No Yes Yes
Automatic profile detection No x Yes x x
Profile Collection Need to create profile manually Does not need a profile Profile repository holds all the profile- stored as JSON data files. Profile repository currently has 600 Linux profiles 2000 windows profiles 50 OSX profiles. x x
Need to capture the memory Yes Yes No Yes Yes
Kernel version is important Yes No Yes No No
Can analyze the memory for Processes Yes Yes Yes No No
Process Memory Yes Yes Yes No No
Kernel Memory and Objects Yes No Yes No No
Rootkit Detection Yes Yes Yes Yes Yes
Networking Yes Yes Yes No No
System Information Yes Yes Yes Yes Yes
Can do keyword search Yes Yes Yes Yes No
Can recover all files Yes No Yes No Yes
Can recover hidden files Yes No Yes Yes Yes
39
Overall Speed Fast Slow Fast Fast Medium
Automatic Report No No No No No

• None of the memory analysis tool provides details
  automatic report. On windows version many tool
  provides automatic report and neat graphs to give
  an overview understanding about the analysis.
  Linux memory analysis tool has still room to
  grow.
• Both volatility and rekall has a wide range of
  plugins. Both of them can perform memory analysis
  on captured memory but only rekall can perform
  memory analysis on a live running machine without
  capturing the memory first. Also rekall has a
  benefit of using API that allows it to see OS
  related stuffs very easily. All of tools that
  were used during practical work, allowed us to
  learn new techniques on Linux memory analysis
  from scratch, I was able to perform Linux memory
  analysis thoroughly and showed techniques like
  how to
• Capture memory using third party tool
• Create profile according to kernel version
• Analyze the memory for processes (Both active and
  ended)
• Analyze for process memory, kernel memory and
  object.
• Extract information like cpu version, arp tables
• Find previous activity on the target machine
• Analyze to detect rootkit
• Find details path of all the files
• Perform string search to find specific thing on
  raw image
• Recover all the files including hidden ones
• I was able to detect malwares on a machine which
  thought to be a clean machine , recover username,
  password , emails get a clear pic
• On the other hand, neither volatility nor rekall
  can do keyword search or recover the files
  directly, which can be easily achieved by cat ,
  grep and photorec recovery tool.
• Discussion

40
the right one might wont be there as if the
target machine hasnt been updated for a while
and LiME failed to build or if any dependencies
are missing. This can cost valuable time and if
we look at the table on the previous section, we
can see volatility always require a profile to
analyze the memory. Creating a profile can be
time consuming as it also requires prior
knowledge about the target machine and the target
machine needs to have various third party plugins
installed to use volatility to create the
profile. This can not only delay the incident
response but also there is always a possibility
to make changes on the system during acquiring
memory and that is a major concern as it starts
up a process and add new data on the memory. The
same thing happen while performing live memory
analysis without capturing the memory image. This
newly added data could override the original data
which is vital for the investigation. Another
related problem is when an analyst capturing the
data, the new data is also getting written by the
memory and therefore the analyst is actually
mixing both the new and old data together. Like
we saw, after I ran the LiME for the first time
and analyzed that with volatility, no rootkit was
found but later on I performed the memory
analysis live on the same target machine without
any major changes but it was able to quickly
detect rootkits. This clearly points our Rekall
has benefits over volatility but on the other
hand, rekalls live api based plugins are limited
that can run without a profile. However, rekall
also has live memory analysis mode that can auto
detect a profile from its large collection of
profile repository. Even though the techniques
shown above were for demonstration purpose, it is
highly recommended to avoid any case where there
is a chance to make major changes on the target
machine. For the first time it can create a
confusion on the analysts mind and make it
essential for them to differentiate between old
and new data. Analysts should make themselves
familiarize with the footprint of the memory
acquisition tool and eliminate that data quickly
instead of wasting valuable time on finding what
is relevant and what is not when the situation
arise. It is also very important for an analyst
to keep in mind that in no cost, it is possible
to revert back to original state of the memory
once something is been performed on that, even if
someone tries to reboot and reload all the
programs as memory uses different parts of it
every time we start a program, so definitely the
investigation scenario after a crime occurred
wont be identical after someone uses the machine
for other purposes. Volatility has different
mother category for plugins that specifically
point out which plugins are for what purpose .
Like network analysis, rootkit detetction etc but
rekall doesnt have any specific sub category for
all the plugins except filesystem and major OS.
This can make it difficult for an analyst to find
the required plugins in a short time when it
matters the most. One of the major dilemma for an
analyst is to confirm if the evidence what he/she
has collected from the compromised system is
authentic or not. The main reason of this dilemma
is whether to trust the OS or not as there is
always a possibility that an attacker can alter/
erase or insert the data into the memory what he/
she wants an analyst to see or not to see. The
only way to identify is to have a details
understanding about the memory structure and
always use their own tool instead of relying on
publicly available and easily modifiable tools.
Though volatility and rekall are open source but
they are dependable as they have official
repository to get the original code. In the
practical work, I had shown how to extract all
the information from raw memory in binary or text
format and save it in a dictionary.txt file using
cat. I also showed how to recover hidden data and
different file types separately using photorec.
These extraction procedure is fast but the data
is too
41
much that it can be overwhelming for the analyst
and sometime an investigator can completely be
clueless about from where to start his/ her
analysis or verify the findings , if a particular
finding is related to the last occurred crime
scene or it has happened long before in a
different state. It is also very hard to say the
source of the file sometime as an advanced
attacker can always leave some clues in purpose
and mislead an analyst. It is very important for
an investigator to know the difference between
something made up and original clues. All these
above downside doesnt make the tools or
techniques any less significant but they are real
facts that an analyst has to face in everyday
crime scene. This discussion section provided a
clear overview about the limitations of the tools
and techniques to create an awareness for
analysts so they can be prepared to face any of
the situation mentioned above. Conclusion All
of us admit live memory forensic on Linux is
something not perfect and easy to perform.
Analysts are developing new tools and techniques
often to make the investigation procedure much
smoother as the new technology is emerging. The
need of memory forensic has risen to a very high
level and it is not possible to discredit that
anyhow as attackers are finding their ways to
exploit the revolving technology always. There
are still many works that can be done on live
memory forensic. In this paper I discussed the
basic memory structure and importance of memory
forensic. Some major Linux memory analysis
focused work was reviewed. Some leading tools
were used in practical work to show most common
and required techniques in an incident response.
Different methods were introduced for live memory
analysis, a details procedure and methodology was
developed for the convenience of analysts. A
comparison was made between different tools and
their features, strength and weakness were
identified and listed in a table. I also did a
brief discussion about the details findings,
limitation of different approach of various tools
and what to expect in a real life scenario and
how to overcome those challenges to perform a
safe investigation. Many researchers and analysts
believer attackers are focusing toward memory
more than ever as they can easily execute
malicious code there without getting traced back
and volatile memory also a safe choice to prevent
analysts from performing any reverse engineering
and explore sensitive information about the
attack method performed by the malicious
attacker. Many analyst dont focus on memory
analysis or used the dead box traditional
approach. The analysts must also adapt to the new
techniques and tools related to live memory
forensic as the memory focused attack are on
rise. This paper explores various techniques with
existing tools to encourage analysts to focus
more on live memory analysis and new comers to
view the forensic analysis from a new perspective
in the field of cybercrime and forensic
investigation.
42

• References Literature Reivew
• Automatic profile generation for live Linux
  Memory analysis
• Dynamic recreation of kernel data structures for
  live forensics
• L. Wang, R. Zhang., and S. Zhang, A model of
  computer live forensics based on physical memory
  analysis, In IEEE Information Science and
  Engineering (ICISE), 2009 1st International
  Conference on, (2009), December, pp. 4647-4649.
• A. Aljaedi., D. Lindskog, P. Zavarsky, R. Ruhl,
  and F. Almari,, Comparative Analysis of Volatile
  Memory Forensics Live Response vs. Memory
  Imaging, In IEEE Privacy, security, risk and
  trust (passat), 2011 ieee third international
  conference on and 2011 ieee
• S. Balogh, and M. Pondelik, Capturing encryption
  keys for digital analysis, In IEEE Intelligent
  Data Acquisition and Advanced Computing Systems
  (IDAACS), 2011 IEEE 6th International Conference
  on, vol. 2, (2011), September, pp. 759-763.
• Forensic Analysis of a Live Linux System, Pt. 1
  and 2
• Rekall plugins http//www.rekall-forensic.com/doc
  umentation-1/rekall-documentation/plugins
• Rekall memory analysis framework
  https//holdmybeersecurity.com/2017/07/29/rekall-
  memory-analysis-framework-for-windows-Linux-and-ma
  c-osx/
• Rekall memory forensic cheatsheet
  https//digital-forensics.sans.org/media/rekall-me
  mory- forensics-cheatsheet.pdf
• Scalabel memory analysis using Rekall and GRR
  https//www.dfrws.org/sites/default/files/session-
  files/pres_using_grr_and_rekall_for_scalable_memo
  ry_analysis.pdf
• Linux memory dump with rekall https//isc.sans.ed
  u/diary/LinuxMemoryDumpwithRekall/17775
• Memory image forensic analysis using Volatility
  tool in kali Linux https//singhgurjot.wordpress.
  com/2015/09/02/memory-image-forensic-analysis-usin
  g- volatility-tool-in-kali-Linux/
• Volatility guideline https//github.com/volatilit
  yfoundation/volatility/wiki/Linuxacquiring-
  memory
• Linux memory analysis with volatility
  http//dfir.org/research/omfw.pdf
• Volatility description https//tools.kali.org/fo
  rensics/volatility
• Volatility Linux command reference
  https//github.com/volatilityfoundation/volatility
  /wiki/Linux-Command- ReferenceLinux_check_syscall
  
• Methodology of memory analysis and optimization
  in embedded system https//www.esat.kuleuven.be/c
  osic/publications/article-683.pdf
• Techniques and tools for recovering and analyzing
  data from volatile memory