ESXi is an enterprise level computer
virtualization product offered by VMWare, the makers of VMWare Player and
Workstation. ESXi can be used to
facilitate the centralized management of many different types of Windows, Unix,
and Linux systems. Unlike its cousins
Player and Workstation, ESXi runs directly on server hardware (bare metal)
without the need for an underlying operating system. Management of the ESXi server can and is
often done remotely.
As this type of virtualized server
environment continues to gain even more popularity, you are bound to encounter
it more and more. It seems that we see
one of these in just about every case.
If you are in the DFIR world then you know that it is our responsibility
to get usable images out of these things.
In this article we will go through
the process of acquiring a Linux Virtual Machine (VM) running on an ESXi
server. Since there is more than one way
of accomplishing this, we will talk about some of the options that are
available to us. We will talk about the
pros and cons of each so that you get to choose one when your time comes to
implement it.
Tools:
For the purposes of this article,
all of the systems except for the ESXi server, were installed and ran using VMware
Player. Our host machine was an
examination computer with Ubuntu 13.10 installed to the hard drive. Except for the Windows VM, all of the tools
used in the article are free and are available for download.
The test:
The first thing we need to do is get
ESXi up and running. VMware offers a
free version of their EXSi product that can be downloaded here. To get the download, you are going to have to
create and account with them. Once you
have the ISO, go ahead and install it.
There are many tutorials online on how to do this. I don't want to go over the steps of
installing ESXi here due to the many resources available and how much longer it
would make the write up. We went ahead
and installed ESXi version 5.5 on an old laptop with a core 2 duo processor and
4GB of memory. It worked fine. ESXi was installed using all of the
defaults. Upon startup the ESXi server
was given an IP via DHCP. It was
assigned IP address 10.0.1.11
ESXi servers can be managed locally from the
command line or remotely from another host using the VMware vSphere Client,
which as of this writing only works on Windows hosts. Using another VMware Player VM with Windows
7, we installed the vSphere client and logged in to the ESXi server.
We went ahead and installed one VM
on the ESXi server. The VM on the server
is a fresh install of Ubuntu 14.04. We
named the VM, Ubuntu14_04. This will be
the VM that we will we use to go through the process of acquisition.
Now that we have an ESXi server up
and running and a VM to acquire, we need to determine what our best course of
action for acquisition is. This will be
decided by your conversation with the system administrator (sysadmin) and the
company's ability to be able to take the “server” off-line. In most situations these are production
environment servers that need to stay up to not disrupt the business. If you are lucky and find that the sysadmin
is willing to take the server off-line, then suspending the VM is probably the
better method of acquiring a VM out of the server. We will get to do this just a little later.
For now, let's assume that the
server is a money making server that cannot be taken off-line. When encountered with this issue we have
conducted successful acquisitions by using a combination of SSH and an
externally available source to send the data to. For example, you can SSH into the server and use
the linux native tool dd, to acquire the virtual disks and send the acquisition
image to a mounted NFS share.
Let's simulate this scenario by
using our previously created Ubuntu14_04 VM.
Notice that our last screenshot tells us that our Ubuntu VM was assigned
IP address 10.0.1.12. The command $ ssh
carlos@10.0.1.12 can be used to SSH into the VM. If you are following along, make sure that
your VM has an SSH server installed.
Once the key pairing has been created and the password has been entered
you will be given a command prompt just as if you were on a shell terminal on
the system. Notice that my prompt
changed to carlos@esxivm:~$ which is an indication that I am now inside of the
VM.
The next step is to mount the NFS
share that we will use to receive our acquisition image. To simulate this part, we used another VM and
the free version of the popular open source NAS solution called FreeNAS. The FreeNAS VM was assigned IP address
10.0.1.13.
Run
the below command to mount the NFS share to the Ubuntu VM. If you are following along make sure that
your Ubuntu VM has nfs-common installed.
$ sudo mount -t nfs -o proto=tcp,port=2049 10.0.1.13:/mnt/vol1
/mnt/nfs/
Mount
is the command to mount the share, -t tells mount which type of mount that it
is about to do, -o are the options that are passed to mount. Proto= tells mount to use tcp as a protocol,
this is done to ensure compatibility with other operating systems. Port=2049 tells it which port to use. 10.1.1.13:/mnt/vol1 is the IP of the FreeNAS
VM and the location of the share on the NAS.
/Mnt/nfs is a previously created mount point used to mount the NFS.
We
are now ready to send the acquisition to the NFS using dd. We navigated to the share and kicked it off.
$ sudo dd if=/dev/sda of=esxivm.dd bs=4096 conv=sync,noerror
The
acquisition successfully completed. Mine
took a while as I piped the data wirelessly rather than through the wire. Speeds at your client location will vary
depending on bandwidth. In reference to
image verification, since this was a live acquisition, doing a hash comparison
at this point will not be of much value.
At the very least, check and compare that the size of your dd image
matches the amount of bytes contained in /dev/sda. This can be accomplished by comparing the
output of fdisk -l /dev/sda against the size of the dd.
Great,
so now we have an image of our server sitting on a NAS and we need to bring it
back to our system. Well you have
options. If the image is too large to
pull it back over the wire, you can have the sysadmin locate another system close
to the NAS that has access to the NAS and also has USB 3.0. This can even be a Windows system. If this is not an option for you, and you
have no choice but to pull it back over the network, use your favorite ftp or
sfpt tool to access the NAS and bring the image back to your system. For the purposes of this article we used
FileZilla.
And
there you have it. You have successfully
acquired a live image out of a VM running on an ESXi server.
Now
let’s take some time and talk about another way of doing things. If you are lucky and the sysadmin said he is
good with taking the server off-line, then you are in luck and things just got
easier for you. Suspending the VM is also
a good way of collecting an image of that VM.
When you suspend a VM on ESXi, the current state of the VM is saved so
that it can later be loaded and reused.
Additionally, the VM's memory gets written to a .vmss file that contains
the entire memory allotted to the VM.
This, in essence, can be considered a simulated way of acquiring memory
from the VM. The point is that with one
click of a button you get the VM's memory and a current state of the VM's
virtual disk. The vmdk file will contain
the entire raw virtual disk.
Once
you have suspended the VM, the process of getting the data out of the server
can be accomplished just like in our first scenario. So let us first talk about the method which
involves the combination of SSH and the NFS share.
If
the sysadmin allowed you to take their server off-line to suspend it, then he
may also be willing to give you the password so that you can SSH into the ESXi
server.
Once
you have SSH'ed into the sever, locate the VM directory by navigating to
/vmfs/volumes/datastore1. There, we located the directory for our
Ubuntu14_04 VM.
The
Ubuntu14_04 directory contained the following files.
Ubuntu14_04-167f1682.vmss Is the memory file
Ubuntu14_04-flat.vmdk Is the VM disk
Those
are the main files that we need to get out of the server. If you had to go the mounted share route,
then the process of mounting the NFS goes like this.
# esxcfg-nas -a -o 10.0.1.13 -s /mnt/vol1 nas
Esxcfg-nas
is the ESXi command to mount the share, -a is to add a share, -o is the address
of the host and -s /mnt/vol1 is the share directory to mount, finally nas is
the name that we are going to assign to the mounted NFS share. Be aware that unlike in Ubuntu, you do not
have to create a mount point, the NFS share will be created under
/vmfs/volumes.
Once
the NFS is mounted you can use the cp command to copy your files over to the
NFS. Upon completion, it would be a good
idea that you implement some way of validating that your data is a true
representation of the original. Hashing
the files on the server and comparing them to the ones on the NFS could be a
way.
To get
the images out of the NAS, again, use your favorite ftp or sftp tool to
retrieve them. It should be noted that
we went over the process of mounting an NFS on ESXi, so that you could have
that as an option. If SSH is enabled on
the EXSi server don't forget that you can access it directly with scp or sftp
and drag the images back to your system.
FastSCP on the windows side also works well.
To
disconnect the NFS from the server run $ esxcfg-nas -d nas.
Conclusion:
There
are many available options of acquiring images out of these servers. Discussed above were only a couple of these
options. If this procedure helped your
investigation, we would like to hear from you.
You can leave a comment or reach me on twitter: @carlos_cajigas
