It won't happen very often when you will find yourself holding in your hand a hard drive that belonged to an ESXi server. These servers usually house production machines that just don't get shutdown very often. Why the decision has been made to turn it off is one that I am sure was not made lightly. Whatever the scenario is, it is what it is. It wasn't your call, but the client decided to shut down their ESXi server and subsequently shipped it to you for analysis. Now you have the drive in your hand and you have been tasked with extracting the Virtual Machines out of the drive for analysis.
The underlying file system of an ESXi server is the VMFS file system. It stands for Virtual Machine File System. VMFS is Vmware, Inc.'s clustered file system used by the company's flagship server visualization suite, vSphere. It was developed to store virtual machine disk images, including snapshots. Multiple servers can read/write the same file system simultaneously while individual virtual machine files are locked (Source Wikipedia).
As of the date of this writing, not all of the big forensic suites have the ability to read this file system. And I can understand why, as is extremely difficult for the commercial suites to offer support for all available file systems. Fortunately for us, it is very possible to read this file system using Linux.
The purpose of this article is to go over the steps required to mount the VMFS file system of the drive from an ESXi server. Once access to the file system has been accomplished, we will acquire a Virtual Machine stored on the drive.
Installing the Tools:
For you to be able to accomplish the task, you will have to make sure that you have vmfs-tools installed on your Linux examination machine. You can get the package from the repositories by running $ sudo apt-get install vmfs-tools. Vmfs-tools is included by default in LosBuntu. LosBuntu is our own Ubuntu 14.04 distribution that can be downloaded here. If you download and boot your machine with LosBuntu, you will be able to follow along and have the exact same environment described in this write-up.
To illustrate the steps of mounting the partition containing the VMFS file system on the drive, I will use a 2TB hard drive with ESXi 6.0 installed on it. This drive is from an ESXi server that I own. The ESXi server drive is currently housing some virtual machines that we will be able to see, once the file system is mounted. I booted an examination machine with a live version on LosBuntu and connected the drive to the machine. LosBuntu’s default behavior is to never auto-mount drives.
Now, fire up the terminal and let's begin the first step of identifying the drive. Usually the first step involves running fdisk, so that we can identify which physical assignment was given to the drive. Running $ sudo fdisk –l lists the physical drives attached to the system, the flag -l tells fdisk to list the partition table. Sudo gives fdisk superuser privileges for the operations. Press enter and type the root password (if needed, pw is "mtk").
$ sudo fdisk -l
Not show on the screen is /dev/sda, which is my first internal drive, therefore /dev/sdb should the drive of the ESXi server. The output of fdisk give us a warning that /dev/sdb may have been partitioned with GPT and fdisk was unable to read the partition table. Fdisk is telling us to use parted, so let’s do that. The following parted command will hopefully get us closer to what we need.
$ sudo parted /dev/sdb print
From the output, we can see that yes, it is indeed a GPT partitioned drive, containing multiple partitions. The last displayed partition, which is actually partition number three, looks to be the largest partition of them all. Although parted was able to read the partition table, it was unable to identify the file system contained in partition three. We currently have a strong suspicion that /dev/sdb is our target drive containing our target partition, but it would be nice to have confirmation. Let's run one more command.
$ sudo blkid -s TYPE /dev/sdb*
Blkid is a command that has the ability to print or display block device attributes. The flag -s TYPE will print the file system type of the partitions contained in /dev/sdb. We used an asterisk “*” after sdb so that blkid can show us the file system types of all partitions located in physical device sdb like sdb1, sdb2, sdb3 and so on.
Finally, we can now see that /dev/sdb3 is the partition that contains the VMFS volume.
To mount the file system we are going to have to call upon vmfs-fuse, which is one of the commands contained within the vmfs-tools package built into LosBuntu. But before we call upon vmfs-fuse, we need to create a directory to mount the VMFS volume. Type $ sudo mkdir /mnt/vmfs to create our mount point.
Mount the VMFS file system contained in /dev/sdb3 to /mnt/vmfs with the below command
$ sudo vmfs-fuse /dev/sdb3 /mnt/vmfs/
As you can see, the execution of the command simply gave us our prompt back. As my friend Gene says. “You will not get a pat on the back telling you that you ran your command correctly or that it ran successfully, so we need to go check.” True and amusing at the same time…
Check the contents of /mnt/vmfs by first elevating our privileges to root, with $ sudo su and then by listing its contents with # ls -l /mnt/vmfs.
Great! We can read the volume and we see that we have many directories belonging to Virtual Machines. From here you can remain in the terminal and navigate to any of these directories, or you can fire up nautilus and have a GUI to navigate. The following command will open nautilus at the location of your mount point as root. It is important to open nautilus as root so that your GUI can have the necessary permissions to navigate the vmfs mount point that was created by root.
# nautilus /mnt/vmfs
Insert another drive to your examination machine and copy out any of the Virtual Machines that are in scope.
Another option would be to make a forensic image of the Virtual Machine. For example, we can navigate to the Server2008R2DC01 directory, which houses the Domain Controller used on the previous write-up about examining Security logs. Find that article here.
In this specific instance, this Virtual Machine does not contain snapshots. This means that the Server2008R2DC01-flat.vmdk is the only virtual disk in this directory responsible for storing the data on disk about this server. If the opposite were true, you would have to collect all of the delta-snapshot.vmdk files to put back together at a later time.
The Server2008R2DC01-flat.vmdk file is a raw representation of the disk. It is not compressed and can be read and mounted directly. The partition table can be read with the sleuthkit tool mmls. Mmls is a tool that can display the partition layout of volumes. Type the following into the terminal and press enter. The flag -a is to show allocated volumes, and the flag -B is to include a column with the partition sizes in bytes.
# mmls -aB Server2008R2DC01-flat.vmdk
You can see that the 50GB NTFS file system starts at sector offset 206848.
If you want to acquire this virtual disk in E01 format, add the flat-vmdk file to Guymager as a special device and acquire it to another drive.
And there you have it!
Using free and open source tools you have been able to mount and acquire images of Virtual Machines contained in the file system of a drive belonging to an ESXi server. If this procedure helped your investigation, we would like to hear from you. You can leave a comment or reach me on twitter: @carlos_cajigas
This was great information! As you said, it doesn't happen often that you will get the drive from an ESXi system, but when you do this is way helpful!ReplyDelete
Bryan, Thanks for coming by. Glad it may be helpful. CarlosDelete
This write up was perfect. Provided an easy no cost solution so I could then image the OS's inside EXSI and analyze with my forensic software. Thanks for saving me from closing my exam as a 'cold case'.ReplyDelete
Thanks and great to hear. This is the reason why we write this stuff up!!!Delete
Very helpful. I've messed my partitions by connecting my ESXi to a windows machine (just to have a look a it :)). When I installed the HDD back to my Gen8 server, Esxi 6.5 wouldn't boot anymore. I wanted to backup my VMs prior trying to fix the issue. Thanks!ReplyDelete
Glad this helped!!! Come back soon!Delete
My server has been updated to 6.5 Update 1. I installed vmfs-tools in a Ubuntu VM, and fdisk shows my drive as /dev/sda1, however, parted can't see anything else, I tried, sdb, sdc, etc. I'm missing something ?ReplyDelete
What abour vmfs version 6? vmfs-fuse is unable to open it and produces just "VMFS: Unsupported version 6"ReplyDelete
Can you clone the drive and put it back on the original system and SSH into it? That is a workaround.Delete
Carlos, I have seven servers with ESXi to process. One server alone has 32 TB of VMFS data storage. Without your blog (and the assistance of your little buddy Gene) I would have returned this to the client without ever identifying what data they contained. Thanks for sharing!!! Brent.ReplyDelete
I would like to say "Thank you so much" this article save me....ReplyDelete
The OS and the steps are easy to follow, especially for someone who has very little experience in linux like myself.
after entering: sudo vmfs-fuse /dev/sdc1 /mnt/vmfs/ReplyDelete
I get the error: *Error stat()ing '/mnt/vmfs/'
I just followed the directions for another partion (sdb3) and am now attempting to do it again but now its giving me this error, any advice?