Protecting ESX VMFS Stores with Automation

Some time ago I shared some interesting information about VMFS volumes that I found using direct analysis in my blog named Understanding VMFS volumes. This spawned some discussions on the VMware Community forums and it became apparent that an automated backup of the critical VMFS info could be useful in the event of an undesirable security event that impacts our system availability. By creating a simple backup script process we can provide the ability to recover much more quickly from such events. In this howto guide we will enable this process with a cron job using the existing /etc/cron.daily/ job location directory. We simply need to copy an automation script to this location and it will run daily. Or if your change rate is less frequent maybe the /etc/cron.weekly location is more suitable.    

I have provided a simple script named vmfs-bu linked here which creates VMFS header file backups using dd for all the VMFS volumes visible on an ESX host. The script will output it’s backups to the /var/log directory. Here is a sample of what the backups look like from one my lab hosts.

-rw-r–r–    1 root     root      2097152 Mar 28 21:01 vmfs-header-backup-sda.hex
-rw-r–r–    1 root     root      2097152 Mar 28 21:01 vmfs-header-backup-sdb.hex
-rw-r–r–    1 root     root      2097152 Mar 28 21:01 vmfs-header-backup-sdc.hex
-rw-r–r–    1 root     root      2097152 Mar 28 21:01 vmfs-header-backup-sdd.hex
-r——–    1 root     root      8388608 Mar 28 21:01 vmfs-metadata-sda-vh.sf.bu
-r——–    1 root     root      8388608 Mar 28 21:01 vmfs-metadata-sdb-vh.sf.bu
-r——–    1 root     root      8388608 Mar 28 21:01 vmfs-metadata-sdc-vh.sf.bu
-r——–    1 root     root      8388608 Mar 28 21:01 vmfs-metadata-sdd-vh.sf.bu

Also be sure that the script permissions are configured as executable using the follwing.

chmod 755 /etc/cron.daily/vmfs-bu

With these simple easy steps we can create a proactive failsafe for VMFS recovery in the event of those undesirable moments we hope will never occur.

In addition to this basic method we can extend it’s functionally to recovery from undesirable VM deletions by automating the experimental vmfs-undelete tool. Before I demonstrate this functionally I must advise you that this tool may have some negative impacts to certain configurations. The vmfs-undelete tool is a set of python scripts that basically read vmks block allocations and outputs them to an archive file. In order to read the vmdks the scripts will invoke VM snapshots and this will not work if a current snapshot exists. This is not a big issue since you should not be leaving any snapshots around anyway as a best practice. However this action can negatively impact VCB or other backup agents that require snapshoting to grant vmdk access, Thus if you do not carefully time the script invocation when we have other snapshot dependant activities it could cause failures of those functions.

With this important information considered we can proceed to look at how to invoke the additional protection feature. The scripts that were originally developed by VMware are designed to be user interactive and cannot be used as originally coded therefore I have modified them in order to provision some basic automation. You can access the modified scripts named vmfs-undelete-auto-script and here.

The modified script needs to be placed within the /usr/lib/vmware/python2.2/site-packages/vmware/undeletemods directory. While I could have just modified the existing script it is subject to change so this method prevents potential conflict issues. The vmfs-undelete-auto-script script location is optional and can be placed where ever you find apropriate. I chose to place it in the /root directory. The script requires a single argument which is to direct it output location with a path and file name. Since there is a potention for conflict with other snapshot based services the script should be invoked using a cron job outside of the daily or other predefined jobs. This cron job can be implemented using the crontab facility. Here is an example of how to create it while logged in as root.

crontab -e

This command will create a cron entry in /var/spool/cron/root and will load vi as a editor you will need to create the following cron details to invoke the vmfs-undelete script. You can find more info on cron editing at

05 22 * * * python /root/vmfs-undelete-auto-script /var/log/vmfs-undelete-archive

This entry will execute at 10:05PM every day. Again be sure to set the security of the script file as executable as shown here.

chmod 755 /root/vmfs-undelete-auto-script

When the script executes it can take several minutes to complete.

Hope you found this information to be useful.




Site Contents: © 2009  Mike La Spina

Automated VMware Tools upgrades using your VC and Perl

I was upgrading a VI 2.0.2/3.0.2 system to VI 2.5/3.5 and found that upgrading VMware Tools was becoming a real pain. So I decided to look for a better way. I first did the usual search hoping to find a quickie but no luck. I searched the VMware forums to find that everyone was using the MSI package in a manner external to the VI system. This did not address any Linux or Unix systems so here is what I came up with to address the lack of automation.

First I installed the VMware VI perl Tool Kit on the VC.

I changed the NTFS rights to the Tool Kit directory to an dedicated admin user ID and SYSTEM only, I then populated the visdk.rc config file in the same directory with the following parms


I also added the following ENV variable to the SYSTEM Context.

System variable ss

I wrote a short simple Perl script to make use of the power of the VC and have it take care of the task using it’s own built capability. Frankly I just don’t see why VMware did not have a scheduled task item to handle this job in the VC.  

#!/usr/bin/perl -w
# Requires a text file named VMToolsUpgradeList.txt
# containing VM names which the last of must not end with a CR
# Author: Mike La Spina
#   Date: May 24, 2008
# –host TheVC

use strict;
use warnings;
use VMware::VIRuntime;

my %opts = (
   ‘host’ => {
      type => “=s”,
      help => “Host is the VIM API service target”,
      required => 0,

my ($property_name, $property_value);
   $property_name = “”;
   $property_value = ”;

# get all VMs with out of date tools
my $vm_views = Vim::find_entity_views(view_type => ‘VirtualMachine’,
                  filter => { ‘guest.toolsStatus’ => ‘toolsOld’ });

foreach (@$vm_views) {
      my $nameVM = $_->name;
      #print $nameVM . “n”;
      open FILE, “<”, “VMToolsUpgradeList.txt” or die $!;
      my @lines = <FILE>;
      foreach my $vmname (@lines) {
         if ($nameVM eq $vmname) {
            print “Issued VMware Tools upgrade task on ” . $nameVM . “n”;
   next ;
print “Tools Upgrade scan completedn”;

The script requires a text file which I populated all of the VM names that I wanted to upgrade VMware Tools on. It’s noted in the script.

I created a windows task to run the perl script at 3:30AM and set it up to login in with the dedicated adminID and password.

I can now add any VM names to the VMToolsUpgradeList.txt and schedule the event to run after hours.

There are no issues with MSI deployements or Linux installs etc.

Till next time…


Site Contents: © 2008  Mike La Spina