Protecting ESX VMFS Stores with Automation

Some time ago I shared some interesting information about VMFS volumes that I found using direct analysis in my blog named Understanding VMFS volumes. This spawned some discussions on the VMware Community forums and it became apparent that an automated backup of the critical VMFS info could be useful in the event of an undesirable security event that impacts our system availability. By creating a simple backup script process we can provide the ability to recover much more quickly from such events. In this howto guide we will enable this process with a cron job using the existing /etc/cron.daily/ job location directory. We simply need to copy an automation script to this location and it will run daily. Or if your change rate is less frequent maybe the /etc/cron.weekly location is more suitable.    

I have provided a simple script named vmfs-bu linked here which creates VMFS header file backups using dd for all the VMFS volumes visible on an ESX host. The script will output it’s backups to the /var/log directory. Here is a sample of what the backups look like from one my lab hosts.

-rw-r–r–    1 root     root      2097152 Mar 28 21:01 vmfs-header-backup-sda.hex
-rw-r–r–    1 root     root      2097152 Mar 28 21:01 vmfs-header-backup-sdb.hex
-rw-r–r–    1 root     root      2097152 Mar 28 21:01 vmfs-header-backup-sdc.hex
-rw-r–r–    1 root     root      2097152 Mar 28 21:01 vmfs-header-backup-sdd.hex
-r——–    1 root     root      8388608 Mar 28 21:01 vmfs-metadata-sda-vh.sf.bu
-r——–    1 root     root      8388608 Mar 28 21:01 vmfs-metadata-sdb-vh.sf.bu
-r——–    1 root     root      8388608 Mar 28 21:01 vmfs-metadata-sdc-vh.sf.bu
-r——–    1 root     root      8388608 Mar 28 21:01 vmfs-metadata-sdd-vh.sf.bu

Also be sure that the script permissions are configured as executable using the follwing.

chmod 755 /etc/cron.daily/vmfs-bu

With these simple easy steps we can create a proactive failsafe for VMFS recovery in the event of those undesirable moments we hope will never occur.

In addition to this basic method we can extend it’s functionally to recovery from undesirable VM deletions by automating the experimental vmfs-undelete tool. Before I demonstrate this functionally I must advise you that this tool may have some negative impacts to certain configurations. The vmfs-undelete tool is a set of python scripts that basically read vmks block allocations and outputs them to an archive file. In order to read the vmdks the scripts will invoke VM snapshots and this will not work if a current snapshot exists. This is not a big issue since you should not be leaving any snapshots around anyway as a best practice. However this action can negatively impact VCB or other backup agents that require snapshoting to grant vmdk access, Thus if you do not carefully time the script invocation when we have other snapshot dependant activities it could cause failures of those functions.

With this important information considered we can proceed to look at how to invoke the additional protection feature. The scripts that were originally developed by VMware are designed to be user interactive and cannot be used as originally coded therefore I have modified them in order to provision some basic automation. You can access the modified scripts named vmfs-undelete-auto-script and here.

The modified script needs to be placed within the /usr/lib/vmware/python2.2/site-packages/vmware/undeletemods directory. While I could have just modified the existing script it is subject to change so this method prevents potential conflict issues. The vmfs-undelete-auto-script script location is optional and can be placed where ever you find apropriate. I chose to place it in the /root directory. The script requires a single argument which is to direct it output location with a path and file name. Since there is a potention for conflict with other snapshot based services the script should be invoked using a cron job outside of the daily or other predefined jobs. This cron job can be implemented using the crontab facility. Here is an example of how to create it while logged in as root.

crontab -e

This command will create a cron entry in /var/spool/cron/root and will load vi as a editor you will need to create the following cron details to invoke the vmfs-undelete script. You can find more info on cron editing at

05 22 * * * python /root/vmfs-undelete-auto-script /var/log/vmfs-undelete-archive

This entry will execute at 10:05PM every day. Again be sure to set the security of the script file as executable as shown here.

chmod 755 /root/vmfs-undelete-auto-script

When the script executes it can take several minutes to complete.

Hope you found this information to be useful.



Site Contents: © 2009  Mike La Spina

Awarded VMware vExpert Status

While I was away at the OpenSolaris Storage Summit in San Francisco a really nice email appeared in my inbox which put a grateful smile on my face. John Troyer and his team felt that I was worthy of the vExperts award. I’m honoured to receive the recognition and thank-you to the judging panel members 

Congrats to all the other recipients as well.

Site Contents: © 2009  Mike La Spina

Multi Protocol Storage Provisioning with COMSTAR

 COMSTAR is a new breed of open source storage product available to the world. What was traditionally a closed and proprietary storage capability is now available to our open source communities. With OpenSolaris and COMSTAR the ability to freely provision virtual storage services over very mature high end protocols on standard commodity server hardware is now a reality. High performance transports are integral within the feature sets of COMSTAR and Sun’s open source portfolio of projects. The COMSTAR product is revolutionary in its method of provisioning storage virtualization and transport services to storage resource consumers.
COMSTAR provisions virtualized SCSI block storage over multiple SCSI transport protocols. While this function class is not new to us the ease of implementation using COMSTAR certainly is. All the complexities of using a multi protocol target services platform are cleaned up. It is simple to use and facilitates advanced high performance storage provisioning at the block level.
The services within this product have multiple common storage provisioning applications. One very interesting application is a storage gateway server and this blog demonstrates howto build a Fiber Channel (FC) storage gateway using the COMSTAR service layers and as well provision some additional features using the target services.


 COMSTAR FC Gateway Architecture by Mike La Spina

In this example instance we are re-provisioning an existing storage system with an OpenSolaris COMSTAR configuration running on a commodity white box which functions as a storage server head that can compress, scrub, thin provision, replicate, snapshot and clone the existing block storage attachments. The example FC based storage could also be comprised of a JBOD FC array directly attached to the OpenSolaris storage head if we so desired or many other commonly available SCSI attachment methods. The objective here is to extend and enhance any block storage system with high performance transports and virtualization features. Of course we could also formalize the white box to an industrial strength host once we are satisfied that the proof of concept is mature and optimal.  

The reality is that many older existing FC storage systems are installed without these features primarily due to the excessive licensing costs of them. And even when these features are available, its use is probably restricted to like proprietary systems thus obsolescing the entire lot of any useful future functionality. But what if you could re-purpose an older storage system to act as a DR store or backup cache system or maybe a test and development environment. With today’s economy this is from a cost perspective, very attractive and can be accomplished with very little risk on the investment side.

One of the possible applications for this flexible storage service is the re-provisioning of existing LUN’s from an existing system to newer more flexible SCSI transport protocols. This is particularly useful when we need to re-target the existing storage system from FC to iSCSI or the likes of. We can begin by exploring this functionality and explain how COMSTAR can provide us with this service.

First we need to understand the high level functionality of the COMSTAR service layers. Virtual LUN’s on COMSTAR are provisioned with a service layer named the LU provider. This layer maps backing stores of various types to a storage GUID assignment and additionally defines other properties like the LUN ID and size dimensions. This layer allows us to carve out the available block storage devices that are accessible on our OpenSolaris storage host. For example if we attached an FC Initiator to an external storage system we can then map the accessible SCSI block devices to the LU provider layer and then present this virtualized LUN to the other COMSTAR service layers for further processing.

Once we have defined the LU’s we can present this storage resource to the SCSI Target Mode Framework Service (STMF) layer which acts as the storage gate keeper. At this layer we define which clients (initiators) can connect to the LU’s based on Membership of Target Groups and Host Groups that are assigned logical views of the LU(s). The STMF layer routes the defined LU(s) as SCSI targets over a multiprotocol interface connection pool to a Port Provider. Port Providers are the protocol connection service instances which can be the likes of FC, iSCSI, SAS, iSER, FCoE and so on. 

With these COMSTAR basics in mind let us begin by diving into some of the details of how this can be applied.  

Sun has detailed howto setup COMSTAR at so no need to re-invent the wheel here.

Just as a note SXCE snv_103 and up integrate the COMSTAR FC and iSCSI port provider code. With the COMSTAR software components and FC target setup we can demonstrate the re-provisioning of an existing FC based storage server. Since I don’t have the luxury of having a proprietary storage server at home I will emulate this storage using an additional COMSTAR white box to act as the FC storage target to be re-provisioned.

On the existing FC target system we need to create Raid0 arrays of three disks each which will total up to a set of six trios. We will use these six non-fault tolerant disk groups as vdevs for a ZFS raidz2 group. This will allow us to create fault tolerant arrays from the existing storage server. The reasons for sets of three Raid0 groupings are to reduce the possibility of reaching the LUN maximums of the proprietary storage system and also we do not want to erode the performance by layering Raid 5 groups. As well we can tolerate a disk failure in two of the trios since we have Raidz2 across the Raid0 trio groups. Additionally using these Raid0 disk groups actually lowers the array failure probability rate. For example if a second disk were to failure in a single Raid0 set there would be no additional impact to other trios, thus reducing the overall failure rate. 

To create the emulated FC storage system I have defined the following 16G ZFS sparse volumes respectively named trio1 through trio6 each as a representation of the 3 disk Raid0 spanned LUN on a source storage host named ss1. 

root@ss1:~# zfs create sp1/gw
root@ss1:~# zfs create -s -V 16G sp1/gw/trio1
root@ss1:~# zfs create -s -V 16G sp1/gw/trio2
root@ss1:~# zfs create -s -V 16G sp1/gw/trio3
root@ss1:~# zfs create -s -V 16G sp1/gw/trio4
root@ss1:~# zfs create -s -V 16G sp1/gw/trio5
root@ss1:~# zfs create -s -V 16G sp1/gw/trio6

Once these mockup volumes are created they are then defined as backing stores using the sbdadm utility as follows.

root@ss1:~# sbdadm create-lu /dev/zvol/rdsk/sp1/gw/trio1

Created the following LU:

              GUID                    DATA SIZE           SOURCE
——————————–  ——————-  —————-
600144f01eb3862c0000494b55cd0001      17179803648      /dev/zvol/rdsk/sp1/gw/trio1

All the backing stores were added to the LU provider service layer to which in turn were assigned to the STMF service layer. Here we can see the automatically generated GUID’s that are assigned to the ZFS backing stores.

root@ss1:~# sbdadm list-lu

Found 6 LU(s)

              GUID                    DATA SIZE           SOURCE
——————————–  ——————-  —————-
600144f01eb3862c0000494b56000006      17179803648      /dev/zvol/rdsk/sp1/gw/trio6
600144f01eb3862c0000494b55fd0005      17179803648      /dev/zvol/rdsk/sp1/gw/trio5
600144f01eb3862c0000494b55fa0004      17179803648      /dev/zvol/rdsk/sp1/gw/trio4
600144f01eb3862c0000494b55f80003      17179803648      /dev/zvol/rdsk/sp1/gw/trio3
600144f01eb3862c0000494b55f50002      17179803648      /dev/zvol/rdsk/sp1/gw/trio2
600144f01eb3862c0000494b55cd0001      17179803648      /dev/zvol/rdsk/sp1/gw/trio1

A host group was defined named GW1 and respectively these LU GUID’s were added to the GW1 host group as LU views assigning LUN 0 to 5.

Just as a note the group names are case sensitive. 
root@ss1:~#stmfadm create-hg GW1

Here we assigned the GUID’s a LUN value on the GW1 host group with the -n parm.   

root@ss1:~# stmfadm add-view -h GW1 -n 0 600144F01EB3862C0000494B55CD0001
root@ss1:~# stmfadm add-view -h GW1 -n 1 600144F01EB3862C0000494B55F50002
root@ss1:~# stmfadm add-view -h GW1 -n 2 600144F01EB3862C0000494B55F80003
root@ss1:~# stmfadm add-view -h GW1 -n 3 600144F01EB3862C0000494B55FA0004
root@ss1:~# stmfadm add-view -h GW1 -n 4 600144F01EB3862C0000494B55FD0005
root@ss1:~# stmfadm add-view -h GW1 -n 5 600144F01EB3862C0000494B56000006

With the LU’s now available in a host group view we can add the COMSTAR re-provisioning gateway server FC wwn’s to this host group and it will become available as a storage resource on the re-provisioning gateway server named ss2. We need to obtain the wwn from the gateway server using the fcinfo hba-port command.  
root@ss2:~# fcinfo hba-port
HBA Port WWN: 210000e08b100163
        Port Mode: Initiator
        Port ID: 10300
        OS Device Name: /dev/cfg/c8
        Manufacturer: QLogic Corp.
        Model: QLA2300
        Firmware Version: 03.03.27
        FCode/BIOS Version:  BIOS: 1.47;
        Serial Number: not available
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200000e08b100163
        NPIV Not Supported

Using the stmfadm utility we add the gateway server’s wwn address to the GW1 host group. 
root@ss1:~# stmfadm add-hg-member -g GW1 wwn.210000e08b100163

Once added to ss1 we can see that it is indeed available and online. 
root@ss1:~# stmfadm list-target -v

Target: wwn.2100001B320EFD58
    Operational Status: Online
    Provider Name     : qlt
    Alias             : qlt2,0
    Sessions          : 1
        Initiator: wwn.210000E08B100163
            Alias: :qlc1
            Logged in since: Fri Dec 19 01:47:07 2008

The cfgadm command will scan for the newly available LUN’s and now we can access the emulated (aka boat anchor) storage system using our gateway server ss2. Of course we could also set up more initiators and access it over a multipath connection.  

cfgadm -a

root@ss2:~# format
Searching for disks…done

       0. c0t600144F01EB3862C0000494B55CD0001d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>

       1. c0t600144F01EB3862C0000494B55F50002d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>

       2. c0t600144F01EB3862C0000494B55F80003d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>

       3. c0t600144F01EB3862C0000494B55FA0004d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>

       4. c0t600144F01EB3862C0000494B55FD0005d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>

       5. c0t600144F01EB3862C0000494B56000006d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>

Now that we some FC LUN connections configured from to the storage system to be re-provisioned we can create a ZFS based pool which grants us the ability to carve out the block storage in a virtual manner. As discussed previously we will use raid dp a.k.a. raidz2 to provide a higher level of availability with the zpool create raidz2 option command.

root@ss2:~# zpool create gwrp1 raidz2 c0t600144F01EB3862C0000494B55CD0001d0 c0t600144F01EB3862C0000494B55F50002d0 c0t600144F01EB3862C0000494B55F80003d0 c0t600144F01EB3862C0000494B55FA0004d0 c0t600144F01EB3862C0000494B55FD0005d0 c0t600144F01EB3862C0000494B56000006d0

A quick status check reveals all is well with the ZFS pool.

root@ss2:~# zpool status gwrp1
  pool: gwrp1
 state: ONLINE
 scrub: none requested

        NAME                                       STATE     READ WRITE CKSUM
        gwrp1                                      ONLINE       0     0     0
          raidz2                                   ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55CD0001d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55F50002d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55F80003d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55FA0004d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55FD0005d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B56000006d0  ONLINE       0     0     0

Let’s carve out some of this newly created pool as a 32GB sparse volume. The -p option creates the full path if it does not currently exist.

root@ss2:~# zfs create -p -s -V 32G gwrp1/stores/lun0

root@ss2:~# zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
gwrp1                        220K  62.6G  38.0K  /gwrp1
gwrp1/stores                67.9K  62.6G  36.0K  /gwrp1/stores
gwrp1/stores/lun0           32.0K  62.6G  32.0K  –

With a slice of the pool created we can now assign a GUID within the LU Provider layer using the sbdadm utility.

root@ss2:~# sbdadm create-lu /dev/zvol/rdsk/gwrp1/stores/lun0

Created the following LU:

              GUID                    DATA SIZE           SOURCE
——————————–  ——————-  —————-
600144f07ed404000000496813070001      34359672832      /dev/zvol/rdsk/gwrp1/stores/lun0

The LU Provider layer can also provision sparse based storage. However in this case the ZFS backing store is already thin provisioned. If this were a physical disk backing store it would be prudent to use the LU Provider’s sparse/thin provisioning feature. At this point we are ready to create the STMF Host Group and View that will be used to demonstrate a real world example of the multi protocol capability with the COMSTAR OpenStorage ss2 host. In this case I will use VMware ESX as a storage consumer. To reflect the host group type we will name it ESX1 and then we need to add a view for the LU GUID of the virtualized storage.

root@ss2:~# stmfadm create-hg ESX1

root@ss2:~# stmfadm add-view -h ESX1 -n 1 600144f07ed404000000496813070001

root@ss2:~# stmfadm list-view -l 600144F07ED404000000496813070001
View Entry: 0
    Host group   : ESX1
    Target group : All
    LUN          : 1

With a view defined for the VMware hosts let’s add an ESX host FC HBA wwn membership to the defined ESX1 host group. We need to retrieve the wwn from the VMware server using either the console or a Virtual Infrastructure Client GUI. Personally I like the console esxcfg-info tool, however if it’s an ESXi host then the GUI will serve the info just as well.

VMware Screen shot WWN by Mike La Spina

[root@vh1 root]# esxcfg-info -s | grep ‘Adapter WWN’
                     |—-Adapter WWNN…………………………20:00:00:e0:8b:01:f7:e2

root@ss2:~# stmfadm add-hg-member -g ESX1 wwn.210000e08b01f7e2

And the result of this change after we issue a rescan on vmhba1 and create a VMFS volume named ss2-cstar-zs0.0 with the re-provisioned storage is reflected here.

VMware Screen shot VMFS volume by Mike La Spina

This crafted storage is now a thinly provisioned VMFS store that can deliver replication, snapshots, cloning, advanced error detection and can also be re-platformed to a new storage system at a later date using ZFS’s hardware autonomy. The storage server is very attractive as it creates a level of future proofing and insulates the storage consumers from proprietary vendor lock in. But that’s not the best part of this example. Let’s say you wish provide different tiers of connectivity services for your storage consumers. For example we could attach a development or test environment using an iSCSI protocol and the more critical environments can use FC or FCoE based protocol.
So let’s look at how we can add a second SCSI transport protocol to this interesting configuration.
Just as a note the new iSCSI port provider is a kernel based implementation and has superior performance to its predecessor iscsitgt user land implementation.

To add the iSCSI protocol we need to enable the iscsi/target port provider service.





root@ss2:~# svcadm enable iscsi/target

Now we need to create an iSCSI target and iSCSI initiator definition so that we can add the iSCSI initiator to the ESX1 host group. As well we should define a target portal group so we can control what host IP(s) will service this target.

root@ss2:~# itadm create-tpg 2

root@ss2:~# itadm create-target

root@ss2:~# itadm create-target -n -t 2
Target successfully created

By default the iqn will be created as a member of the All targets group.

If we left out the parameters the itadm utility would create an iqn GUID and use the default target portal group of 1. And yes for those familiar with the predecessor iscsitadm utility we can now create a iqn name at the command line.

At this point we need to define the initiator iqn to the iSCSI port provider service and if required additionally secure it using CHAP. We need to retrieve the VMware initiator iqn name from either the Virtual Infrastructure Client GUI or console command line. Just as a note if we did not specify a host group when we defined our view the default would allow any initiator FC, iSCSI or otherwise to connect to the LU and this may have a purpose but generally it is a bad practice to allow in most configurations. Once created the initiator is added to the ESX1 host group thus enables our second access protocol to the same LU.

[root@vh1 root]# esxcfg-info -s | grep ‘iqn’
         |—-ISCSI Name……………………………………
         |—-ISCSI Alias……………………………………

root@ss2:~# itadm create-initiator

root@ss2:~# stmfadm add-hg-member -g ESX1

After adding the ss2 iSCSI interface IP to VMware’s Software iSCSI initiator we now have a multipath multiprotocol connection to our COMSTAR storage host.

 VMware iqn example By Mike La Spina

VMware mpath example by Mike La Spina 

This is simply the most functional and advanced Open Source storage product in the world today. Here we have commodity white boxes serving advanced storage protocols in my home lab, can you imagine what could be done with Data Center class server hardware and Fishworks. You can begin to see the advantages of this future proof platform. As protocols like FCoE, Infiniband and iSER (iSCSI without the TCP session overhead) already working in COMSTAR the Sun Software Engineers and OpenSolaris community are crafting outstanding storage products.

Hope you found this blog to be interesting.






Site Contents: © 2009  Mike La Spina

Understanding VMFS volumes

Understanding VMFS volumes is an important element within VMware ESX environments. When storage issues surface we need to correctly evaluate the VMFS volume states and apply the appropriate corrective actions to remediate undesirable storage events. VMFS architecture is not publically available and this certainly adds to the challenge when we need to correct a volume configuration or change issue. So lets begin to look at the components of a VMFS from what I have been able to decrypt using direct analysis.

All VMFS volume partitions will have a partition ID value of fb. Running fdisk can identify any partitions that are flagged as VMFS as shown here.

[root@vh1 ]# fdisk -lu /dev/sdc

Disk /dev/sdc: 274.8 GB, 274877889536 bytes
255 heads, 63 sectors/track, 33418 cylinders, total 536870878 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sdc1           128 536860169 268430021   fb  Unknown

What’s important to note here is the sector size = 512 and the starting/ending blocks.

Many VMFS volume configuration elements are visible in the /vmfs mount folder. Within the directory the are two subdirectories, the volumes directory and the devices directory. The volumes directory provisions the mount point location and the devices directory holds configuration elements. Within the devices directory the are several subdirectories of which I can explain the disks and lvm folders, the others are not known to me outside of theory only.

A key part of a VMFS volume is it’s UUID (aka Universally Unique Identifier) and as the name suggests it used to ensure uniqueness when more than one volume is in use. The UUID is generated on the initial ESX host that created the VMFS volume based on the UUID creation standards. You can determine which ESX host the initial VMFS volume was created on by referring to the last 6 bytes of the UUID. This value is the same as the last six bytes of the ESX host’s system UUID found in the /etc/vmware/esx.conf file.

By far one of the most critical elements on a VMFS volume is the GUID. The GUID is integral within the volume because it is used to form the vml path (aka virtual multipath link). The GUID is stored within the VMFS volume header and begins at address 0x10002E.

The format of the GUID can vary based on different implementations of SCSI transport protocols but generally you will see some obvious length variances of the vml path identifiers which stem from the use of T11 and T10 Standard SCSI address formats like EUI-64, and NAA 64. Regardless of those variables there are components outside of the GUID within the vml that we should take notice of. The vml construct contains references to the LUN and partition values and these are useful to know about. The following illustrates where these elements appear in some real examples.

When we issue an ls -l from the /vmfs/devices/disks directory the following info is observed.

vhhba#:Target:LUN:Partition -> vml:??_LUN_??_GUID:Partition

                       LUN     GUID                         PARTITION
                       ^       ^                            ^
vmhba0:1:0:0  -> vml.02000000005005076719d163d844544e313436
vmhba0:1:0:1  -> vml.02000000005005076719d163d844544e313436:1
vmhba32:1:3:0 -> vml.0200030000600144f07ed404000000496ff8cd0003434f4d535441
vmhba32:1:3:1 -> vml.0200030000600144f07ed404000000496ff8cd0003434f4d535441:1

As well the issuing ls -l on the /vmfs/volumes list the VMFS UUID’s and the link name which is what we see displayed in the GUI client. In this example we will follow the UUID shown in blue and the named ss2-cstar-zs0.2 volume.

ss2-cstar-zs0.2 -> 49716cd8-ebcbbf9a-6792-000d60d46e2e

Additionally we can use esxcfg-vmkhbadevs -m to list the vmhba, dev and UUID associations.

[root@vh1 ]#  esxcfg-vmhbadevs -m
vmhba0:1:0:1    /dev/sdd1                        48a3b0f3-736b896e-af8f-00025567144e
vmhba32:1:3:1   /dev/sdf1                        49716cd8-ebcbbf9a-6792-000d60d46e2e

As you can see we indeed have different GUID lengths in this example. We also can see that the vmhba device is linked to a vml construct and this is how the kernel defines paths to a visible SCSI LUN. The vml path hosts the LUN ID, GUID and partition number information and this is also stored in the volumes VMFS header. As well the header contains a UUID signature but this is not the VMFS UUID.

If we use hexdump as illustrated below we can see these elements in the VMFS header directly.

[root@vh1 root]# hexdump -C -s 0x100000 -n 800 /dev/sdf1
00100000  0d d0 01 c0 03 00 00 00  10 00 00 00 02 16 03 00  |                | <- LUN ID
00100010  00 06 53 55 4e 20 20 20  20 20 43 4f 4d 53 54 41  |  SUN     COMSTA| <- Target Label
00100020  52 20 20 20 20 20 20 20  20 20 31 2e 30 20 60 01  |R         1.0 ` | <- LUN GUID
00100030  44 f0 7e d4 04 00 00 00  49 6f f8 cd 00 03 43 4f  |D ~     Io    CO|
00100040  4d 53 54 41 00 00 00 00  00 00 00 00 00 00 00 00  |MSTA            |
00100050  00 00 00 00 00 00 00 00  00 00 00 02 00 00 00 fc  |                | <- Volume Size
00100060  e9 ff 18 00 00 00 01 00  00 00 8f 01 00 00 8e 01  |                |
00100070  00 00 91 01 00 00 00 00  00 00 00 00 10 01 00 00  |                |
00100080  00 00 d8 6c 71 49 b0 aa  97 9b 6c 2f 00 0d 60 d4  |   lqI    l/  ` |
00100090  6e 2e 6e 89 19 fb a6 60  04 00 a7 ce 20 fb a6 60  |n n    `       `|
001000a0  04 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
001000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
00100200  00 00 00 f0 18 00 00 00  90 01 00 00 00 00 00 00  |                |
00100210  01 00 00 00 34 39 37 31  36 63 64 38 2d 36 30 37  |    49716cd8-607| <- SEG UUID in ASCII
00100220  35 38 39 39 61 2d 61 64  31 63 2d 30 30 30 64 36  |5899a-ad1c-000d6|
00100230  30 64 34 36 65 32 65 00  00 00 00 00 00 00 00 00  |0d46e2e         |
00100240  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
00100250  00 00 00 00 d8 6c 71 49  9a 89 75 60 1c ad 00 0d  |     lqI  u`    | <- SEG UUID
00100260  60 d4 6e 2e 01 00 00 00  e1 9c 19 fb a6 60 04 00  |` n          `  |
00100270  00 00 00 00 8f 01 00 00  00 00 00 00 00 00 00 00  |                |
00100280  8e 01 00 00 00 00 00 00  64 cc 20 fb a6 60 04 00  |        d    `  |
00100290  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
001002a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |


In addition to the VMFS header block we have the hidden metadata files of the volume which you can list using ls -al. The vh.sf contains the UUID of the VMFS store and any member segments info. (I would presume the name vh stands for Volume Header … ;D) 



[root@vh1 ]# hexdump -C -s 0x200000 -n 256 /vmfs/volumes/49716cd8-ebcbbf9a-6792-000d60d46e2e/.vh.sf
00200000  5e f1 ab 2f 04 00 00 00  1f d8 6c 71 49 9a bf cb  |^   /  lqI      | <- VMFS UUID
00200010  eb 92 67 00 0d 60 d4 6e  2e 02 00 00 00 73 73 32  |  g  ` n     ss2| <- Volume Name
00200020  2d 63 73 74 61 72 2d 7a  73 30 2e 32 00 00 00 00  |-cstar-zs0.2    |
00200030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
00200090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 02 00  |                |
002000a0  00 00 00 10 00 00 00 00  00 d8 6c 71 49 01 00 00  |          lqI   |
002000b0  00 d8 6c 71 49 9a 89 75  60 1c ad 00 0d 60 d4 6e  |  lqI  u`    ` n| <- SEG UUID
002000c0  2e 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
002000d0  00 00 00 01 00 20 00 00  00 00 00 01 00 00 00 00  |                |
002000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |

And of course we can not leave out the partition entry block data for the device.

hexdump -C -n 256 /dev/sdf1

00000000  fa b8 00 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |                |
00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |   |         !  |
00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |    8 u        u|
00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 01 8b  |         |   t  |
00000040  4c 02 cd 13 ea 00 7c 00  00 eb fe 00 00 00 00 00  |L     |         |
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 02  |                |
000001c0  03 00 fb fe ff ff 80 00  00 00 72 ef bf 5d 00 00  |          r  ]  | Type Start End
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |

With this detailed information it is possible to solve some common security issues with VMware stores like volume deletion and unintentional LUN ID changes.

Recently VMware added a some what useful command line tool named vmfs-undelete which exports metadata to a recovery log file which can restore vmdk block addresses in the event of deletion. It’s a simple tool and at present it’s experimental and unsupported and is not available on ESXi. The tool of course demands that you were proactive and ran it’s backup function in order to use it. Well I think this falls well short of what we need here. What if you have no previous backups of the VMFS configuration, so we really need to know what to look for and how to correct it and that’s exactly why I created this blog.  

The volume deletion event is quite easy to fix and thats simply because the VMFS volume header is not actually deleted. The partition block data is what gets trashed and you can just about get way with murder when it comes to recreating that part. Within the first 128 sectors is the peice we need to fix. One method is to create a store with the same storage volume and then block copy the partition to a file which can be block copied to the deleted device partition data blocks and this will fix the issue.

For example we create a new VMFS store on the same storage backing with the same LUN size as the original and it shows up as a LUN with a device name of /dev/sdd we can use esxcfg-vmhbadevs -m to find it if required

The deleted device name was /dev/sdc

We use the dd command to do a block copy from the new partition to a file or even directly in this case.

Remember to back it up first!

dd if=/dev/sdc of=/var/log/part-backup-sdc-1st.hex  bs=512 count=1

then issue

dd if=/dev/sdd of=/dev/sdc bs=512 count=1


dd if=/dev/sdd of=/var/log/part-backup-sdd.hex bs=512 count=1

dd if=/var/log/part-backup-sdd.hex of=/dev/sdc bs=512 count=1

I personally like using a file to perform this function as this becomes a future backup element which you can move to a safe location. The file can actually be edited with other utilities to provide more flexibility. e.g. hexedit etc. Addtitionally you could use fdisk to directly edit the partition table and provide the correct start and end addresses. This is something you should only do if you are well versed in it’s usage.

As as an additional level of protection we could even include making backups of the vh.sf metadata file and the VMFS header.

cp /vmfs/volumes/49716cd8-ebcbbf9a-6792-000d60d46e2e/.vh.sf /var/log/vh.sf.bu

dd if=/dev/sdc of=/var/log/vmfsheader-bu-sdc.hex bs=512 count=4096

This would grant the ability for support to examine the exact details of the VMFS configuration and potentially allow recovery from more complex issues.

One of the most annoying security events is when a VMFS LUN get’s changed inadvertently. If a VMFS volume LUN ID changes and is presented to an ESX host then the presented volume will be treated as a potential snapshot LUN. If this event occurs and the ESX servers advanced LVM parameter settings are at default the ESX host will not mount the volume. This behaviour is to prevent the possibility of corruption and downing the host since it can not determine which VM metadata inventory is correct.

If you are aware that the LUN ID has changed then the best course of action is to re-establish the correct LUN ID at the storage server first and rescan the affected vmhba’s. This is important because if you need to resignature the VMFS volume it will also require that the VM’s be imported back into inventory. Virtual Center logging and other various settings will be lost when this action is performed. This is a result of now having an incorrect UUID between the metadata, mount location and the vmx file UUID value.

If the storage change cannot be reverted back then a VMFS resignature method is the only option for reprovisioning a VMFS volume mount.

This is invoked by setting the LVM.DisallowSnapshotLun = 0 and LVM.EnableResignature = 1 and these should reverted back once the VMFS resignature operation is complete.



Site Contents: © 2009  Mike La Spina

A centrally based method for patching ESX3 VMWare Servers

I have updated my ESX servers manually many times and I find the process to say at the least is “annoying” so I decided to change it to an http based method with a modified patch configuration. I found that it really works well.

I did some searching prior to the method I settled on and found this blog which is quite good.

I felt it does have some issues and I wanted to avoid running custom scripts on the server side. So here is what I build for my patch management solution.

Using the standard tools on the ESX3 server of cron jobs and esxupdate I created the following on my servers.

Define a new cron entry for running esxupdate every first Sunday of the month.
I chose to create a separate entry avoiding the esx installed ones for safety reasons. 

Edit the file /etc/crontab and add the line in bold as show. 

[root@yourserver /]nano /etc/crontab


# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
30 3 1 * 0 root run-parts /etc/cron.esxupdate

Create a new directory for the esxupdate cron run part as follows.

[root@yourserver /]mkdir /etc/cron.esxupdate

Create a new command file for the esxupdate cron run part and add the text lines as follows.

[root@yourserver /]nano /etc/cron.esxupdate/esxpatch

esxupdate -r http://yourhttpserver.fq.dns:8088/esx3.all update

Set the file as executable using chmod

chmod 755 /etc/cron.esxupdate/esxpatch

Allow outgoing traffic from the server using the esxcfg-firewall script as follows.

[root@yourserver /]esxcfg-firewall –allowoutgoing

Create an http server to host the patch repository.
I built an IIS server on my Virtual Center 3 server on port 8088 as follows

Install IIS (Be sure to secure it).

Create a patch folder named C:VMWarePatches and set your IIS home directory as follows.

 IIS example

You will need to add the following content types from a right mouse click-> properties of the IIS server definition within the Computer Manager mmc as shown.

Content types .hdr and .info need to be present.

 IIS Mime Example

Download the patches you require and extract them to c:vmwarepatches in the usual manner for repo based deployment. For example I downloaded the available esx-upgrade-from-esx3-3.0.2-61618.tar.gz upgrade file and extracted it to c:vmwarepatches which leaves a directory named 61818 in the http distro folder.

Rename this directory to esx3.all

Now this is were things get interesting as we are going to customize this patch package to include the latest additional patches that are available.

VMWare is using the installation and update functionally of Redhat rpm, yum and esxupdate scripts which means we can modify the repo source by conforming to the existing configuration rules. This configuration of the repo is based on xml tags of which I modified three tag element types.

1) info

The descriptor xml file found in the root of the update package which contains the descriptions of various patches in the package and looks like the following.

<descriptor version=”1.0″>
  <vendor>VMware, Inc.</vendor>
  <product>VMware ESX Server</product>
  <releasedate>Fri Nov 2 05:01:31 PDT 2007</releasedate>
  <summary>3.0.2 Update 1 of VMware ESX Server</summary>
  <description>This is 3.0.2 Update 1 rollup to Nov 2nd,2007 of VMware ESX Server.
It contains:
3.0.2-52542  Full 3.0.2 release of VMware ESX Server
ESX-1001724  Security bugs fixed in vmx rpm.
ESX-1001735  To update tzdata rpm.

ESX-1002424  VMotion RARP broadcast to multiple vmnic.
ESX-1002425  VMware-hostd-esx 3.0.2-62488
ESX-1002429  Path failback issue with EMC iSCSI array.

This xml file is used by the esxupdate script and can be modified to support additional patches. The parts we need to add are some descriptor text element data so that we know what it covers. I added ESX-1002424, 25 and 29 to the descriptor tag. And edited the headings date to reflect its current value.

2) rpms

The second element type I changed was the rpmlist tag. It looks like the following which is only a partial.

    <rpm arch=”i386″ rel=”8.37.15″ ver=”2.4″>kernel-utils</rpm>
    <rpm arch=”i686″ rel=”47.0.1.EL.62488″ ver=”2.4.21″>kernel-vmnix</rpm>
    <rpm arch=”i386″ rel=”66″ ver=”1.2.7″>krb5-libs</rpm>
    <rpm arch=”i386″ rel=”11″ ver=”1.1.1″>krbafs</rpm>
    <rpm arch=”i386″ rel=”3vmw” ver=”″>kudzu</rpm>
    <rpm arch=”i386″ rel=”70RHEL3″ ver=”0.1″>laus-libs</rpm>
    <rpm arch=”i386″ rel=”12″ ver=”378″>less</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-perftools</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-uwlibs</rpm>
    <rpm arch=”i386″ rel=”62488″ ver=”3.0.2″>VMware-esx-vmx</rpm>
    <rpm arch=”i386″ rel=”61818″ ver=”3.0.2″>VMware-esx-vmkernel</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-vmkctl</rpm>
    <rpm arch=”i386″ rel=”55869″ ver=”3.0.2″>VMware-esx-tools</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-srvrmgmt</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-scripts</rpm>
When you extract a newer esx3 patch it will contain its own descriptor rpmlist tags etc. and you can update this major upgrade list to include those new elements. You need to replace the old tag info if it exists which it usually will. For example the bold entry above for VMware-esx-vmkernel needs to be completely updated by removing the old tag values like rel=61818 and insert the new value of rel=62488. I simply searched for the tag name VMware-esx-vmkernel deleted the line and inserted the updated one. I will eventually write a script to do the config edits, but for now these hacks will do the trick.

3) nodeps

The third element I changed will not normally be necessary but in this case there is a bug in this upgrade package that did not account for an rpm option value of –U which means upgrade an rpm if present. The descriptor included a tag named nodeps which I don’t think works well when the rpm already exists of the same version so we need to remove the tag all together for previous base installs above version esx 3.0.0.

I deleted these tags for my config.


In addition to the descriptor xml file edits we will need to copy the new rpm files into the esx3.all folder, remove the old ones, edit the file found in the headers sub folder, copy the new hdr files to the headers folder and remove the old ones in the headers folder.

For example in adding ESX-1002424 I copied

From c:vmwarepatchesesx-1002424



To c:vmwarepatchesesx3.all

From c:vmwarepatchesesx-1002424headers



To c:vmwarepatchesesx3.allheaders

You should delete any older file versions matching the prefixes portions of the newer files to avoid confusion. It will still function if you leave the old ones but it’s not a best practice.

The last step is to edit the file, search for the text lines of the rpm file prefix and replace the info lines with the newer ones found from the updated patch file.

For example search for text VMware-esx-vmx in the master file which looks like the following partial example.


Replace it completely with the newer text entry.

I tested my configuration on serveral base install versions and is was nice to see that on an esx 3.0.2-61818 it only downloaded and installed the addtional patches that were added to the master repo esx3.all

You should keep in mind that based on the cron schedule your VM’s may need to go down on that host and you must plan those outage times and gracefully shutdown where applicable.

Till next time …  happy esx patching.



Site Contents: © 2008  Mike La Spina

« Previous PageNext Page »