Ubiquitous Talk

Power House Blog on iSCSI and VMware lead by Chad Sakac

Site Contents: © 2009  Mike La Spina

Multi Protocol Storage Provisioning with COMSTAR

 

 

In this example instance we are re-provisioning an existing storage system with an OpenSolaris COMSTAR configuration running on a commodity white box which functions as a storage server head that can compress, scrub, thin provision, replicate, snapshot and clone the existing block storage attachments. The example FC based storage could also be comprised of a JBOD FC array directly attached to the OpenSolaris storage head if we so desired or many other commonly available SCSI attachment methods. The objective here is to extend and enhance any block storage system with high performance transports and virtualization features. Of course we could also formalize the white box to an industrial strength host once we are satisfied that the proof of concept is mature and optimal.  

The reality is that many older existing FC storage systems are installed without these features primarily due to the excessive licensing costs of them. And even when these features are available, its use is probably restricted to like proprietary systems thus obsolescing the entire lot of any useful future functionality. But what if you could re-purpose an older storage system to act as a DR store or backup cache system or maybe a test and development environment. With today’s economy this is from a cost perspective, very attractive and can be accomplished with very little risk on the investment side.

One of the possible applications for this flexible storage service is the re-provisioning of existing LUN’s from an existing system to newer more flexible SCSI transport protocols. This is particularly useful when we need to re-target the existing storage system from FC to iSCSI or the likes of. We can begin by exploring this functionality and explain how COMSTAR can provide us with this service.

First we need to understand the high level functionality of the COMSTAR service layers. Virtual LUN’s on COMSTAR are provisioned with a service layer named the LU provider. This layer maps backing stores of various types to a storage GUID assignment and additionally defines other properties like the LUN ID and size dimensions. This layer allows us to carve out the available block storage devices that are accessible on our OpenSolaris storage host. For example if we attached an FC Initiator to an external storage system we can then map the accessible SCSI block devices to the LU provider layer and then present this virtualized LUN to the other COMSTAR service layers for further processing.

Once we have defined the LU’s we can present this storage resource to the SCSI Target Mode Framework Service (STMF) layer which acts as the storage gate keeper. At this layer we define which clients (initiators) can connect to the LU’s based on Membership of Target Groups and Host Groups that are assigned logical views of the LU(s). The STMF layer routes the defined LU(s) as SCSI targets over a multiprotocol interface connection pool to a Port Provider. Port Providers are the protocol connection service instances which can be the likes of FC, iSCSI, SAS, iSER, FCoE and so on. 

With these COMSTAR basics in mind let us begin by diving into some of the details of how this can be applied.  

Sun has detailed howto setup COMSTAR at dlc.sun.com so no need to re-invent the wheel here.

Just as a note SXCE snv_103 and up integrate the COMSTAR FC and iSCSI port provider code. With the COMSTAR software components and FC target setup we can demonstrate the re-provisioning of an existing FC based storage server. Since I don’t have the luxury of having a proprietary storage server at home I will emulate this storage using an additional COMSTAR white box to act as the FC storage target to be re-provisioned.

On the existing FC target system we need to create Raid0 arrays of three disks each which will total up to a set of six trios. We will use these six non-fault tolerant disk groups as vdevs for a ZFS raidz2 group. This will allow us to create fault tolerant arrays from the existing storage server. The reasons for sets of three Raid0 groupings are to reduce the possibility of reaching the LUN maximums of the proprietary storage system and also we do not want to erode the performance by layering Raid 5 groups. As well we can tolerate a disk failure in two of the trios since we have Raidz2 across the Raid0 trio groups. Additionally using these Raid0 disk groups actually lowers the array failure probability rate. For example if a second disk were to failure in a single Raid0 set there would be no additional impact to other trios, thus reducing the overall failure rate. 

To create the emulated FC storage system I have defined the following 16G ZFS sparse volumes respectively named trio1 through trio6 each as a representation of the 3 disk Raid0 spanned LUN on a source storage host named ss1. 

root@ss1:~# zfs create sp1/gw
root@ss1:~# zfs create -s -V 16G sp1/gw/trio1
root@ss1:~# zfs create -s -V 16G sp1/gw/trio2
root@ss1:~# zfs create -s -V 16G sp1/gw/trio3
root@ss1:~# zfs create -s -V 16G sp1/gw/trio4
root@ss1:~# zfs create -s -V 16G sp1/gw/trio5
root@ss1:~# zfs create -s -V 16G sp1/gw/trio6

Once these mockup volumes are created they are then defined as backing stores using the sbdadm utility as follows.

root@ss1:~# sbdadm create-lu /dev/zvol/rdsk/sp1/gw/trio1

Created the following LU:

              GUID                    DATA SIZE           SOURCE
——————————–  ——————-  —————-
600144f01eb3862c0000494b55cd0001      17179803648      /dev/zvol/rdsk/sp1/gw/trio1

All the backing stores were added to the LU provider service layer to which in turn were assigned to the STMF service layer. Here we can see the automatically generated GUID’s that are assigned to the ZFS backing stores.

root@ss1:~# sbdadm list-lu

Found 6 LU(s)

              GUID                    DATA SIZE           SOURCE
——————————–  ——————-  —————-
600144f01eb3862c0000494b56000006      17179803648      /dev/zvol/rdsk/sp1/gw/trio6
600144f01eb3862c0000494b55fd0005      17179803648      /dev/zvol/rdsk/sp1/gw/trio5
600144f01eb3862c0000494b55fa0004      17179803648      /dev/zvol/rdsk/sp1/gw/trio4
600144f01eb3862c0000494b55f80003      17179803648      /dev/zvol/rdsk/sp1/gw/trio3
600144f01eb3862c0000494b55f50002      17179803648      /dev/zvol/rdsk/sp1/gw/trio2
600144f01eb3862c0000494b55cd0001      17179803648      /dev/zvol/rdsk/sp1/gw/trio1

A host group was defined named GW1 and respectively these LU GUID’s were added to the GW1 host group as LU views assigning LUN 0 to 5.

Just as a note the group names are case sensitive. 
root@ss1:~#stmfadm create-hg GW1

Here we assigned the GUID’s a LUN value on the GW1 host group with the -n parm.   

root@ss1:~# stmfadm add-view -h GW1 -n 0 600144F01EB3862C0000494B55CD0001
root@ss1:~# stmfadm add-view -h GW1 -n 1 600144F01EB3862C0000494B55F50002
root@ss1:~# stmfadm add-view -h GW1 -n 2 600144F01EB3862C0000494B55F80003
root@ss1:~# stmfadm add-view -h GW1 -n 3 600144F01EB3862C0000494B55FA0004
root@ss1:~# stmfadm add-view -h GW1 -n 4 600144F01EB3862C0000494B55FD0005
root@ss1:~# stmfadm add-view -h GW1 -n 5 600144F01EB3862C0000494B56000006

With the LU’s now available in a host group view we can add the COMSTAR re-provisioning gateway server FC wwn’s to this host group and it will become available as a storage resource on the re-provisioning gateway server named ss2. We need to obtain the wwn from the gateway server using the fcinfo hba-port command.  
root@ss2:~# fcinfo hba-port
HBA Port WWN: 210000e08b100163
        Port Mode: Initiator
        Port ID: 10300
        OS Device Name: /dev/cfg/c8
        Manufacturer: QLogic Corp.
        Model: QLA2300
        Firmware Version: 03.03.27
        FCode/BIOS Version:  BIOS: 1.47;
        Serial Number: not available
        Driver Name: qlc
        Driver Version: 20080617-2.30
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200000e08b100163
        NPIV Not Supported

Using the stmfadm utility we add the gateway server’s wwn address to the GW1 host group. 
root@ss1:~# stmfadm add-hg-member -g GW1 wwn.210000e08b100163

Once added to ss1 we can see that it is indeed available and online. 
root@ss1:~# stmfadm list-target -v

Target: wwn.2100001B320EFD58
    Operational Status: Online
    Provider Name     : qlt
    Alias             : qlt2,0
    Sessions          : 1
        Initiator: wwn.210000E08B100163
            Alias: :qlc1
            Logged in since: Fri Dec 19 01:47:07 2008

The cfgadm command will scan for the newly available LUN’s and now we can access the emulated (aka boat anchor) storage system using our gateway server ss2. Of course we could also set up more initiators and access it over a multipath connection.  

cfgadm -a

root@ss2:~# format
Searching for disks…done
AVAILABLE DISK SELECTIONS:
       0. c0t600144F01EB3862C0000494B55CD0001d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g600144f01eb3862c0000494b55cd0001
       1. c0t600144F01EB3862C0000494B55F50002d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g600144f01eb3862c0000494b55f50002
       2. c0t600144F01EB3862C0000494B55F80003d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g600144f01eb3862c0000494b55f80003
       3. c0t600144F01EB3862C0000494B55FA0004d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g600144f01eb3862c0000494b55fa0004
       4. c0t600144F01EB3862C0000494B55FD0005d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g600144f01eb3862c0000494b55fd0005
       5. c0t600144F01EB3862C0000494B56000006d0 <DEFAULT cyl 2086 alt 2 hd 255 sec 63>
          /scsi_vhci/disk@g600144f01eb3862c0000494b56000006

Now that we some FC LUN connections configured from to the storage system to be re-provisioned we can create a ZFS based pool which grants us the ability to carve out the block storage in a virtual manner. As discussed previously we will use raid dp a.k.a. raidz2 to provide a higher level of availability with the zpool create raidz2 option command.

root@ss2:~# zpool create gwrp1 raidz2 c0t600144F01EB3862C0000494B55CD0001d0 c0t600144F01EB3862C0000494B55F50002d0 c0t600144F01EB3862C0000494B55F80003d0 c0t600144F01EB3862C0000494B55FA0004d0 c0t600144F01EB3862C0000494B55FD0005d0 c0t600144F01EB3862C0000494B56000006d0

A quick status check reveals all is well with the ZFS pool.

root@ss2:~# zpool status gwrp1
  pool: gwrp1
 state: ONLINE
 scrub: none requested
config:

        NAME                                       STATE     READ WRITE CKSUM
        gwrp1                                      ONLINE       0     0     0
          raidz2                                   ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55CD0001d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55F50002d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55F80003d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55FA0004d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B55FD0005d0  ONLINE       0     0     0
            c0t600144F01EB3862C0000494B56000006d0  ONLINE       0     0     0

Let’s carve out some of this newly created pool as a 32GB sparse volume. The -p option creates the full path if it does not currently exist.

root@ss2:~# zfs create -p -s -V 32G gwrp1/stores/lun0 

root@ss2:~# zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
gwrp1                        220K  62.6G  38.0K  /gwrp1
gwrp1/stores                67.9K  62.6G  36.0K  /gwrp1/stores
gwrp1/stores/lun0           32.0K  62.6G  32.0K  –

With a slice of the pool created we can now assign a GUID within the LU Provider layer using the sbdadm utility.

root@ss2:~# sbdadm create-lu /dev/zvol/rdsk/gwrp1/stores/lun0

Created the following LU:

              GUID                    DATA SIZE           SOURCE
——————————–  ——————-  —————-
600144f07ed404000000496813070001      34359672832      /dev/zvol/rdsk/gwrp1/stores/lun0

The LU Provider layer can also provision sparse based storage. However in this case the ZFS backing store is already thin provisioned. If this were a physical disk backing store it would be prudent to use the LU Provider’s sparse/thin provisioning feature. At this point we are ready to create the STMF Host Group and View that will be used to demonstrate a real world example of the multi protocol capability with the COMSTAR OpenStorage ss2 host. In this case I will use VMware ESX as a storage consumer. To reflect the host group type we will name it ESX1 and then we need to add a view for the LU GUID of the virtualized storage.

root@ss2:~# stmfadm create-hg ESX1

root@ss2:~# stmfadm add-view -h ESX1 -n 1 600144f07ed404000000496813070001

root@ss2:~# stmfadm list-view -l 600144F07ED404000000496813070001
View Entry: 0
    Host group   : ESX1
    Target group : All
    LUN          : 1

With a view defined for the VMware hosts let’s add an ESX host FC HBA wwn membership to the defined ESX1 host group. We need to retrieve the wwn from the VMware server using either the console or a Virtual Infrastructure Client GUI. Personally I like the console esxcfg-info tool, however if it’s an ESXi host then the GUI will serve the info just as well.

[root@vh1 root]# esxcfg-info -s | grep ‘Adapter WWN’
                     |—-Adapter WWNN…………………………20:00:00:e0:8b:01:f7:e2

root@ss2:~# stmfadm add-hg-member -g ESX1 wwn.210000e08b01f7e2

And the result of this change after we issue a rescan on vmhba1 and create a VMFS volume named ss2-cstar-zs0.0 with the re-provisioned storage is reflected here.

 

 

root@ss2:~# svcadm enable iscsi/target

Now we need to create an iSCSI target and iSCSI initiator definition so that we can add the iSCSI initiator to the ESX1 host group. As well we should define a target portal group so we can control what host IP(s) will service this target.

root@ss2:~# itadm create-tpg 2 10.0.0.1

root@ss2:~# itadm create-target

root@ss2:~# itadm create-target -n iqn.1986-03.com.sun:02:ss2.0 -t 2
Target iqn.1986-03.com.sun:02:ss2.0 successfully created

By default the iqn will be created as a member of the All targets group.

If we left out the parameters the itadm utility would create an iqn GUID and use the default target portal group of 1. And yes for those familiar with the predecessor iscsitadm utility we can now create a iqn name at the command line.

At this point we need to define the initiator iqn to the iSCSI port provider service and if required additionally secure it using CHAP. We need to retrieve the VMware initiator iqn name from either the Virtual Infrastructure Client GUI or console command line. Just as a note if we did not specify a host group when we defined our view the default would allow any initiator FC, iSCSI or otherwise to connect to the LU and this may have a purpose but generally it is a bad practice to allow in most configurations. Once created the initiator is added to the ESX1 host group thus enables our second access protocol to the same LU.

[root@vh1 root]# esxcfg-info -s | grep ‘iqn’
         |—-ISCSI Name……………………………………..iqn.1998-01.com.vmware:vh1.1
         |—-ISCSI Alias…………………………………….iqn.1998-01.com.vmware:vh1.1

root@ss2:~# itadm create-initiator iqn.1998-01.com.vmware:vh1.1

root@ss2:~# stmfadm add-hg-member -g ESX1 iqn.1998-01.com.vmware:vh1.1

After adding the ss2 iSCSI interface IP to VMware’s Software iSCSI initiator we now have a multipath multiprotocol connection to our COMSTAR storage host.

 

 

This is simply the most functional and advanced Open Source storage product in the world today. Here we have commodity white boxes serving advanced storage protocols in my home lab, can you imagine what could be done with Data Center class server hardware and Fishworks. You can begin to see the advantages of this future proof platform. As protocols like FCoE, Infiniband and iSER (iSCSI without the TCP session overhead) already working in COMSTAR the Sun Software Engineers and OpenSolaris community are crafting outstanding storage products.

 

 

Site Contents: © 2009  Mike La Spina

Understanding VMFS volumes

Understanding VMFS volumes is an important element within VMware ESX environments. When storage issues surface we need to correctly evaluate the VMFS volume states and apply the appropriate corrective actions to remediate undesirable storage events. VMFS architecture is not publically available and this certainly adds to the challenge when we need to correct a volume configuration or change issue. So lets begin to look at the components of a VMFS from what I have been able to decrypt using direct analysis.

All VMFS volume partitions will have a partition ID value of fb. Running fdisk can identify any partitions that are flagged as VMFS as shown here.

[root@vh1 ]# fdisk -lu /dev/sdc

Disk /dev/sdc: 274.8 GB, 274877889536 bytes
255 heads, 63 sectors/track, 33418 cylinders, total 536870878 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sdc1           128 536860169 268430021   fb  Unknown

What’s important to note here is the sector size = 512 and the starting/ending blocks.

Many VMFS volume configuration elements are visible in the /vmfs mount folder. Within the directory the are two subdirectories, the volumes directory and the devices directory. The volumes directory provisions the mount point location and the devices directory holds configuration elements. Within the devices directory the are several subdirectories of which I can explain the disks and lvm folders, the others are not known to me outside of theory only.

A key part of a VMFS volume is it’s UUID (aka Universally Unique Identifier) and as the name suggests it used to ensure uniqueness when more than one volume is in use. The UUID is generated on the initial ESX host that created the VMFS volume based on the UUID creation standards. You can determine which ESX host the initial VMFS volume was created on by referring to the last 6 bytes of the UUID. This value is the same as the last six bytes of the ESX host’s system UUID found in the /etc/vmware/esx.conf file.

By far one of the most critical elements on a VMFS volume is the GUID. The GUID is integral within the volume because it is used to form the vml path (aka virtual multipath link). The GUID is stored within the VMFS volume header and begins at address 0x10002E.

The format of the GUID can vary based on different implementations of SCSI transport protocols but generally you will see some obvious length variances of the vml path identifiers which stem from the use of T11 and T10 Standard SCSI address formats like EUI-64, and NAA 64. Regardless of those variables there are components outside of the GUID within the vml that we should take notice of. The vml construct contains references to the LUN and partition values and these are useful to know about. The following illustrates where these elements appear in some real examples.

When we issue an ls -l from the /vmfs/devices/disks directory the following info is observed.

vhhba#:Target:LUN:Partition -> vml:??_LUN_??_GUID:Partition

                       LUN     GUID                         PARTITION
                       ^       ^                            ^
vmhba0:1:0:0  -> vml.02000000005005076719d163d844544e313436
vmhba0:1:0:1  -> vml.02000000005005076719d163d844544e313436:1
vmhba32:1:3:0 -> vml.0200030000600144f07ed404000000496ff8cd0003434f4d535441
vmhba32:1:3:1 -> vml.0200030000600144f07ed404000000496ff8cd0003434f4d535441:1

As well the issuing ls -l on the /vmfs/volumes list the VMFS UUID’s and the link name which is what we see displayed in the GUI client. In this example we will follow the UUID shown in blue and the named ss2-cstar-zs0.2 volume.

ss2-cstar-zs0.2 -> 49716cd8-ebcbbf9a-6792-000d60d46e2e

Additionally we can use esxcfg-vmkhbadevs -m to list the vmhba, dev and UUID associations.

[root@vh1 ]#  esxcfg-vmhbadevs -m
vmhba0:1:0:1    /dev/sdd1                        48a3b0f3-736b896e-af8f-00025567144e
vmhba32:1:3:1   /dev/sdf1                        49716cd8-ebcbbf9a-6792-000d60d46e2e

As you can see we indeed have different GUID lengths in this example. We also can see that the vmhba device is linked to a vml construct and this is how the kernel defines paths to a visible SCSI LUN. The vml path hosts the LUN ID, GUID and partition number information and this is also stored in the volumes VMFS header. As well the header contains a UUID signature but this is not the VMFS UUID.

If we use hexdump as illustrated below we can see these elements in the VMFS header directly.

 

 

 

[root@vh1 ]# hexdump -C -s 0x200000 -n 256 /vmfs/volumes/49716cd8-ebcbbf9a-6792-000d60d46e2e/.vh.sf
00200000  5e f1 ab 2f 04 00 00 00  1f d8 6c 71 49 9a bf cb  |^   /  lqI      | <- VMFS UUID
00200010  eb 92 67 00 0d 60 d4 6e  2e 02 00 00 00 73 73 32  |  g  ` n     ss2| <- Volume Name
00200020  2d 63 73 74 61 72 2d 7a  73 30 2e 32 00 00 00 00  |-cstar-zs0.2    |
00200030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
*
00200090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 02 00  |                |
002000a0  00 00 00 10 00 00 00 00  00 d8 6c 71 49 01 00 00  |          lqI   |
002000b0  00 d8 6c 71 49 9a 89 75  60 1c ad 00 0d 60 d4 6e  |  lqI  u`    ` n| <- SEG UUID
002000c0  2e 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
002000d0  00 00 00 01 00 20 00 00  00 00 00 01 00 00 00 00  |                |
002000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |

And of course we can not leave out the partition entry block data for the device.

hexdump -C -n 256 /dev/sdf1

00000000  fa b8 00 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |                |
00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |   |         !  |
00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |    8 u        u|
00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 01 8b  |         |   t  |
00000040  4c 02 cd 13 ea 00 7c 00  00 eb fe 00 00 00 00 00  |L     |         |
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |
*
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 02  |                |
000001c0  03 00 fb fe ff ff 80 00  00 00 72 ef bf 5d 00 00  |          r  ]  | Type Start End
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |                |

With this detailed information it is possible to solve some common security issues with VMware stores like volume deletion and unintentional LUN ID changes.

Recently VMware added a some what useful command line tool named vmfs-undelete which exports metadata to a recovery log file which can restore vmdk block addresses in the event of deletion. It’s a simple tool and at present it’s experimental and unsupported and is not available on ESXi. The tool of course demands that you were proactive and ran it’s backup function in order to use it. Well I think this falls well short of what we need here. What if you have no previous backups of the VMFS configuration, so we really need to know what to look for and how to correct it and that’s exactly why I created this blog.  

The volume deletion event is quite easy to fix and thats simply because the VMFS volume header is not actually deleted. The partition block data is what gets trashed and you can just about get way with murder when it comes to recreating that part. Within the first 128 sectors is the peice we need to fix. One method is to create a store with the same storage volume and then block copy the partition to a file which can be block copied to the deleted device partition data blocks and this will fix the issue.

For example we create a new VMFS store on the same storage backing with the same LUN size as the original and it shows up as a LUN with a device name of /dev/sdd we can use esxcfg-vmhbadevs -m to find it if required

The deleted device name was /dev/sdc

We use the dd command to do a block copy from the new partition to a file or even directly in this case.

Remember to back it up first!

dd if=/dev/sdc of=/var/log/part-backup-sdc-1st.hex  bs=512 count=1

then issue

dd if=/dev/sdd of=/dev/sdc bs=512 count=1

or 

dd if=/dev/sdd of=/var/log/part-backup-sdd.hex bs=512 count=1

dd if=/var/log/part-backup-sdd.hex of=/dev/sdc bs=512 count=1

I personally like using a file to perform this function as this becomes a future backup element which you can move to a safe location. The file can actually be edited with other utilities to provide more flexibility. e.g. hexedit etc. Addtitionally you could use fdisk to directly edit the partition table and provide the correct start and end addresses. This is something you should only do if you are well versed in it’s usage.

As as an additional level of protection we could even include making backups of the vh.sf metadata file and the VMFS header.

cp /vmfs/volumes/49716cd8-ebcbbf9a-6792-000d60d46e2e/.vh.sf /var/log/vh.sf.bu

dd if=/dev/sdc of=/var/log/vmfsheader-bu-sdc.hex bs=512 count=4096

This would grant the ability for support to examine the exact details of the VMFS configuration and potentially allow recovery from more complex issues.

One of the most annoying security events is when a VMFS LUN get’s changed inadvertently. If a VMFS volume LUN ID changes and is presented to an ESX host then the presented volume will be treated as a potential snapshot LUN. If this event occurs and the ESX servers advanced LVM parameter settings are at default the ESX host will not mount the volume. This behaviour is to prevent the possibility of corruption and downing the host since it can not determine which VM metadata inventory is correct.

If you are aware that the LUN ID has changed then the best course of action is to re-establish the correct LUN ID at the storage server first and rescan the affected vmhba’s. This is important because if you need to resignature the VMFS volume it will also require that the VM’s be imported back into inventory. Virtual Center logging and other various settings will be lost when this action is performed. This is a result of now having an incorrect UUID between the metadata, mount location and the vmx file UUID value.

If the storage change cannot be reverted back then a VMFS resignature method is the only option for reprovisioning a VMFS volume mount.

This is invoked by setting the LVM.DisallowSnapshotLun = 0 and LVM.EnableResignature = 1 and these should reverted back once the VMFS resignature operation is complete.

Regards,

Mike

Site Contents: © 2009  Mike La Spina

Sun’s S7000 Storage Systems with Fishworks is Really Awesome

 

I came across Brendan Gregg’s blog and found this entry http://blogs.sun.com/brendan/entry/unusual_disk_latency to be really interesting. He demonstrates how powerful Fishworks analytics really are. This is a great example of what capability Dtrace and the Fishworks application brings to the storage platform. It’s a cool demo and I encourage you to have a look at what this Open Storage product can do.

Regards,

 

Mike

Site Contents: © 2009  Mike La Spina

ZFS Snapshot Rollup Bash Script

Site Contents: © 2008  Mike La Spina