Ubiquitous Talk

7 Comments

• NiTRo says:

  January 6, 2010 at 4:47 PM

  Hi Mike, I was using your great vmfs metadata backup script with ESX 3 but now i’m working on ESXi 4 and i still would like to use it. But i also would like to be able to restore in case of corruption but the dd command give me an error when i try (dd: can’t open ‘/dev/disks/mpx.vmhba1:C0:T1:L0:1′: Function not implemented) can you help me ? Regards.
• Mike La Spina says:

  January 6, 2010 at 11:32 PM

  Hi Raphael,

  I believe the reason for the “function not implemented” message is a bit misleading. I suspect the real cause is most likely from a read only file system mount on /dev. You cannot create any files on that mount point. To change this behavior would be very challenging and completely unsupported. I would suggest using an automated build of an ESX 4 host to recover any ESXi 4 vmfs partition deletion. Then you could access the vmfs partition over the /dev/sdxx devices there. No need to license it since it would only be a temporary instance.

  Regards,

  Mike
• NiTRo says:

  January 7, 2010 at 3:56 AM

  Mike, thanks a lot for your answer. You think there is not chance to find the real device path to use in dd ?
• Frederik says:

  March 19, 2010 at 12:18 PM

  Hi Mike,

  thank you for the nice blog. I liked the DDRdriveX1 review. It’s a nice device and it seems to do it’s job well. Unfortunately it’s rather steep price keeps it out of the soho realm. Which brings me to my question. esx(i) and opensolaris based storage. I currently run esxi hosted vm’s from an nfs b104 based backend. Works like a charm. Most important I understand why there’s no problem (data loss or consistancy) in case of a storage failure. All data is committed safely to disk.
  Now a recent (b125 or later) iscsi comstar based backend. I know you prefer iscsi for your esx setups. My concerns are possible dataloss or corruption in case of storage failure (power, panic, loss of paths etc) when using iscsi due to caching. Recent discussions on the storage or zfs mailing lists didn’t led to a clear recommandation. To be clear, I don’t have nor want to disable the zil. And at the moment I don’t have an slog added to my pools.
  I tested two iscsi scenarios. A vmfs3 based on a single lun and a windows 2003 vm using two rdm lun’s. The same vm was also used on the vmfs3 datastore. The lu’s where created from based /dev/RDSK/pool/vols (the raw ones not the cached DSK version). Checking the lu’s showed that the writeback cache was enabled.
  I did run some tests with a java program which used sql server and did do a lot of file reading and writing apart from the sql server io’s. When using the rdm backed vm I did see zil usage (with zilstat from Richard Elling) when the sql part got executed. When using the vmfs3 backed vm I never saw zil usage, what was clearly visible was the default 30 second flush.
  What can go wrong with this setup? Can the database in the vm lose transactions when the power is cut before the 30 seconds elapse? Should the writeback cache be disabled? How big is the risk of losing or corrupting the vmfs3 in case of a storage failure?

  These are questions a lot of people have on their mind but no clear answers can be found for in the mailing lists nor in the various blogs.

  Thank you for your time,

  Frederik
• Mike La Spina says:

  March 19, 2010 at 4:09 PM

  Hi Frederik,

  Thanks for the acknowledgment.
  You have a good question that many people are concerned about and very few will give you a confident answer on.

  For the soho world there is no real risk since the data volume movement is so low that the probability of data loss is not a concern. All modern databases have the ability to recover from incomplete transactions and some can even deal with transactions that are missing but flagged as committed. I do allow write caching at the LUN level for iSCSI volumes in my soho/lab environment. I have seen multiple power outages with MSSQL 2005 running on a VM against my iSCSI targets with caching on and have not dealt with a corruption event to date. Completely disabling ZIL is only something I would only do if I have a UPS protecting the storage head. With NFS as a VMware shared store you will have more writes that require your ZIL protection since VMware always writes with the O_Sync flag when using NFS so more caution may be warranted. I would say that vmfs3 will survive in almost all crash events when the IO load is low and your iSCSI target LUN caching is enabled. Keep in mind that ZIL is active even when the LUN cache is on which is very safe as far as the soho world goes. I have hard failed a ZFS based NFS heads multiple times and have not seen any failure that rendered a share unusable But I have killed some data in that testing process and it was very minor.

  If a crash occurs and the cache flush event was not complete then that data will be dropped however this is no different than the network ring buffers being dropped at a crash event.

  Of course in a business environment it would not be advisable to disable your ZIL functionality in any scenario.

  In closing always remember that “A ZFS snapshot is your friend” when is comes to a recovery job.

  Regards,

  Mike
• NiTRo says:

  April 5, 2010 at 10:59 AM

  Hi Mike, i found the trick about the “function not implemented” issue in this kb : http://kb.vmware.com/kb/1008886 you need “conv=notrunc” parameter to get dd works
• Mike La Spina says:

  April 5, 2010 at 9:05 PM

  Great find NiTro!
  Thanks for the info.