Ubiquitous Talk

I have updated my ESX servers manually many times and I find the process to say at the least is “annoying” so I decided to change it to an http based method with a modified patch configuration. I found that it really works well.

I did some searching prior to the method I settled on and found this blog which is quite good.

http://virtrix.blogspot.com/2007/03/vmware-autopatching-your-esx-host.html

I felt it does have some issues and I wanted to avoid running custom scripts on the server side. So here is what I build for my patch management solution.

Using the standard tools on the ESX3 server of cron jobs and esxupdate I created the following on my servers.

Define a new cron entry for running esxupdate every first Sunday of the month.
I chose to create a separate entry avoiding the esx installed ones for safety reasons. 

Edit the file /etc/crontab and add the line in bold as show. 

[root@yourserver /]nano /etc/crontab

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
30 3 1 * 0 root run-parts /etc/cron.esxupdate

Create a new directory for the esxupdate cron run part as follows.

[root@yourserver /]mkdir /etc/cron.esxupdate

Create a new command file for the esxupdate cron run part and add the text lines as follows.

[root@yourserver /]nano /etc/cron.esxupdate/esxpatch

esxupdate -r http://yourhttpserver.fq.dns:8088/esx3.all update

Set the file as executable using chmod

chmod 755 /etc/cron.esxupdate/esxpatch

Allow outgoing traffic from the server using the esxcfg-firewall script as follows.

[root@yourserver /]esxcfg-firewall –allowoutgoing

Create an http server to host the patch repository.
I built an IIS server on my Virtual Center 3 server on port 8088 as follows

Install IIS (Be sure to secure it).

Create a patch folder named C:VMWarePatches and set your IIS home directory as follows.

 

 
You will need to add the following content types from a right mouse click-> properties of the IIS server definition within the Computer Manager mmc as shown.

Content types .hdr and .info need to be present.

 

 
Download the patches you require and extract them to c:vmwarepatches in the usual manner for repo based deployment. For example I downloaded the available esx-upgrade-from-esx3-3.0.2-61618.tar.gz upgrade file and extracted it to c:vmwarepatches which leaves a directory named 61818 in the http distro folder.

Rename this directory to esx3.all

Now this is were things get interesting as we are going to customize this patch package to include the latest additional patches that are available.

VMWare is using the installation and update functionally of Redhat rpm, yum and esxupdate scripts which means we can modify the repo source by conforming to the existing configuration rules. This configuration of the repo is based on xml tags of which I modified three tag element types.

1) info

The descriptor xml file found in the root of the update package which contains the descriptions of various patches in the package and looks like the following.

<descriptor version=”1.0″>
  <vendor>VMware, Inc.</vendor>
  <product>VMware ESX Server</product>
  <release>3.0.2-62488</release>
  <releasedate>Fri Nov 2 05:01:31 PDT 2007</releasedate>
  <summary>3.0.2 Update 1 of VMware ESX Server</summary>
  <description>This is 3.0.2 Update 1 rollup to Nov 2nd,2007 of VMware ESX Server.
It contains:
3.0.2-52542  Full 3.0.2 release of VMware ESX Server
ESX-1001724  Security bugs fixed in vmx rpm.
ESX-1001735  To update tzdata rpm.
…
ESX-1002424  VMotion RARP broadcast to multiple vmnic.
ESX-1002425  VMware-hostd-esx 3.0.2-62488
ESX-1002429  Path failback issue with EMC iSCSI array.
</description>

This xml file is used by the esxupdate script and can be modified to support additional patches. The parts we need to add are some descriptor text element data so that we know what it covers. I added ESX-1002424, 25 and 29 to the descriptor tag. And edited the headings date to reflect its current value.

2) rpms

The second element type I changed was the rpmlist tag. It looks like the following which is only a partial.

  <rpmlist>
    <rpm arch=”i386″ rel=”8.37.15″ ver=”2.4″>kernel-utils</rpm>
    <rpm arch=”i686″ rel=”47.0.1.EL.62488″ ver=”2.4.21″>kernel-vmnix</rpm>
    <rpm arch=”i386″ rel=”66″ ver=”1.2.7″>krb5-libs</rpm>
    <rpm arch=”i386″ rel=”11″ ver=”1.1.1″>krbafs</rpm>
    <rpm arch=”i386″ rel=”3vmw” ver=”1.1.22.15″>kudzu</rpm>
    <rpm arch=”i386″ rel=”70RHEL3″ ver=”0.1″>laus-libs</rpm>
    <rpm arch=”i386″ rel=”12″ ver=”378″>less</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-perftools</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-uwlibs</rpm>
    <rpm arch=”i386″ rel=”62488″ ver=”3.0.2″>VMware-esx-vmx</rpm>
    <rpm arch=”i386″ rel=”61818″ ver=”3.0.2″>VMware-esx-vmkernel</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-vmkctl</rpm>
    <rpm arch=”i386″ rel=”55869″ ver=”3.0.2″>VMware-esx-tools</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-srvrmgmt</rpm>
    <rpm arch=”i386″ rel=”52542″ ver=”3.0.2″>VMware-esx-scripts</rpm>
  </rpmlist>
When you extract a newer esx3 patch it will contain its own descriptor rpmlist tags etc. and you can update this major upgrade list to include those new elements. You need to replace the old tag info if it exists which it usually will. For example the bold entry above for VMware-esx-vmkernel needs to be completely updated by removing the old tag values like rel=61818 and insert the new value of rel=62488. I simply searched for the tag name VMware-esx-vmkernel deleted the line and inserted the updated one. I will eventually write a script to do the config edits, but for now these hacks will do the trick.

3) nodeps

The third element I changed will not normally be necessary but in this case there is a bug in this upgrade package that did not account for an rpm option value of –U which means upgrade an rpm if present. The descriptor included a tag named nodeps which I don’t think works well when the rpm already exists of the same version so we need to remove the tag all together for previous base installs above version esx 3.0.0.

I deleted these tags for my config.

<nodeps>
<rpm>kbd</rpm>
<rpm>nfs-utils</rpm>
</nodeps>

In addition to the descriptor xml file edits we will need to copy the new rpm files into the esx3.all folder, remove the old ones, edit the header.info file found in the headers sub folder, copy the new hdr files to the headers folder and remove the old ones in the headers folder.

For example in adding ESX-1002424 I copied

From c:vmwarepatchesesx-1002424

Files

VMware-esx-apps-3.0.2-62488.i386.rpm
VMware-esx-vmkernel-3.0.2-62488.i386.rpm
VMware-esx-vmx-3.0.2-62488.i386.rpm

To c:vmwarepatchesesx3.all

From c:vmwarepatchesesx-1002424headers

Files

VMware-esx-apps-0-3.0.2-62488.i386.hdr
VMware-esx-vmkernel-0-3.0.2-62488.i386.hdr
VMware-esx-vmx-0-3.0.2-62488.i386.hdr

To c:vmwarepatchesesx3.allheaders

You should delete any older file versions matching the prefixes portions of the newer files to avoid confusion. It will still function if you leave the old ones but it’s not a best practice.

The last step is to edit the header.info file, search for the text lines of the rpm file prefix and replace the info lines with the newer ones found from the updated patch header.info file.

For example search for text VMware-esx-vmx in the master header.info file which looks like the following partial example.

0:openssh-3.6.1p2-33.30.13vmw.i386=openssh-3.6.1p2-33.30.13vmw.i386.rpm
0:net-snmp-libs-5.0.9-2.30E.20.i386=net-snmp-libs-5.0.9-2.30E.20.i386.rpm
0:libtermcap-2.0.8-35.i386=libtermcap-2.0.8-35.i386.rpm
0:VMware-esx-vmx-3.0.2-62488.i386=VMware-esx-vmx-3.0.2-62488.i386.rpm
0:acl-2.2.3-1.i386=acl-2.2.3-1.i386.rpm
0:dev-3.3.12.3-1.i386=dev-3.3.12.3-1.i386.rpm
0:ethtool-1.8-3.3.i386=ethtool-1.8-3.3.i386.rpm

Replace it completely with the newer header.info text entry.

I tested my configuration on serveral base install versions and is was nice to see that on an esx 3.0.2-61818 it only downloaded and installed the addtional patches that were added to the master repo esx3.all

You should keep in mind that based on the cron schedule your VM’s may need to go down on that host and you must plan those outage times and gracefully shutdown where applicable.

Till next time …  happy esx patching.

Regards,

Mike

Tags: esx, method, patch, VMware

This entry was posted on Sunday, December 7th, 2008 at 11:45 PM and is filed under Security, VMware. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Site Contents: © 2008  Mike La Spina