Automating vSphere ESX4 host installations
Automating ESX 4 installations is a great way to save time and to provide a method of server recovery in the event of hardware or software failure. It creates consistent high quality repeatable installations that can be quickly modified to handle new and changing hardware. The process can also provide some detailed levels of VMware ESX server instance documentation. This blog will discuss how the process works and how-to create the required elements for you to implement your own automated process.
The vSphere ESX 4 install process uses an updated linux boot release commonly referred to as Syslinux. This Syslinux release version 3.63 supports a variety of popular protocols to facilitate a remote central install repository. FTP, HTTP, NFS and gPXE are all available options for provisioning network attachment to a remote install repository.
From the Syslinux boot process vSphere launches it’s initrd.img kernel instance which is a custom VMware/Linux kernel containing a multitude of VMware ESX 4 drivers and components. The custom drivers allow for a more closely integrated VMware ESX 4 install process that targets an improved ESX 4 server configuration result.
The custom VMware kernel incorporates Linux kickstart scripting functionality to invoke automated installations. The script location is defined as part of the Syslinux functionality and is available as a menu at boot time. A control file located on the boot media provides these variable control elements. Depending on the media type Syslinux uses a respective cfg file to implement this function. The various available Syslinux boot methods that I am aware of are USB, CD, DVD, PXE and gPXE. In this blog I will demonstrate an ISO CDROM method to perform the automated boot cycle. Any of the boot methods mentioned will all work and have varying levels of complexity to achieve.
The ISO CD and DVD based Syslinux configuration uses a config file named isolinux.cfg, USB boot images would use syslinux.cfg as well as gPXE based boot services can use either depending on the final gPXE target image.
Here is a example and description of the boot time menu functional elements for the isolinux.cfg ISO based file in this demonstration.
default Default
gfxboot bootlogo
prompt 1
timeout 3000
label Default
menu default
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/default.cfg
label vh0
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/vh0.cfg
label vh1
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/vh1.cfg
This cfg file provides three menu choices of default, vh0 and vh1. It will invoke the default after 300 seconds (timeout 3000 is not a typo) or you can manually select the other menu items. The “ks=” append option entry can also be one of file://… cdrom://… ftp://… nfs://… usb and UUID:ID/… The initrd.img element is an ESX initial ram disk image and it needs to version match your repository for a successful install process.
This isolinux.cfg file relies on DHCP to provide IP services. If DHCP is not an option you can use static methods to provision the same by passing the IP specific info into the initial ram disk image.
Here is an example of static IP parameters within the isolinux.cfg file.
label vh1
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ksdevice=eth0 ip=10.10.0.1 netmask=255.255.255.0 gateway=10.10.0.254 nameserver=10.10.0.253 ks=http://vc.laspina.ca:8088/esx/4.0/vh1.cfg
Static IP parameters can also be defined in the kickstart ks file but then the ks file would need to be locally available in order for it to work.
Let’s now look at how we can create a remote repository based, automated ESX 4 ISO CD installation boot image. My tool of choice for this process was ISOMagic of which you can make images 300MB or less in size for free. Of course you can use others like PowerISO which is also one of my favourites. The first step in the process is to open the vSphere installation ISO and delete all but the highlighted files show in the graphic. While you could leave them intact I prefer to remove elements that are not required as this can be a template for USB or other boot images.
The ISO boot method is quite simple, all we need to do is create a text isolinux.cfg file based on the example show previously and drag it to the MagicISO window onto the isolinux folder. Once the file is replaced we can use SaveAs an ISO to the name of your choice then burn it or mount it on your server’s ILOM interface. I plan on making a USB based image later so stay tuned for that in another blog entry.
Be wary of using a windows based text editors as they do work well with Unix based text processing operations since it adds invisible characters to the edited files that will cause some of the unix processes to fail. I normally use Ultraedit to edit the Unix targeted files since it has a function that allows you to convert and save in Unix file format.
To provision an install repository is a matter of choice, you can use any of the many different hosts that can serve one of the supported protocols. If your going to use an http repo you will need to take note of Mime types that may not be defined on the web service of choice. In this blog example we are going to define an http based repo on a Virtual Center Server (VC).
Create a base directory on the VC to host the install repo and extract the ESX 4 ISO to an appropriately named subfolder.
e.g. My base is D:VMwareRepo and the subfolder is esx4.0
We need to setup an IIS service instance on the VC and create a WEB site on port 8088. Do not use the default port 80 as it will conflict with other VC services.
We assign our base repo directory to this site and allow directory browsing.
The addition of MIME type pkl is required, right mouse -> Properties on the IIS instance within your Computer Manager MSC
That’s all you need for provisioning an http repo with IIS on your VC. Once you have a repo defined and running make sure you can browse it using your favourite browser.
As an added layer of security I only allow ESX console IP interfaces of a specific subnet on the repo site. Here is an example screen shot. Remember to check browsing availability before you enable any subnet restrictions.
The next step is the most involved and interesting part of the process. Let’s use an example script named vh1.cfg to examine and discuss one of my scripted processes. The script is normally stored in your repo e.g. D:VMwareRepoesx4.0vh1.cfg.
##########################################################
# ESX 4 Kickstart installation script
# © Mike La Spina – Ubiquitous Talk
# File name: vh1.cfg
##########################################################
# Install or Upgrade
install url http://vc.laspina.ca:8088/esx/4.0
“We first define a source for our ESX 4 install files, the possible methods are file://, ftp:// nfs:// cdrom://, take note that you can use a custom port like 8088.”
##########################################################
#Network install type
network –bootproto=static –ip=10.20.0.1 –gateway=10.20.0.254 –netmask=255.255.255.0 –hostname=vh1.laspina.ca –nameserver=10.20.0.200 –device=vmnic0 –addvmportgroup=0
“This defines our final static IP on the vswif0 management interface of vmnic0 (aka the Service Console) and addvmportgroup=0 disables the default VM network creation.”
##########################################################
# root Password
rootpw changeme
“Obviously this sets a root password, however I do not recommend you use an encrypted password method as it can be reversed with simple tools. It is better to just immediately change it to a secured one.”
##########################################################
# Authconfig
authconfig –enableshadow –enablemd5
“Enables a local password shadow file and stores the passwords as MD5 hashes.”
##########################################################
# Regional Settings
keyboard us
timezone America/Winnipeg
“Obvious”
##########################################################
# Firewall settings
firewall –allowOutgoing
“Obvious”
##########################################################
# Enable reboot after script
reboot
“Obvious”
##########################################################
# Boot Config
bootloader –location=mbr
“Installs a master boot record on the firstdisk by default”
##########################################################
# Disk Partitioning
clearpart –firstdisk –overwritevmfs
part /boot –fstype=ext3 –size=250 –onfirstdisk –asprimary
part vh1-local0 –fstype=vmfs3 –size=16384 –grow –onfirstdisk
part None –fstype=vmkcore –size=100 –onfirstdisk
# Create the vmdk on the cos vmfs partition.
virtualdisk cos –size=8192 –onvmfs=vh1-local0
# Partition the virtual disk.
part / –fstype=ext3 –size=4096 –grow –onvirtualdisk=cos
part swap –fstype=swap –size=256 –onvirtualdisk=cos
“Creates the ESX boot, core dump and VMFS partitions as we would expect. We have new partition function available, we can now create our Console Operating System on a vmdk. Here we are defining a virtual disk vmdk named cos on VMFS volume vh1-local0. Very cool, the ESX kernel can now snapshot itself. Take note of the –overwritevmfs option, this can wipe out any perfectly healthy production VMFS volume, I recommend that you remove this option once your testing cycle is complete and only add it to destroy a confirmed existing targeted VMFS volume.”
##########################################################
# Accept the EULA
vmaccepteula
“Obvious”
##########################################################
#
%post –interpreter=bash
“In pre VMware ESX 4 releases it was not possible to directly configure most of the ESX config elements. ESX 4’s initrd.img contains all most everything to need to configure the host without creating any special first time startup scripts on the systems reboot cycle. Now if we can just use vimsh directly … it’s still very cool!”
##########################################################
# Allow hostd etc. some time to load
/usr/bin/sleep 90
“We need to create a delay to aloow the VMware processes some time to load, this is required in order to run vim commands. We could have grepped the output of ps but it still would not tell us its ready to accept work thus a simple delay will do. I am using 90 seconds here but some slower servers may require more. “
##########################################################
# Enable Kerberos Auth
/usr/sbin/esxcfg-auth –enablead –addomain=domain.local –addc=domain.local
“Sets up the Linux Plugable Authentication Module (PAM) to autheticate users against a Window Domain over Kerberos”
##########################################################
# Add Groups and Users
/usr/sbin/groupadd -g 5000 lg-esxsu
/usr/sbin/useradd -u 501 -G lg-esxsu super1
/usr/sbin/useradd -u 502 -G lg-esxsu super2
/usr/sbin/useradd -u 503 -G lg-esxsu super3
“Create a local group which will allow members full admin rights to the ESX ha-folder-root and create three user id’s that are members of the group. These user id’s will be authenticated against the domain.local Windows Domain”
##########################################################
# NTP time config
esxcfg-firewall -e ntpClient
echo restrict default kod nomodify notrap noquerynopeer > /etc/ntp.conf
echo restrict 127.0.0.1 >> /etc/ntp.conf
echo server 10.20.0.200 >> /etc/ntp.conf
echo driftfile /var/lib/ntp/drift >> /etc/ntp.conf
/sbin/chkconfig –level 345 ntpd on
/etc/init.d/ntpd start
“Enable outgoing NTP client port access and build the ntp.conf file to use a Windows Domain time service, this is important for kerberos authentication. Start the NTP client daemon.”
# Create vSwitch0, VMMotion1 and the Service Console port group
/usr/sbin/esxcfg-vswitch -a vSwitch0:64
/usr/sbin/esxcfg-vswitch -A “Service Console” vSwitch0
/usr/sbin/esxcfg-vswitch -A Network-00 vSwitch0
/usr/sbin/esxcfg-vswitch -A VMMotion1 vSwitch0
/usr/sbin/esxcfg-vswitch -p VMMotion1 -v 600 vSwitch0
/usr/sbin/esxcfg-vswitch -p Network-00 -v 700 vSwitch0
/usr/sbin/esxcfg-vswitch -L vmnic1 vSwitch0
/usr/sbin/esxcfg-vswitch -L vmnic0 vSwitch0
/usr/sbin/esxcfg-vswitch -M vmnic0 vSwitch0 -p “Service Console”
/usr/sbin/esxcfg-vswitch -M vmnic0 vSwitch0 -p VMMotion1
/usr/sbin/esxcfg-vswitch -M vmnic1 vSwitch0 -p Network-00
“Defines vSwitch0 for the Service Console and the default gateway IP, vMotion on VLAN 600 with a vMotion IP and create a VM Network-00. This vSwitch will be further configured for a custom standby adapter during the initial reboot with a post config script”
# Create vSwitch1 for iSCSI traffic
/usr/sbin/esxcfg-vswitch -a vSwitch1:64
/usr/sbin/esxcfg-vswitch -A iSCSI_Initiator vSwitch1
/usr/sbin/esxcfg-vswitch -p iSCSI_Initiator -v 500 vSwitch1
/usr/sbin/esxcfg-vswitch -L vmnic3 vSwitch1
/usr/sbin/esxcfg-vswitch -L vmnic2 vSwitch1
/usr/sbin/esxcfg-vswitch -M vmnic3 vSwitch1 -p iSCSI_Initiator
/usr/sbin/esxcfg-vswitch -M vmnic2 vSwitch1 -p iSCSI_Initiator
/usr/sbin/esxcfg-vmknic -a -i 10.20.10.64 -n 255.255.255.0 iSCSI_Initiator
“Defines vSwitch1 for iSCSI on VLAN 500”
##########################################################
#
%post –interpreter=bash
# Create post config script
cat << EOF > /etc/rc3.d/S99postconf
#!/bin/bash
“Some configuration elements still require a post run this statement sends the follow on text to the s99postconf file until the EOF marker is met.”
# Enable TCP outgoing kerberos, there are issues with udp and enable blockOutgoing
/usr/sbin/esxcfg-firewall -–openport 88,tcp,out,KerberosClientTCP
/usr/sbin/esxcfg-firewall –blockOutgoing
“Seriously, this is important, udp kerberos port 88 is defaulted with ‘esxcfg-auth –enablead ..’, bad default! Also the VWware Kerberos client default uses tcp first and this needs to be fixed as it does not comply with RFC 4120. Even if this works why would we risk dropping an auth packet since any fragged udp packet would be dropped over VPN’s etc. Also turn on the outgoing firewall rules that were disabled previously.”
# Enable VMotion on the VMKernel Interface
/usr/bin/vmware-vim-cmd hostsvc/vmotion/vnic_set vmk1
“Enables vMotion on vmkernel interface 1”
# Define Active and Standby failover for shared vSwitche0
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 ‘Service Console’ –nicorderpolicy-active vmnic0 –nicorderpolicy-standby vmnic1
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 VMMotion1 –nicorderpolicy-active vmnic0 –nicorderpolicy-standby vmnic1
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 Network-00 –nicorderpolicy-active vmnic1 –nicorderpolicy-standby vmnic0
“Here we are overriding our adapter team for vSwitch0 so that we can separate our active traffic on the two adapters while maintaining failover capability.”
# Grant the group named lg-esxsu admin permission to ha-folder-root
/usr/bin/vmware-vim-cmd vimsvc/auth/entity_permission_add vim.Folder:ha-folder-root lg-esxsu true Admin true
“Enables any member of the local group lg-esxsu Administrator permissions to the VMware host”
# Reset system to normal boot mode
echo “Removing automated post script.”
rm /etc/rc3.d/S99postconf
EOF
chmod +x /etc/rc3.d/S99postconf
“Obvious”
As you can see the process is quite involved, however the benefits are outstanding. I can build or recover an ESX 4 host in 10 minutes or less and I can reconfigure it to a different target with ease.
Hope you found the entry usefull and interesting.
Regards,
Mike
Site Contents: © 2009 Mike La Spina
Subscribe in a reader
Follow me on Twitter