Securing COMSTAR and VMware iSCSI connections

Connecting VMware iSCSI sessions to COMSTAR or any iSCSI target provider securely is required to maintain a reliable system. Without some level of initiator to target connection gate keeping we will eventually encounter a security event. This can happen from a variety of sources, for example a non-cluster aware OS can connect to an unsecured VMware shared storage LUN and cause severe damage to it since the OS has no shared LUN access knowledge.  All to often we make assumptions that security is about confidentiality when it is actually more commonly about data availability and integrity which will both be compromised if an unintentional connection were to write on a shared LUN.

At the very minimum security level we should apply non-authenticated named initiator access grants to our targets. This low security method defines initiator to target connection states for lower security tolerant environments. This security method is applicable when confidentiality is not as important and security is maintained with the physical access control realm. As well it should also coincide with SAN fabric isolation and be strictly managed by the Virtual System or Storage Administrators. Additionally we can increase access security control by enabling CHAP authentication which is a serious improvement over named initiators. I will demonstrate both of these security methods using COMSTAR iSCSI Providers and VMware within this blog entry.

Before we dive into the configuration details lets examine how LU’s are exposed. COMSTAR controls iSCSI target access using several combined elements. One of these elements is within the COMSTAR STMF facility where we can assign membership of host and target groups. By default if we do not define a host or target group any created target will belong to an implied ALL group. This group as we would expect grants any connecting initiator membership to the ALL group assigned LUN’s. These assignments are called views in the STMF state machine and are a mapping function of the Storage Block Driver service (SBD) to the STMF IT_nexus state tables.

This means that if we were to create an initiator without assigning a host group or host/target group combination, an initiator would be allowed unrestricted connectivity to any ALL group LUN views and possibly without any authentication at all. Allowing this to occur would of course be very undesirable from a security perspective in almost all cases. Conversely if we use a target group definition then only the initiators that connect to the respective target will see the LUN views which are mapped on that target definition instance.

While target groups do not significantly improve access security it does provide a means controlling accessibility based on the definition of interface connectivity classes which in turn can be mapped out on respective VLAN priority groups, bandwidth availability and applicable path fault tolerance capabilities which are all important aspects of availability and unfortunately are seldom considered security concepts in many architectures.

Generally on most simple storage configurations the use of target groups is not a requirement. However they do provide a level of access control with LUN views. For example we can assign LUN views to a target group which in turn frees us from having to add the LUN view to each host group within shared LUN configurations like VMware stores. With combination’s of host and target groups we can create more flexible methods in respect to shared LUN visibility. With the addition of simple CHAP authentication we can more effectively insulate target groups. This is primarily due to the ability to assign separate CHAP user and password values for each target.

Lets look at this visual depiction to help see the effect of using target and host groups.

COMSTAR host and target view depiction

In this depiction any initiator that connects to the target group prod-tg1 will by default see the views that are mapped to that target groups interfaces. Additionally if the initiator is also a member of the host group prod-esx1 those view mapping will also be visible.

One major difference with target groups verses the all group is that you can define LU views on mass to an entire class of initiator connections e.g. a production class. This becomes an important control element in a unified media environment where the use of VLANs separates visibility. Virtual interfaces can be created at the storage server and attached to VLANs respectively. Target groups become a very desirable as a control within a unified computing context.

Named Initiator Access

Enabling named initiator to target using unauthenticated access with COMSTAR and VMware iSCSI services is a relatively simple operation. Let’s examine how this method controls initiator access.

We will define two host groups, one for production esx hosts and one for test esx hosts.

# stmfadm create-hg prod-esx1

# stmfadm create-hg test-esx1

With these host groups defined we individually assign LU’s views to the host groups and then we define any initiator to be a member of one of the host groups to which it would only see the views which belong to the host group and additionally any views assigned to the default all group.

To add a host initiator to a host group, we must first create it in the port provider of choice which in this case is the iSCSI port provider.

# itadm create-initiator iqn.1998-01.com.vmware:vh1.1

Once created the defined initiator can be added to a host group.

# stmfadm add-hg-member -g prod-esx1 iqn.1998-01.com.vmware:vh1.1

An ESX host initiator with this iqn name can now attach to our COMSTAR targets and will see any LU views that are added to the prod-esx1 host group. But there are still some issues here, for example any ESX host with this initiator name will be able to connect to our targets and see the LUs. This is where CHAP can help to improve access control.

Adding CHAP Authentication on the iSCSI Target

Adding CHAP authentication is very easy to accomplish, we simply need to set a chap user name and secret on the respective iSCSI target. Here is an example of its application.

# itadm modify-target -s -u tcuid1 iqn.2009-06.target.ss1.1

Enter CHAP secret:
Re-enter secret:

The CHAP secret must be between 12 and 255 characters long. The addition of CHAP allows us to further reduce any risks of a potential storage security event. We can define an additional target and they can have a different chap user names and or secrets.

CHAP is more secure when used in a mutual authentication back to the source initiator which is my preferred way to implement it on ESX 4 (ESX 3 does not support mutual chap). This mode does not stop a successful one-way authentication from an initiator to the target, it allows the initiator to request that the target host system iSCSI services must authenticate back to the initiator which provides validation that the target is indeed the correct one. Here is an example of the target side initiator definition that would provide this capability.

# itadm modify-initiator -s -u icuid1 iqn.1998-01.com.vmware:vh1.1

Enter CHAP secret:
Re-enter secret:

Configuring the ESX 4 Software iSCSI Initiator

On the ESX 4 host side we need to enter our initiator side CHAP values.

ESX 4 iSCSI Mutual CHAP

 

Be careful here, there are three places we can configure CHAP elements. The general tab allows a global point of admin where any target will inherit those entered values by default where applicable e.g. target chap settings. The the dynamic tab can override the global settings and as well the static tab overrides the global and dynamic ones. In this example we are configuring a dynamically discovered target to use mutual (aka bidirectional) authentication.

In closing CHAP is a reasonable method to ensure that we correctly grant initiator to target connectivity assignments in an effort to promote better integrity and availability. It does not however provide much on the side of confidentially for that we need more complex solutions like IPSec.

Hope you found this blog interesting.

Regards,

Mike

Site Contents: © 2009  Mike La Spina

Creating USB based boot media for ESX 4 installs

As a follow on to my Automating vSphere ESX4 Host Installations blog I have detailed a howto create USB based boot media using syslinux 3.82 and the ESX 4 installation source files. The process is actually quite simple as we can create the bootable USB from a Windows system.  You can also do the same with extlinux but most people will have a Windows based management system so lets only focus on this Windows based method within this blog.

The first step is to ofcourse obtain a copy of the Syslinux 3.82 or higher zip package from  http://syslinux.zytor.com/ and extract to a  file store of your choice.

Prepare the media:

To prepare a USB memory stick we need to format it with a FAT32 file system. Windows explorer provides that functionallity with a simple right click on your USB device.

Format USB device

Generate a bootable media device:

Once formated we will need to open a cmd prompt and go to our syslinux file store and execute the following example.

Syslinux cmd prompt

In this example the syslinux win32 tool creates a grub based loader and boot sector on the USB memory device mapped to drive G: the tool also defines the syslinux directory using the -d option as the root path and this is where we will copy the ESX 4 initial ramdisk image file and some additional syslinux text menu files.  If your planning to use the usb device as a source for the ESX 4 packages then those files  e.g. the VMware directory etc. would need to be placed in the root directory of the usb device and not the syslinux directory. In this blog the usb device is only used to launch a remote source file install.

Copy menu and ESX 4 install files:

From the ESX 4 ISO or CD copy the isolinux directory to G: and rename it to syslinux also copy the build_numbler file to G:  additionally explore the downloaded syslinux file store and locate ..syslinuxcom32menumenu.c32, copy this file to the G:syslinux location, you may also want to copy vesamenu.c32 if you wish to checkout a GUI based menu. That’s really just eye candy on the requirements side but it can provide some cool background display capabilities.

Create your selectable boot time menu:

Now we are ready to create the syslinux.cfg configuration file in the syslinux directory.  Here is an example I created for this blog.

default menu.c32
prompt 0
timeout 9000
menu title ESX 4 Automated Install VC1 HTTP Repo

label Default
kernel vmlinuz
append initrd=initrd.img vmkopts=debugLogToSerial:1 mem=512M quiet ks=http://vc1.laspina.ca:8088/esx/4.0/default.cfg

label vh0
kernel vmlinuz
append initrd=initrd.img vmkopts=debugLogToSerial:1 mem=512M quiet ks=http://vc1.laspina.ca:8088/esx/4.0/vh0.cfg

label vh1
kernel vmlinuz
append initrd=initrd.img vmkopts=debugLogToSerial:1 mem=512M quiet ks=http://vc1.laspina.ca:8088/esx/4.0/vh1.cfg

Once your cfg file is created your ready to boot the USB device either on your server or over RDAC/ILOM interfaces,  select a server target from the menu and walk away.

Yes it’s that simple and easy to create USB bootable media for your ESX 4 installs.

Regards,

Mike





Site Contents: © 2009  Mike La Spina

Automating vSphere ESX4 host installations


Automating ESX 4 installations is a great way to save time and to provide a method of server recovery in the event of hardware or software failure. It creates consistent high quality repeatable installations that can be quickly modified to handle new and changing hardware. The process can also provide some detailed levels of VMware ESX server instance documentation. This blog will discuss how the process works and how-to create the required elements for you to implement your own automated process.

The vSphere ESX 4 install process uses an updated linux boot release commonly referred to as Syslinux. This Syslinux release version 3.63 supports a variety of popular protocols to facilitate a remote central install repository. FTP, HTTP, NFS and gPXE are all available options for provisioning network attachment to a remote install repository.

From the Syslinux boot process vSphere launches it’s initrd.img kernel instance which is a custom VMware/Linux kernel containing a multitude of VMware ESX 4 drivers and components. The custom drivers allow for a more closely integrated VMware ESX 4 install process that targets an improved ESX 4 server configuration result.

The custom VMware kernel incorporates Linux kickstart scripting functionality to invoke automated installations. The script location is defined as part of the Syslinux functionality and is available as a menu at boot time. A control file located on the boot media provides these variable control elements. Depending on the media type Syslinux uses a respective cfg file to implement this function. The various available Syslinux boot methods that I am aware of are USB, CD, DVD, PXE and gPXE. In this blog I will demonstrate an ISO CDROM method to perform the automated boot cycle. Any of the boot methods mentioned will all work and have varying levels of complexity to achieve.

The ISO CD and DVD based Syslinux configuration uses a config file named isolinux.cfg, USB boot images would use syslinux.cfg as well as gPXE based boot services can use either depending on the final gPXE target image.
Here is a example and description of the boot time menu functional elements for the isolinux.cfg ISO based file in this demonstration.

default Default
gfxboot bootlogo
prompt 1
timeout 3000

label Default
menu default
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/default.cfg

label vh0
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/vh0.cfg

label vh1
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ks=http://vc.laspina.ca:8088/esx/4.0/vh1.cfg

This cfg file provides three menu choices of default, vh0 and vh1. It will invoke the default after 300 seconds (timeout 3000 is not a typo) or you can manually select the other menu items. The “ks=” append option entry can also be one of file://… cdrom://… ftp://… nfs://…  usb and UUID:ID/… The initrd.img element is an ESX initial ram disk image and it needs to version match your repository for a successful install process.

This isolinux.cfg file relies on DHCP to provide IP services. If DHCP is not an option you can use static methods to provision the same by passing the IP specific info into the initial ram disk image.

Here is an example of static IP parameters within the isolinux.cfg file.

label vh1
kernel vmlinuz
append initrd=initrd.img mem=512M quiet ksdevice=eth0 ip=10.10.0.1 netmask=255.255.255.0 gateway=10.10.0.254 nameserver=10.10.0.253 ks=http://vc.laspina.ca:8088/esx/4.0/vh1.cfg

Static IP parameters can also be defined in the kickstart ks file but then the ks file would need to be locally available in order for it to work.

Let’s now look at how we can create a remote repository based, automated ESX 4 ISO CD installation boot image. My tool of choice for this process was ISOMagic of which you can make images 300MB or less in size for free. Of course you can use others like PowerISO which is also one of my favourites. The first step in the process is to open the vSphere installation ISO and delete all but the highlighted files show in the graphic. While you could leave them intact I prefer to remove elements that are not required as this can be a template for USB or other boot images.

ISOMagic SS

The ISO boot method is quite simple, all we need to do is create a text isolinux.cfg file based on the example show previously and drag it to the MagicISO window onto the isolinux folder. Once the file is replaced we can use SaveAs an ISO to the name of your choice then burn it or mount it on your server’s ILOM interface. I plan on making a USB based image later so stay tuned for that in another blog entry.

Be wary of using a windows based text editors as they do work well with Unix based text processing operations since it adds invisible characters to the edited files that will cause some of the unix processes to fail. I normally use Ultraedit to edit the Unix targeted files since it has a function that allows you to convert and save in Unix file format.  
To provision an install repository is a matter of choice, you can use any of the many different hosts that can serve one of the supported protocols. If your going to use an http repo you will need to take note of Mime types that may not be defined on the web service of choice. In this blog example we are going to define an http based repo on a Virtual Center Server (VC).

Create a base directory on the VC to host the install repo and extract the ESX 4 ISO to an appropriately named subfolder.
e.g. My base is D:VMwareRepo and the subfolder is esx4.0

We need to setup an IIS service instance on the VC and create a WEB site on port 8088. Do not use the default port 80 as it will conflict with other VC services.

IIS Repo Config 1

We assign our base repo directory to this site and allow directory browsing.

IIS Repo Config 2

The addition of MIME type pkl is required, right mouse -> Properties on the IIS instance within your Computer Manager MSC

IIS Mine Type SS

That’s all you need for provisioning an http repo with IIS on your VC. Once you have a repo defined and running make sure you can browse it using your favourite browser.
As an added layer of security I only allow ESX console IP interfaces of a specific subnet on the repo site. Here is an example screen shot. Remember to check browsing availability before you enable any subnet restrictions.

IIS Restricted Range

The next step is the most involved and interesting part of the process. Let’s use an example script named vh1.cfg to examine and discuss one of my scripted processes. The script is normally stored in your repo e.g. D:VMwareRepoesx4.0vh1.cfg.

##########################################################
# ESX 4 Kickstart installation script
# © Mike La Spina – Ubiquitous Talk
# File name: vh1.cfg

##########################################################
# Install or Upgrade
install url
http://vc.laspina.ca:8088/esx/4.0

“We first define a source for our ESX 4 install files, the possible methods are file://, ftp:// nfs:// cdrom://, take note that you can use a custom port like 8088.”

##########################################################
#Network install type
network –bootproto=static –ip=10.20.0.1 –gateway=10.20.0.254 –netmask=255.255.255.0 –hostname=vh1.laspina.ca –nameserver=10.20.0.200 –device=vmnic0 –addvmportgroup=0

“This defines our final static IP on the vswif0 management interface of vmnic0 (aka the Service Console) and addvmportgroup=0 disables the default VM network creation.”

##########################################################
# root Password
rootpw changeme

“Obviously this sets a root password, however I do not recommend you use an encrypted password method as it can be reversed with simple tools. It is better to just immediately change it to a secured one.”

##########################################################
# Authconfig
authconfig –enableshadow –enablemd5

“Enables a local password shadow file and stores the passwords as MD5 hashes.”

##########################################################
# Regional Settings
keyboard us
timezone America/Winnipeg

“Obvious”

##########################################################
# Firewall settings
firewall –allowOutgoing

“Obvious”

##########################################################
# Enable reboot after script
reboot

“Obvious”

##########################################################
# Boot Config
bootloader –location=mbr

“Installs a master boot record on the firstdisk by default”

##########################################################
# Disk Partitioning
clearpart –firstdisk –overwritevmfs
part /boot       –fstype=ext3    –size=250   –onfirstdisk  –asprimary
part vh1-local0  –fstype=vmfs3   –size=16384 –grow         –onfirstdisk
part None        –fstype=vmkcore –size=100   –onfirstdisk
# Create the vmdk on the cos vmfs partition.
virtualdisk cos –size=8192 –onvmfs=vh1-local0
# Partition the virtual disk.
part / –fstype=ext3 –size=4096 –grow –onvirtualdisk=cos
part swap –fstype=swap –size=256 –onvirtualdisk=cos

“Creates the ESX boot, core dump and VMFS partitions as we would expect. We have new partition function available, we can now create our Console Operating System on a vmdk. Here we are defining a virtual disk vmdk named cos on VMFS volume vh1-local0. Very cool, the ESX kernel can now snapshot itself. Take note of the –overwritevmfs option, this can wipe out any perfectly healthy production VMFS volume, I recommend that you remove this option once your testing cycle is complete and only add it to destroy a confirmed existing targeted VMFS volume.”

##########################################################
# Accept the EULA
vmaccepteula

“Obvious”

##########################################################
#
%post –interpreter=bash

“In pre VMware ESX 4 releases it was not possible to directly configure most of the ESX config elements. ESX 4′s initrd.img contains all most everything to need to configure the host without creating any special first time startup scripts on the systems reboot cycle. Now if we can just use vimsh directly … it’s still very cool!”

##########################################################
# Allow hostd etc. some time to load
/usr/bin/sleep
90

“We need to create a delay to aloow the VMware processes some time to load, this is required in order to run vim commands. We could have grepped the output of ps but it still would not tell us its ready to accept work thus a simple delay will do. I am using 90 seconds here but some slower servers may require more. “

##########################################################
# Enable Kerberos Auth
/usr/sbin/esxcfg-auth –enablead –addomain=domain.local –addc=domain.local

“Sets up the Linux Plugable Authentication Module (PAM) to autheticate users against a Window Domain over Kerberos”

##########################################################
# Add Groups and Users
/usr/sbin/groupadd -g 5000 lg-esxsu
/usr/sbin/useradd -u 501 -G lg-esxsu super1
/usr/sbin/useradd -u 502 -G lg-esxsu super2
/usr/sbin/useradd -u 503 -G lg-esxsu super3

“Create a local group which will allow members full admin rights to the ESX ha-folder-root and create three user id’s that are members of the group. These user id’s will be authenticated against the domain.local Windows Domain”

##########################################################
# NTP time config
esxcfg-firewall -e ntpClient
echo restrict default kod nomodify notrap noquerynopeer > /etc/ntp.conf
echo restrict 127.0.0.1 >> /etc/ntp.conf
echo server 10.20.0.200 >> /etc/ntp.conf
echo driftfile /var/lib/ntp/drift >> /etc/ntp.conf
/sbin/chkconfig –level 345 ntpd on
/etc/init.d/ntpd start

“Enable outgoing NTP client port access and build the ntp.conf file to use a Windows Domain time service, this is important for kerberos authentication. Start the NTP client daemon.”

# Create vSwitch0, VMMotion1 and the Service Console port group
/usr/sbin/esxcfg-vswitch -a vSwitch0:64
/usr/sbin/esxcfg-vswitch -A “Service Console” vSwitch0
/usr/sbin/esxcfg-vswitch -A Network-00 vSwitch0
/usr/sbin/esxcfg-vswitch -A VMMotion1 vSwitch0
/usr/sbin/esxcfg-vswitch -p VMMotion1 -v 600 vSwitch0
/usr/sbin/esxcfg-vswitch -p Network-00 -v 700 vSwitch0
/usr/sbin/esxcfg-vswitch -L vmnic1 vSwitch0
/usr/sbin/esxcfg-vswitch -L vmnic0 vSwitch0
/usr/sbin/esxcfg-vswitch -M vmnic0 vSwitch0 -p “Service Console”
/usr/sbin/esxcfg-vswitch -M vmnic0 vSwitch0 -p VMMotion1
/usr/sbin/esxcfg-vswitch -M vmnic1 vSwitch0 -p Network-00

“Defines vSwitch0 for the Service Console and the default gateway IP, vMotion on VLAN 600 with a vMotion IP and create a VM Network-00. This vSwitch will be further configured for a custom standby adapter during the initial reboot with a post config script”

# Create vSwitch1 for iSCSI traffic
/usr/sbin/esxcfg-vswitch -a vSwitch1:64
/usr/sbin/esxcfg-vswitch -A iSCSI_Initiator vSwitch1
/usr/sbin/esxcfg-vswitch -p iSCSI_Initiator -v 500 vSwitch1
/usr/sbin/esxcfg-vswitch -L vmnic3 vSwitch1
/usr/sbin/esxcfg-vswitch -L vmnic2 vSwitch1
/usr/sbin/esxcfg-vswitch -M vmnic3 vSwitch1 -p iSCSI_Initiator
/usr/sbin/esxcfg-vswitch -M vmnic2 vSwitch1 -p iSCSI_Initiator
/usr/sbin/esxcfg-vmknic -a -i 10.20.10.64 -n 255.255.255.0 iSCSI_Initiator

“Defines vSwitch1 for iSCSI on VLAN 500″

##########################################################
#
%post –interpreter=bash

# Create post config script
cat << EOF > /etc/rc3.d/S99postconf
#!/bin/bash

“Some configuration elements still require a post run this statement sends the follow on text to the s99postconf file until the EOF marker is met.”

# Enable TCP outgoing kerberos, there are issues with udp and enable blockOutgoing
/usr/sbin/esxcfg-firewall -–openport 88,tcp,out,KerberosClientTCP
/usr/sbin/esxcfg-firewall –blockOutgoing

“Seriously, this is important, udp kerberos port 88 is defaulted with ‘esxcfg-auth –enablead ..’, bad default! Also the VWware Kerberos client default uses tcp first and this needs to be fixed as it does not comply with RFC 4120. Even if this works why would we risk dropping an auth packet since any fragged udp packet would be dropped over VPN’s etc. Also turn on the outgoing firewall rules that were disabled previously.”

# Enable VMotion on the VMKernel Interface
/usr/bin/vmware-vim-cmd hostsvc/vmotion/vnic_set vmk1

“Enables vMotion on vmkernel interface 1″

# Define Active and Standby failover for shared vSwitche0
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 ‘Service Console’ –nicorderpolicy-active vmnic0 –nicorderpolicy-standby vmnic1
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 VMMotion1 –nicorderpolicy-active vmnic0 –nicorderpolicy-standby vmnic1
/usr/bin/vmware-vim-cmd hostsvc/net/portgroup_set vSwitch0 Network-00 –nicorderpolicy-active vmnic1 –nicorderpolicy-standby vmnic0

“Here we are overriding our adapter team for vSwitch0 so that we can separate our active traffic on the two adapters while maintaining failover capability.”

# Grant the group named lg-esxsu admin permission to ha-folder-root
/usr/bin/vmware-vim-cmd vimsvc/auth/entity_permission_add vim.Folder:ha-folder-root lg-esxsu true Admin true

“Enables any member of the local group lg-esxsu Administrator permissions to the VMware host”

# Reset system to normal boot mode
echo “Removing automated post script.”
rm /etc/rc3.d/S99postconf
EOF
chmod +x /etc/rc3.d/S99postconf

“Obvious”

As you can see the process is quite involved, however the benefits are outstanding. I can build or recover an ESX 4 host in 10 minutes or less and I can reconfigure it to a different target with ease.

Hope you found the entry usefull and interesting.
Regards,
Mike

Site Contents: © 2009  Mike La Spina

Power House Blog on NFS and VMware lead by Chad Sakac

Another great read from Chad Sakac and company.

Quote:

We were quite a bit surprised to see how popular our “Multivendor iSCSI” postwas. The feedback was overwhelming and very supportive of industry leaders partnering to ensure customer’s success with VMware. While writing that post, we (Vaughn Stewart from NetApp and Chad Sakac from EMC) discussed following up the iSCSI post with one focused on deploying VMware over NFS. The most difficult part around creating this post is that we couldn’t do it with our iSCSI-focused colleagues.

A “Multivendor Post” to help our mutual NFS customers using VMware

Many thanks to Chad and all of the contributors!

Regards,

Mike

Site Contents: © 2009  Mike La Spina