Ubiquitous Talk

iSCSI Security Basics

With iSCSI’s growing popularity the need for improved iSCSI security understanding is becoming very important. Multiple issues arise when we choose to transport storage over our networks. The fundamental security areas of availability, confidentiality and integrity are all at risk when iSCSI best practices are not implemented. For example a single attachment error can corrupt an iSCSI attached device at the speed of electrons. An entire data volume can be connected to an unauthorized users system. Access can be interrupted by a simple network problem. The possible outage issues are many and far reaching thus we must carefully assess our risk and mitigate them where appropriate.

The most important step to a best practice is good old planning. It never fails to amaze me when I see instances of adhoc implementations of highly sensitive storage system architectures. Without fail the adhoc unplanned method will result in unstable systems. Primarily due to forced system changes that were required to correct a multitude of issues like miss-configuration, poor performance, miss matched firmware etc. Planning is one of the most difficult parts of the process where we almost need to be a fortune teller. You have to ask yourself the proverbial 1000 questions and come up with the right answers. So where do you begin? Well a good start would be to use a risk based approach. How much would it cost if a failure occurred and what data is most critical, how much is the data worth etc. We need to walk through the possible events and determine what make sense to prevent the events and when not to overdo it.

Some of the more obvious ways to prevent security events are to build on fundamental best practices like the following.

• Build a physically separate network to service iSCSI
• Zone the services with VLANS
• Provide power redundancy to all devices (Separate UPS etc.)  
• Provide a minimum of two data paths to all critical components
• Disable unused ports
• Physically secure all the iSCSI network access points
• Use mutual authentication
• Where appropriate use TOE (TCP Offloading Engine) hardware but be careful of complexity
• Encrypt data crossing a WAN      

The physical separation of the iSCSI network domain is important in many ways. We can prevent unpredictable capacity demands and provide much greater security by reducing the unauthorized physical connectivity. Change control is simplified and can be isolated. External network undesirables are completely eliminated. The dedicated physical network segregation practice is probably the single most important step of building a stable secure iSCSI environment.

Just as we segregate zones on traditional FCP based SAN’s the same practice is beneficial in the iSCSI world, we can filter and control OS based access and data flow we can isolate authentication and name services, performance monitoring can be observed in a VLAN context. Priority can be controlled on a per VLAN context.

I can never forget the day one of our circuit breakers popped in the DC, alarms were TAPped out and a feeling of panic stirred. But that’s when planning pays off. The system continued to run on the secondary power line. So by simply powering on a server you can pop a weak breaker and kill the entire system. So when it comes to power supplies, I take them very seriously. Use TRUE power redundancy on all devices that are part of a critical SAN system, iSCSI or not. It will likely save your bacon in a multitude of scenarios.

No matter how hard we try to be perfect, it is in our nature to make mistakes. And when you are staring at a bundle of cables they really do all look the same. Moving the wrong cable or a port failure can completely take out your iSCSI functionality. Worse yet, a switch module dies on you. For the cost of  a an additional port we can avoid all of these ugly scenarios, besides multi pathing gives you more peek bandwidth. And you gain the ability to do firmware updates without a full outage. Data path redundancy is a must for critical systems.

My neighbor had an interesting story the other day. He kept popping breakers when only two car block heaters were in use. (Block Heater? Yes up north when its cold you have to keep your cars plugged in. And no, we don’t live in igloos) So what does this have to do with iSCSI. Well my neighbor has discovered that if you have a power receptacle available other people will use it. They didn’t ask to use it and he did not expect them to, but you get the point. Your IT neighbors will at some point do the same so disable accessible ports by default and enable only what is in supposed to be in use. 

If my neighbor could have locked the receptacle he would not need to worry about disabling it, but security is about layers of prevention. The more you have the less likely that a breach will happen and preventing physical access is usually easy.

The fastest way to corrupt an iSCSI target is to allow unauthenticated access. It’s just to easy to connect to the wrong node. I decided to do a quick test the other day which involved connecting two Windows initiators to a target. On one I created a directory with jpg photos and on the other I connected and browsed to the photo directory. Bam corrupt target! The second node wrote a thumb nail while the first node copied more photos to the target. Its just completely crazy to run without authentication and one way authentication is not going to prevent this in many cases. Thus we should use a mutual authentication and no it does not make it less hackable but it does help prevent the more probable human error event. For example most IT shops will not want to have separate passwords for every iSCSI CHAP Pair, they will likely use one for many nodes to reduce the administrative overhead. Using the same passwords essentially allows the connection of any correctly configured initiator to any correctly configured target. If you couple a single password config with mutual authentication then you must define the return target access on the respective initiator and this reduces accidental connections to the wrong node. A small amount of effort will prevent this human error and it costs nothing to setup. Of course you can have unique passwords on every iSCSI host, it will however require a central password store and far more administration in larger configurations. There are other solutions but those are out of scope for this basic security blog.

If you anticipate heavy traffic or plan to use IPSec you will want to consider TCP Offload Engine Adaptors (TOE). This does not mean that every server should use TOE on the contrary you should evaluate the iSCSI network traffic of the service and the server CPU utilization level on that server. You can anticipate that 1Gb of sustained iSCSI network will consume about 25% of a 3 Ghz Dual CPU server for strait iSCSI work. IPSec will vary depending on the key size and the OS implementation of the IPSec protocol so I can’t even ball park it. An example where it would make a lot of sense to use a TOE is in the case of a VMware ESX host where you want as much CPU as possible available to VM’s. The TOE Adaptor will do all the iSCSI and IPSec work and the server CPU is all left to the VM’s. It does not make sense if the host will be underutilized and a medium amount of network traffic is predictable. You will also want to consider keeping your IPSec implementation as simple as possible. When a problem occurs, things can get very difficult to analyze and correct with IPSec as well setting up IPSec can be painful if things do not go as planned. (Blog on that subject to come later)

This PDF http://siis.cse.psu.edu/pubs/storagess06.pdf link provides some excelent information on IPSec and iSCSI performance.  

If you have a DR centre or other multisite iSCSI functions across a WAN you will need to consider if data encryption is required. For example if you are using an external network provider then you should definitely be encrypting the data. If securing the data is not critical then consider using compression of some type at the bare minimum. You will promote better availability and as an added bonus the data is will not cross the WAN in the clear for your neighbors to view. If encryption is important then IPSec is a good solid inexpensive way of ensuring confidentiality, integrity and authenticationt. There are many options available to secure the transmission over the WAN. It should be easy to manage and conform to industry standards like 128Bit AES symmetric ciphers which are reasonably fast and extremely secure when managed correctly.

Till Next Time

Regards,

Mike

Site Contents: © 2008  Mike La Spina