Performance over Power : Make the right choice.

Power management is not a new topic when it comes to a hypervisor. We all know that one of the by product of virtualization is "POWER SAVINGS". Even before you start realizing the other benefits of virtualization, power bills is the first Opex savings which makes that return on investment on virtualization speak for itself. 

The reason behind writing this article is to make customers aware that since you have already saved a lot by virtualization, you might not want to cut the corners by trying to save more by scaling down the CPU frequency of an ESXi server to save power. For that matter it applies to all the hypervisors in the industry. We all know that once we start consolidating 10's of physical servers on a single hypervisor, we already end up saving a lot on power and hence you should not worry about throttling down the CPU for saving power on a hypervisor. 

While one can argue that if I can save  more power by using the BIOS features and the hypervisor features to throttle down the CPU frequency, then why not? The answer lies in the trade-off. The trade-off in this case is CPU Performance. While we all know that this throttle is dynamic and will be automatically change on demand, the difference between when the demand is made vs the resource availability leads to Contention. While basic applications might not be impacted by this contention, their would always be applications and the underlying VMs which would not be happy with the latency being introduced due to this throttle. In a lay mans term, this would result in performance issues which are absolutely uncalled for.

I know I am not talking about something unique and every vSphere Admin / Architect is aware of why "High Performance" for power management is critical. I can assure you that there are a number of myths around how power management settings for a hypervisor such as ESXi should be done. Another reason behind highlighting this issue is that vRealize Operations Manager does a great job in tracking the latency which I described earlier. This latency is termed as CPU Contention %. This is the percent of time the virtual machine is unable to run because it is contending for access to the physical CPUs.

If you dissect the statement which I made above, their could be number of reasons behind the inability of the VM to get what it wants, one of them is the efficiency lost due to processor frequency scaling a.k.a power management savings.

The scope of this post is around power management, hence I will not delve into other conditions for now. Before I go further and give you the exact power management settings, I would like to give you a real world example where the CPU contention faced by an application was extremely high due to incorrect power management settings and once they were changed to a mode where we disabled the throttle and made sure CPU was available all the time and never snoozed, the contention dropped down drastically and the application was humming along without any performance bottlenecks. Thanks to vROps that we could identify the issue and solve it within a matter of minutes 💪😁

In the metric chart below, I have a virtual machine which is facing CPU contention % in the range of 10% to 27% in the month of November and December. This was when the application was reported to be sluggish and showing bad performance. In-fact if you observe, the application which was facing an issue is a in memory database with an analytics engine ( it is actually the vROps node itself).

The application actually went into a state where it stopped collecting data as well and hence you see a gap from December 27th to January 8th. This is when the things went out of hand and we decided to take an action to reduce this contention. 

As I explained before, CPU contention could be due to a number of factors. Some of them include, high over commitment, over population of VMs on a host, large virtual machines (crossing NUMA boundaries) and CPU throttle due to power management. Since we knew that this vROps node is the only VM running on that ESXi host, we immediately jumped to check the power management settings on vSphere (hypervisor) and the BIOS (hardware).

Yes, you need to check both and ensure they are set correctly for you to have continuous CPU availability. The correct settings would be:

➦ On vSphere set Power Management to "High Performance"

➦ In BIOS set Power Management to "OS Controlled" (requires restart of ESXi)

You can see from the metric chart below that we are plotting the power management setting of the ESXi hosts (where this VM was running) on both Hardware (Power Management Technology) and vSphere level (Power Management Policy) before and after the change. 

Once the above change was made the CPU Contention % experienced by that virtual machine dropped drastically and we had a well performing application and happy users. You can see the metric chart below which shows the affect on the latency experienced by the VM post the change.

This is a simple yet very powerful example on how Power Management settings play a big role in providing you best performance in your virtual environment. I would recommend that you act immediately to ensure that your environment is not suffering with this issue and the virtual machines are getting what they are suppose to get from a CPU standpoint. Remember a poor CPU performance has a cascading effect on Memory, and I/O buses and hence it is important that this is fixed as soon as possible.

If you have vROps, then it would be very simple for you to visualize the current settings across your environment and track this as a compliance metric going forward to ensure that any new ESXi host added to your environment provides the best in class CPU resources to serve your virtual machines.

If you are vROps 6.4 and beyond, you can simply look at the ESXi host properties by listing them in a view to see what the power management settings are. If they are not correct, you now know that you have a task in hand 😁

Hope this helps..... 👊👊👊

Dividing Bandwidth of a 10 GB CNA Adapter for ESXi Networking and Storage using Network I/O Control (NIOC)

This post circles back to an article which I wrote in the month of January 2013, about Dividing Bandwidth of a 10 GB CNA Adapter for ESXi Networking and Storage!! In that article, I gave you an overview of how you can divide the available bandwidth on a 10 GB CNA card at the hardware level to create multiple vmnics and vmhbas for network and storage traffic respectively.

I got a lot of comments and feedback on that article, wherein some of the experts spoke about doing the same with VMware vSphere Network I/O Control (NIOC). In a recent engagement, we did face a constraint under which the 10 GB adapter could not be segregated at the hardware level. 

This was the opportunity for me to use the 10 GB network with segregation using the vSphere Network I/O Control. I wanted to share the learnings & the experience with my readers as well.

Quick Recap

A CNA card a.k.a "Converged Network Adapter" is an I/O card on a X86 server, that combines the functionality of a host bus adapter (HBA) with a network interface controller (NIC). In other words it "converges" access to, respectively, a storage area network and a general-purpose computer network. As simple as it sounds, it makes things simple in the datacenter as well. Instead of running down those cables from each NIC card/FC HBA or iSCSI cards, you can just use a single cable to do all these tasks for you. This is because the CNA card is converged and can carry all the traffic on a single physical interface.

Since we do not want to segregate the bandwidth on the physical card, we will just do a simple segregation on Network & Storage. This will be done in case we chose our storage medium to be Fiber Channel and not an IP based storage.

If it is IP storage such as NAS or iSCSI, we will divide the entire card into 1 vmnic per physical port in the CNA and then create portgroups for VM Traffic, Management Traffic and IP Storage. However, in my case I had FC storage in place, hence the  bandwidth on the  physical card was divided as shown in the figure below:-

Here, the CNA card has 2 physical ports, each with 10 GB bandwidth. I have further divided this card into 1 network card and 1 FC HBA per physical port. Hence, I will have a total of 2 Network cards and 2 FC HBA per CNA card. If you like the concept of No Single Point of Failure (SPOF) and can afford another card, and then you would end up having 4 NIC Cards and 4 FC HBA Ports per Blade server.

Now a look at how I would use these NICs to configure the networking for the ESXi Server. The diagram below shows how I would configure the networking on my ESXi server to get the best possible configuration out of the available hardware resources. Since we only have 2 Network cards now, we will hook up all the port groups on to it and use Network IO Control to divide the bandwidth at the vSphere layer. Here is how things would be connected logically:-

**NOTE - You need dvSwitch for NIOC configuration, hence you need to ensure that you are on Enterprise Plus Licensing for this to work.

The fun is not over yet. Once you have setup everything on the dvSwitch, things would look like my lab dvSwitch. Look at the screenshot below:-

Now comes the part where you enable Network I/O Control (NIOC) and divide the network resources within the default or defined port group types. Here are the steps to do this.

1- On you vCenter Server click on Home -> Networking -> dvSwitch.

2- Click on the Resources Tab as shown below and Enable NIOC

3- Once it is enabled, click on each resource pool listed for different port group traffic and assign shares.

4- Limit the bandwidth, if you need to for any of the port group.

Here is the screenshot for what I did with my network switch:-

Remember you are free to toggle around the bandwidth for the resource pools on the basis of how much you want for your port groups. The bandwidths which I have mentioned above are a guideline and can be used as they fit in most the bills.

Benefits over CNA level segregation

There are a few benefits of using this method and I will quickly list them down here:-

• You can change the Bandwidths on the Fly as per the requirement.
• Do not need any down times to make changes.
• The single point to configure or change things is the dvSwitch so dependency on each host is ruled out completely.
• Very easy to manage and control
• vSphere Admin has no dependencies and can bump up the vMotion bandwidth if he needs to move VMs across quickly for some reason.

And I can keep on writing...... So if you have Enterprise Plus Licensing, then I would recommend this way of doing things for sure.

Hope this helps you design the network and storage with the 10GB Adapter with using enterprise class features such as VMware Network IO Control.

Don't forget to Share and Spread the Knowledge. This will help others!!

Changing the SSH Port on the ESXi server for Cyber-Ark Integration!!

In one of my recent implementation, I got a request from a client to change the default SSH Port on the ESXi server from Port 22 to Port 63022.

This was a requirement since they have a password management system from Cyber-Ark which would store and reset the root and other user passwords on the ESXi server for security reasons. Cyber-Ark works with any Unix or Linux operating system by using the SSH port. Since ESXi also uses SSH for remote access, we had to integrate Cyber-Ark on SSH port with the ESXi server. Cyber-Ark uses SSH however the integration happens on port 63022 for SSH.

Let's see how I went about changing the SSH Port to 63022 sand made it consistent across ESXi reboots.

We would need to update this configuration in 2 locations for this to work:-

a) /etc/vmware/firewall/ - In this location we would have to place a new firewall rule for SSH port which me manually define. This would be done by creating an XML file which would be saved in this location. Here are the contents if the xml file:-

<ConfigRoot>

<service>

<id>SSH 63022</id>

<rule id = '0000'>

<direction>inbound</direction>

<protocol>tcp</protocol>

<porttype>dst</porttype>

<port>63022</port>

</rule>

<enabled>true</enabled>

<required>false</required>

</service>

</ConfigRoot>

For ease we will call this file ssh63022.xml

We would need to refresh the firewall policies after placing this file in the given location on the ESXi server. Here is the command which will be using:-

#esxcli network firewall refresh

b) /etc/services - The second change would be to create a new services file where we can define the SSH port as 63022 instead of 22. For this we would need to create a new services file. You can copy this file from the default location and place it on a SAN Data-store and then edit it with the new port information. Here is how you can do it:-

# cp /etc/services /vmfs/volumes/EMC-SANLUN-01/ssh

I have created a folder names SSH on my SAN Datastore EMC-SANLUN-01. Then, I am copying the services file to my EMC SAN VMFS datastore which is visible to all my hosts in the cluster. 

Now lets check if the file has moved there:-

~ # cd /vmfs/volumes/EMC-SANLUN-01/ssh

/vmfs/volumes/50f5e6fd-6fa36a6c-8339-000c29c4df2b/ssh # ls -ltrh

-rw-r--r-T    1 root     root        20.3k Jan 16 00:16 services

Now that we have a copy of the services file, lets edit it to change the ssh port. Run the following command:-

/vmfs/volumes/50f5e6fd-6fa36a6c-8339-000c29c4df2b/ssh # vi services

Locate the ssh port setting as shown in the screenshot below:-

Now edit this file and change the port 22 to 63022 as shown below:-

Save the change on this file and run the following command to replace the original file with this file.

~ # cp vmfs/volumes/EMC-SANLUN-01/ssh/services /etc/services

This will change the default ssh port from 22 to 63022.

Now to make it consistent across the reboots, it is important that you perform these 2 steps every time the ESXi server reboots. It is not practical to run these steps manually, hence a better way would be to automate this using the rc.local file which can run simple scripts on the ESXi server during start-up.

Similar to services file in the following location - /vmfs/volumes/EMC-SANLUN-01/ssh, copy the ssh63022.xml which we created in STEP A to this location as well. You can use Datastore Browser on vSphere Client or a utility such as winscp. See screenshot below:-

Now that you have both the files in a shared datastore, update the rc.local file to copy these files to the respective locations everytime the server reboots. You would need to make the following entry in the rc.local file:-

Note - rc.local is located in /etc directory.

Edit the file and update it with the following script:-

#Copy the new firewall rule from vmfs place holder to file system

cp /vmfs/volumes/EMC-SANLUN-01/ssh/ssh63022.xml /etc/vmware/firewall/

#refresh firewall rules

esxcli network firewall refresh

#Copy the modified services file from vmfs place holder to file system

cp /vmfs/volumes/EMC-SANLUN-01/ssh/services /etc/services

#Restart inetd to get the changes

kill -HUP `cat /var/run/inetd.pid`

See screenshot below:-

Run the following command:-

~ # vi /etc/rc.local

Lastly, save this file and Reboot the ESXi host. Now you would have the SSH port set to 63022 and you can easily integrate with Cyber-Ark.

Hope this helps you to make changes to ESXi default ports for 3rd party software integration if needed.

Dividing Bandwidth of a 10 GB CNA Adapter for ESXi Networking andStorage!!

With most of my recent projects customers are moving towards the 10G converged adapters to achieve the benefits of consolidation of network and storage especially on Blade Server Architecture.

I am writing this post to provide you guidelines on how you can divide a 10GB CNA card on your ESXi server to meet all the network and storage requirements. Before that, let’s have a look at what is the 10Gig CNA and what are the brands available in the market available for this technology.

A CNA card a.k.a "Converged Network Adapter" is an I/O card on a X86 server, that combines the functionality of a host bus adapter (HBA) with a network interface controller (NIC). In other words it "converges" access to, respectively, a storage area network and a general-purpose computer network. As simple as it sounds, it makes things simple in the datacenter as well. Instead of running down those cables from each NIC card/FC HBA or iSCSI cards, you can just use a single cable to do all these tasks for you. This is because the CNA card is converged and can carry all the traffic on a single physical interface.

There are a number of manufacturers of such cards who either manufacture these cards themselves or just re-brand them with their logo and custom firmware.. Here are a few examples:-

- Cisco

- Dell

- HP

- Q-Logic

- Emulex

- IBM etc..

So as a customer you have a number of choices and it is important that you choose what fits your existing infrastructure or the new hardware if it is a Greenfield site.

Let's say you bought a CNA which gives you 4 virtual ports per physical port, let’s see how we can divide the bandwidth of this physical port amongst the virtual port to both Storage and Network Communication.

On the physical card the bandwidth can be divided like how it is shown in the figure below:-

Here, the CNA card has 2 physical port each with 10 GB bandwidth. I have further divided this card into 3 network cards and 1 FC HBA per physical port. Hence, I will have a total of 6 Network cards and 2 FC HBA per CAN card. If you like the concept of No Single Point of Failure (SPOF) and can afford another card, and then you would end up having 12 NIC Cards and 4 FC HBA Ports per Blade server.

Isn't that cool?? A Blade server which so many NICs. Well this can be used on Rack servers as well as it will also reduce the back-end cabling!

Now a last look at how I would use these NICs and FC Ports to configure the networking for the ESXi Server. The diagram below shows how I would configure the networking on my ESXi server to get the best possible configuration out of the available hardware resources.

The Diagram above clearly shows how we have divided this bandwidth amongst all the required port groups. If you have 2 such cards, you will have high resiliency in your design and the number of ports would double up providing better performance as well.

Remember you are free to toggle around the bandwidth for the Virtual NICs and Virtual FC HBA’s basis how much you want for your port groups. The bandwidths which I have mentioned above are a guideline and can be used as they fit in most the bills.

Hope this helps you design the network and storage with the 10GB Adapter without issues.

**************************************************

Update - 3rd April - Look at this new article - Dividing Bandwidth of a 10 GB CNA Adapter for ESXi Networking and Storage using Network I/O Control (NIOC) which talks about using Network IO Control to do the network segregation.

Best Practices around using RDMs in vSphere!

One of VMware's partner engineer raised this query on an internal group. He wanted to understand and learn the best practices or the Do's and the Don'ts while using RDM (Raw Device Mappings) Luns in a vSphere environment.

I hope you being a reader understand what an RDM is and what role does it play in a vSphere Environment. In case, you are not aware of RDM, then kindly refer to the following document - vSphere 5.x Storage Guide and read about Rae Device Mapping (RDM).

During this discussion we will consider the following requirements which we need to meet :-

• We need to provision RDM's for more than 20 VM's and size of disks will vary from 1TB to 19TB.

• The RDM is decided for configuring MSCS on VMs like MS Exchange, MSSQL, File Servers etc.

A usual topic of discussion is choosing between RDM and VMDK. Since we have already solved that mystery, there is not much to worry about. We are already following the best practices around application layer by choosing RDM’s instead of VMDK. Now since we are playing with Luns mapped to your virtual machines, there are a few things we should take care of:-

1. Choosing between Physical Compatibility Mode & Virtual Compatibility mode for RDM – A physical RDM is more storage array driven and virtual machine controlled. The VMkernel has limited or no role to play and it literally becomes a postman who delivers IOs from the OS to the LUN (just like an OS running on a physical server saving data on a storage LUN). This will restrict you from using VM level snapshots and other file locking technologies of VMKernel. Since you are talking about file sizes of more than 2TB, please ensure you are on VMFS 5 and use Physical compatibility mode only as VMFS 5 does not support RDM with Virtual compatibility mode for Luns greater than 2TB – 512bytes. On the other hand you would be able to use RDM with Physical compatibility mode for up to 64 TB. (VMFS 5 required)
2. Ensure you have the correct Multipathing and failover settings
3. Please ensure you are zoned appropriately. Test heavily before pushing things into production
4. Follow the MSCS on vSphere guide without fail to avoid any last minutes surprises

Well this should help you do the right things with RDM's. Ensure you go through the document which I mentioned before and you should be good to go.