Performance over Power : Make the right choice.

Power management is not a new topic when it comes to a hypervisor. We all know that one of the by product of virtualization is "POWER SAVINGS". Even before you start realizing the other benefits of virtualization, power bills is the first Opex savings which makes that return on investment on virtualization speak for itself. 

The reason behind writing this article is to make customers aware that since you have already saved a lot by virtualization, you might not want to cut the corners by trying to save more by scaling down the CPU frequency of an ESXi server to save power. For that matter it applies to all the hypervisors in the industry. We all know that once we start consolidating 10's of physical servers on a single hypervisor, we already end up saving a lot on power and hence you should not worry about throttling down the CPU for saving power on a hypervisor. 

While one can argue that if I can save  more power by using the BIOS features and the hypervisor features to throttle down the CPU frequency, then why not? The answer lies in the trade-off. The trade-off in this case is CPU Performance. While we all know that this throttle is dynamic and will be automatically change on demand, the difference between when the demand is made vs the resource availability leads to Contention. While basic applications might not be impacted by this contention, their would always be applications and the underlying VMs which would not be happy with the latency being introduced due to this throttle. In a lay mans term, this would result in performance issues which are absolutely uncalled for.

I know I am not talking about something unique and every vSphere Admin / Architect is aware of why "High Performance" for power management is critical. I can assure you that there are a number of myths around how power management settings for a hypervisor such as ESXi should be done. Another reason behind highlighting this issue is that vRealize Operations Manager does a great job in tracking the latency which I described earlier. This latency is termed as CPU Contention %. This is the percent of time the virtual machine is unable to run because it is contending for access to the physical CPUs.

If you dissect the statement which I made above, their could be number of reasons behind the inability of the VM to get what it wants, one of them is the efficiency lost due to processor frequency scaling a.k.a power management savings.

The scope of this post is around power management, hence I will not delve into other conditions for now. Before I go further and give you the exact power management settings, I would like to give you a real world example where the CPU contention faced by an application was extremely high due to incorrect power management settings and once they were changed to a mode where we disabled the throttle and made sure CPU was available all the time and never snoozed, the contention dropped down drastically and the application was humming along without any performance bottlenecks. Thanks to vROps that we could identify the issue and solve it within a matter of minutes 💪😁

In the metric chart below, I have a virtual machine which is facing CPU contention % in the range of 10% to 27% in the month of November and December. This was when the application was reported to be sluggish and showing bad performance. In-fact if you observe, the application which was facing an issue is a in memory database with an analytics engine ( it is actually the vROps node itself).

The application actually went into a state where it stopped collecting data as well and hence you see a gap from December 27th to January 8th. This is when the things went out of hand and we decided to take an action to reduce this contention. 

As I explained before, CPU contention could be due to a number of factors. Some of them include, high over commitment, over population of VMs on a host, large virtual machines (crossing NUMA boundaries) and CPU throttle due to power management. Since we knew that this vROps node is the only VM running on that ESXi host, we immediately jumped to check the power management settings on vSphere (hypervisor) and the BIOS (hardware).

Yes, you need to check both and ensure they are set correctly for you to have continuous CPU availability. The correct settings would be:

➦ On vSphere set Power Management to "High Performance"

➦ In BIOS set Power Management to "OS Controlled" (requires restart of ESXi)

You can see from the metric chart below that we are plotting the power management setting of the ESXi hosts (where this VM was running) on both Hardware (Power Management Technology) and vSphere level (Power Management Policy) before and after the change. 

Once the above change was made the CPU Contention % experienced by that virtual machine dropped drastically and we had a well performing application and happy users. You can see the metric chart below which shows the affect on the latency experienced by the VM post the change.

This is a simple yet very powerful example on how Power Management settings play a big role in providing you best performance in your virtual environment. I would recommend that you act immediately to ensure that your environment is not suffering with this issue and the virtual machines are getting what they are suppose to get from a CPU standpoint. Remember a poor CPU performance has a cascading effect on Memory, and I/O buses and hence it is important that this is fixed as soon as possible.

If you have vROps, then it would be very simple for you to visualize the current settings across your environment and track this as a compliance metric going forward to ensure that any new ESXi host added to your environment provides the best in class CPU resources to serve your virtual machines.

If you are vROps 6.4 and beyond, you can simply look at the ESXi host properties by listing them in a view to see what the power management settings are. If they are not correct, you now know that you have a task in hand 😁

Hope this helps..... 👊👊👊