Did You Know #6 - Using custom metrics groups in vROps for troubleshooting

Welcome back to the Did You Know Series on vRealize operations Manager. As I mentioned in the first part of this series, the goal here is to unearth the Best Kept Secrets of vRealize Operations Manager. 

This could be across features, functionalities, use cases, integrations, APIs or any tips or tricks which can help make day to day operations of SDDC easier and fun with vROps!

Today I will talk about how you can make troubleshooting an easier process with vROps. Troubleshooting as we all know is not a skill, it is a methodology. It would NOT be incorrect to say that each one of us has a different troubleshooting style. With vRealize Operations you can troubleshoot issues the way you like them. Some prefer OOTB dashboards, some like to create there own personalized views and some prefer to jump into what me and Iwan call God Mode aka the All Metric view in the product.

The All Metrics view of the product can easily become complex as it shows you all the metrics which are associated to an Object Type. If you look at vRealize Operations Manager 6.4, the product gave you some OOTB custom metric groups which can be used to list all common metrics around CPU, Memory and Disk. These were the OOTB options and might not fit all the needs. If you are on vROps 6.4, click on a VM object and click on All Metrics and you will see this:

You can see that apart from all metrics and all properties for the virtual machine, you can see 5 custom categories which list specific metrics which can be used for troubleshooting. As soon as I double click on CPU, I will see all the key metrics pertaining to the CPU Metric group in one shot on the right pane:

While this is a cool feature and make troubleshooting really simple, there was one use case which could not be solved here. If as an admin I wanted to create my own metric group where I want to focus on key metrics of my own choice, I was unable to create a metric group for same. While this was not possible with vROps 6.4, with the arrival of vROps 6.5, this feature is now available and now you can create your own custom metric groups with the metrics you like to use for troubleshooting.

I will show you how to create one custom group at virtual machine level:

1- Click on any virtual machine in your environment.

2- Click on All Metrics.

3- Click on the blue wheel and click on the Add Group option.

4- Provide a name. I will call it "VM KPIs"

5- Now you can drag and drop any metric from the all metrics group to this new created metric group:



Here are a few KPIs which I added in my vROps as they help me troubleshoot in God Mode with a single click.....

VM KPIs

CPU | Demand %

CPU | Usage %

CPU | CPU Contention %

CPU | CO-Stop %

CPU | Ready %

Memory | Usage %

Memory | Contention %

Memory | Balloon %

Memory | Swap In (KB)

Memory | Compressed (KB)

Virtual Disk | Aggregate of all instances | Commands Per Second

Virtual Disk | Aggregate of all instances | Total Latency

Disk Space | Snapshot | Virtual Machine Used (GB)

Guest File System Stats | Total Guest File System Free (GB)

Network I/O|Aggregate of all instances| Packets Dropped %

Host System KPIs

CPU|CPU Contention (%)

CPU|Demand (%)

Memory|Contention (%)

Memory|Total Capacity (KB)

Memory|Consumed (KB)

Memory| Usage (%)

Network I/O|Aggregate of all instances|Packets Dropped (%)

Cluster KPIs

CPU| CPU Contention (%)

CPU|Demand (%)

CPU|Max VM CPU Contention (%)

Memory|Balloon (KB)

Memory|Contention (%)

Memory|Max VM memory Contention (%)

Memory|Usage (%)

 Go configure your vROps with the metrics you like and make troubleshooting an easy and fun process...

And yeah. Keep sharing!!

Did You Know #2 - Leveraging vROps Remote Collectors for Local Adapters!

In this part of the "Did You Know" series, I will talk about a small architectural tip which will not only help you enhance the performance of your vRealize Operations Manager cluster, it will also save you from up-sizing the cluster from let's say, medium to large nodes and at the end of the day save a ton of CPU & Memory in the process.

Did you know that vRealize Operations Manager uses Remote Collectors for collecting data from a Remote Datacenter and send it over to the centralized vROps cluster. The diagram below shows the actual purpose for which a remote collector was introduced in vRealize Operations Manager:

In the above example, we have a vROps Cluster in Site A. This cluster consists of 2 or more nodes which have a local collector module on them. This collector module collects the data from the local data sources, which are also known as adapter instances. Some examples of an adapter instances would be vCenter Adapter, NSX Adapter, MPSD (management pack for Storage Devices) etc.

The Nodes of the cluster here have multiple roles to play. They not only collect the data from the data sources, they also have to crunch this data using the analytics engine, calculate dynamic thresholds, run the capacity engine and host all the data through the CASA and Web UI. 

On the other hand in Site B, we have a remote collector group with 2 or more remote collectors (in a HA mode). Their role is to collect the data from the Site B data sources using the Collector framework on each node and send that data over to the centralized cluster in Site A. The Remote Collectors are small form factor of the vROps appliances which are stateless and the only role thy have is to collect data. Here are a few facts which make them great for playing the role of a collector from a sizing standpoint.

The come in 2 form factors ***: 

SMALL: 2 vCPU / 4 GB RAM. A small RC can collect 1500 Objects (an object can be a VM, Datastore, ESXi Host, LUN, etc) and upto 600,000 metrics. (1 VM usually creates around 250 metrics).

LARGE: 4 vCPU / 16 GB RAM. A small RC can collect 12000 Objects and up-to 3,500,000 metrics.

We all know that the main cluster nodes of vROps can also do collection and as per the sizing guidelines, a medium node can collect up to 7000 Objects, while a large cluster node which is 16 vCPU / 48 GB of ram can also collect up to 12000 Objects.

***Reference VMware KB - https://kb.vmware.com/kb/2130551

SIZING SCENARIO:

Now imagine a scenario, where you have a 4 Medium Node cluster with 4 node in Site A. You have a vCenter Adapter instance which has more than 7000 Objects (5000 VMs, 2000 Datastores, 200 ESXI Hosts etc). In such a situation, in order to collect data from this vCenter Adapter instance, you would have to up size your cluster node to a large node. Since vROps cluster nodes have to be symmetrical, you would have to up-size all your cluster nodes to LARGE NODES. In this situation you would have to invest on 32 vCPU (8 per cluster node to reach 16 vCPUs) and 64 GB of RAM (16 per node to reach 48 GB per cluster node). This in most cases is a huge change since you would have to ensure you have enough resources in the under lying cluster. In some cases you might also go beyond the NUMA boundary which we all know has some performance impact from a CPU standpoint.

With all these concerns in place, it would an excellent opportunity to leverage the Remote Collectors in the local Site A as well. While the name says REMOTE, it is not necessary that remote collectors are deployed only on remote sites. They can also be utilized in a local site to collect data from adapter instances which can be large in size. Taken our example into consideration, we would just need 2 Remote Collectors ( 2 for high availability, in case one fails) to collect from the Site A vCenter. These 2 appliances will only cost as 8 vCPUs and 32 GB of RAM in total). This will reduce the resource requirements by more than half and also ensure that your cluster nodes have no pressure on collector and hence all that CPU and RAM can be utilized by the other roles on the vROps nodes which will eventually give better performance.

So here would be the new architecture with Remote Collectors Everywhere!!!

With this model, we have better performance, more scale and less hardware requirement for deploying large vROps Deployments. Another important thing to note is that you can always migrate from Design 1 to Design 2 or from Design 2 to Design 1 without any downtime or data loss. Hence if you are on the way to scale the environments being monitored by vROps, this tech tip would be very useful for you.

Hope this helps with day to day datacenter operations using vRealize Operations Manager.

Stay tuned for more goodies!

Did You Know #1 - Controlling Alerts Storms During Maintenance with vROps!

Welcome to a new series of "Did You Know" Facts about vRealize Operations Manager! With this series, I will help unearth the Best Kept Secrets of vRealize Operations Manager. 

This could be across, features, functionalities, use cases, integrations, APIs or any tips or tricks which can help make day to day operations of Software Defined Datacenter easier and fun with vRealize Operations Manager!

In this first part of the series, we will look at a generic activity of maintenance which is applicable to every datacenter. In most environments activities like patching, power cycle, upgrades, hardware replacements etc are conducted during change windows. While all these activities are important, it is also important that you use the Maintenance Schedules or Maintenance options on vRealize Operations manager to disable any activity on objects which are under any kind of maintenance. This will help you ensure that you do not trigger any FALSE alerts which you might have configured on that object type.

For instance, if you have an alert which would trigger when a virtual machine is powered off, you should ensure that you put virtual machines in maintenance if you are planning to shut them down due to a scheduled downtime for activities such as patching, migrations, hardware version upgrades etc.

To do this follow the steps mentioned below. Do note that you should be on vROps 6.x or above to make use of these steps:

The GUI way:

1- Login to vRealize Operations Manager instance using credentials with permission to perform maintenance activities. Use admin if you do not have a strict Role Based Access Control (RBAC)

2- Click on Administration -> Inventory Explorer

3- On the top right corner of this page, search for the Object which you want to put in maintenance, in my case I want to upgrade my SQL server from SQL 2008 to SQL 2012, hence I will search for this VM.

4- Once I see the Virtual Machine in the List, I just need to select the same and click on the Maintenance Icon:

5- You will see pop-up which you can use to enter the details of maintenance window on this object. You have the option of either entering the maintenance in minutes or an end date.

That's it. This will ensure that the concerned object is in maintenance and I will not get any alerts for the same when I reboot.

In case you choose the first option to end the maintenance manually, please ensure that you do the same, as soon as you are done with the change you were implementing. You would need to come back to inventory explorer, search for the Virtual Machine, select the Virtual Machine name, and click on End Maintenance as shown below:

The API way:

For the cool kids out there, you can also do this programatically by using the vRealize Operations Manager API. Here are the API details, along with JSON and XML sample requests which can be used to build a script or a workflow and you can put one or multiple Objects into maintenance easily. You can use the API calls to end the maintenance as well.

To access the API documentation, you will need to access the following URL:

https://<your-vrops-ip-or-fqdn?>/suite-api/docs/rest/index.html

Hope this helps with day to day datacenter operations using vRealize Operations Manager.

Stay tuned for more goodies!