Monitor Tanzu Kubernetes clusters using vRealize Operations

Overview

Whether it is containers or virtual machines, the end goal of organizations is to have a highly available, reliable and a scalable platform to run their business applications. With vSphere 7.0, VMware provides a unified platform to run your applications, by leveraging vSphere with Kubernetes.

In a previous post, we shared how vRealize Operations 8.1 & vRealize Operations Cloud can easily discover this new platform. Once discovered, the powerful analytics in vRealize Operations, unlock the use cases of monitoring, troubleshooting & capacity management for these new constructs. In the screenshot from my environment, you can see the newly discovered objects automatically tied to the vCenter inventory.

With this post, we will explore how vRealize Operations and vRealize Operations Cloud can monitor Tanzu Kubernetes cluster to provide your central IT teams with Full Stack Observability, from upstream Kubernetes applications; SDDC infrastructure, all the way down to the physical infrastructure. This will give the peace of mind you need to run your business applications with zero-blind spots and complete control.

What is a Tanzu Kubernetes cluster?

A Tanzu Kubernetes cluster is deployed using VMware Tanzu Kubernetes Grid. provides a consistent, upstream implementation of Kubernetes, that is tested, signed, and supported by VMware. A Tanzu Kubernetes cluster is an opinionated installation of Kubernetes open-source software that is built and supported by VMware.

Whether you deploy a Tanzu Kubernetes cluster on vSphere 7.0 using Tanzu Kubernetes Grid Service for vSphere, on AWS natively using Tanzu Kubernetes Grid or on VMware Cloud on AWS using Tanzu Kubernetes Grid Plus, vRealize Operations (On-Prem & Cloud), provides deep visibility into this new world. As central IT teams start to manage these Kubernetes environments, they can simply extend their existing investment in vRealize products to this new world and empower their teams to have an end to end visibility into business applications comprising of traditional VM based apps and modern microservices-based apps.

Monitoring Kubernetes with vRealize Operations

Prerequisites

Let's start with the simple pre-requisites:

• vRealize Operations Cloud or vRealize Operations 8.1.
• vRealize Operations Container Management Solution located on VMware Solution Exchange. For vRealize Operations (on-prem), follow the steps here to install the management pack. For vRealize Operations Cloud, simply raise a Service Request to have a VMware SRE enable this on your instance.
• A Kubernetes cluster deployed by Tanzu Kubernetes Grid.
• A copy of the vrops-cAdvisor.yaml file available here. vRealize Operations uses opensource cAdvisor to collect additional metrics and properties of Kubernetes objects by running as a DaemonSet in a sidecar container.

Prepping a Kubernetes Cluster for Monitoring

In my example, I am leveraging the awesome work done by William Lam, with his Tanzu Kubernetes Grid Demo Appliance fling and his blog post that explains how to leverage this appliance to deploy Kubernetes clusters on VMware Cloud on AWS. In addition to that, I would be using my instance of vRealize Operations Cloud. Let's get started.

With my Tanzu Kubernetes Grid Demo Appliance, I am pre-authenticated to my VMC on AWS SDDC. The first cluster that Tanzu Kubernetes Grid deployed here is a Management Cluster and in addition to that, I have deployed a couple of guest clusters for test and production use. Here are the clusters in my environment:

To list all available Kubernetes contexts, you can use the following command:

kubectl config get-contexts

I have already configured these 3 clusters to be monitored within vRealize Operations Cloud using the container management solution. You can see in the screenshot below:

Let's deploy a new Kubernetes Cluster and then we will add that to vRealize Operations Cloud for monitoring.

 

Step 1 - We will deploy a small development cluster using the following command

tkg create cluster --plan=dev tkg-cluster-03

Alright looks like my Kubernetes cluster "tkg-cluster-03" is up and running.

 

Step 2 - Let's create a vrops-cAdvisor.yaml file on this cluster and run it as a DaemonSet. Run the following commands

Switch context to the newly deployed cluster by running the command

kubectl config use-context tkg-cluster-03-admin@tkg-cluster-03

Now let's switch to the temp directory and create a vrops-cAdvisor.yaml file using VI command. You can choose to create this file elsewhere as well. Copy the content from the following text into the file into your vrops-cAdvisor.yaml file

If you wish, you can change the namespace where you want to deploy cAdvisor by editing the namespace option in the text. I am using the namespace named kube-system

Save the file using wq! and then lets run the following command to run the cAdvisor as a DaemonSet

kubectl apply -f vrops-cAdvisor.yaml

Alright, the cAdvisor sidecar is deployed. That was simple. You can run the following command to see if the containers are running

kubectl -n kube-system get pod

Step 3 - We would need the IP Address and the Credentials for this cluster to add this to vRealize Operations. This information is available in the config file. Run the following command to read the config file.

less .kube/config

We need 2 things from this config file.

1. The IP Address of the newly deployed Kubernetes cluster. You can see all the guest clusters here and the latest one named "tkg-cluster-03" with its server URL : "https://192.168.2.35:6443."
2. We need a way to authenticate against this guest cluster. vRealize Operations support following. For more information, see Kubernetes Authentication.

In my case, I will use the Client Certification Auth. Just copy the following 3 tokens and keep them, we will use them later. This part can be tricky so make sure you have the right tokens if using Client Cert Auth. You would need the following tokens Certificate Authority Data, Client Certificate Data and Client Key Data as shown below. Ensure to use the tokens for tkg-cluster-03 

 

 

Configure Kubernetes Adapter in vRealize Operations

Now that we have all the ingredients, let's get cooking.

Step 1 - Let's add this to vRealize Operations Cloud to begin monitoring. Click on Administration -> Other Accounts -> Click Add Account.

Step 2 - Select Kubernetes Adapter to fill in the following details. You can also see how I created the credentials using the cert data used before.

Note - The vCenter Server Advanced Setting step is optional, if your Kubernetes Cluster is running on vSphere like mine, you can simply add the vCenter Server here. If you are monitoring this vCenter Server with the same instance of vROps, the Container Solution will automatically connect the Kubernetes Nodes to vCenter Virtual Machines.

That's it. just 2 steps :-)

 

 

Using the Kubernetes Overview Dashboard and Troubleshooting Workbench for containers

This one is out of the box. Click on Dashboards -> Kubernetes Environment -> Kubernetes Overview

Since my Kubernetes cluster is running on a VMware Cloud AWS environment, let's see how that advanced setting creates the relation between the Kubernetes nodes and VMs. Select a node from Widget 5 and click on the Object Details icon from this dashboard.

From Object Details, click on the Metrics tab, and let's expand the related VM Kubernetes objects, here you can see the entire related inventory from a container, all the way to the VMC Org.

 

Here is another example of a Kubernetes application named "YELB", that I have deployed on my Kubernetes cluster. vRealize Operations automatically detects that and I can click on the Troubleshoot to start troubleshooting issues using the "Troubleshooting Workbench".

 

Yelb in Troubleshooting Workbench:

 

Conclusion

• The container management solution of vRealize Operations provides deep visibility into any flavor of Kubernetes running on top of vSphere. Be it Tanzu Kubernetes Grid, deployed Kubernetes clusters, or Openshift, it does not matter.
• When it comes to Container Operations for Central IT teams, full-stack visibility from applications to infrastructure is the need of the hour and with vRealize Operations and vRealize Operations Cloud, we provide this visibility by leveraging your existing investments in VMware.
• Lastly, if you are running Kubernetes clusters on a non-vSphere platform such as AWS, Azure, or GCP, you can still leverage the container solution to get visibility into upstream Kubernetes.

Hope this helps. Leave your comments here or reach out via twitter @sunny_dua for any follow-up conversations. Happy Kubernetting!!

 

vRealize Operations 8.1 is out. Here is everything you need to know.

VMware just announced the General Availability of vRealize Operations 8.1. I wanted to quickly share the high-level capabilities that are coming with this release. These are based on key themes of VMware's Cloud Operations vision. 

vRealize Operations Manager 8.1 delivers new and enhanced capabilities for self-driving operations to help customers optimize, plan, and scale VMware Cloud, which includes on-premises private cloud or VMware SDDC in multiple public clouds such as VMware Cloud on AWS, while at the same time unifying multi-cloud monitoring. Powered by artificial intelligence (AI), this release takes us one step closer to the Cloud Operations Vision of providing a unified operations platform, deliver continuous performance optimization, efficient capacity, and cost management, proactive planning, app-aware intelligent remediation, and integrated compliance. 

The key release themes include:

Integrated and Intelligent

• Support for vSphere 7 with Kubernetes helps operationalize vSphere with Kubernetes by enabling you to monitor the health, performance, and the capacity of constructs such as Namespaces, Tanzu Kubernetes cluster, and vSphere Pods.

• Self-driving Operations for VMware Cloud on AWS provides native support for VMware Cloud on AWS. It simplifies Cloud Account for VMware Cloud on AWS, leveraging VMware Cloud Service Portal Token and will be able to auto-Discover new SDDCs and incorporate for monitoring in a few simple steps.

• Intelligent Alert importance ranking to help prioritize triggered alerts based on access and occurrence frequency analysis. 

• Enhanced and updated management packs to support the new release. There are multiple enhancements to various management packs with this release. Refer to the VMware Product Interoperability Matrix for all VMware Management Packs supported with vRealize Operations Manager 8.1. The 3rd party authored Management Packs that are supported with 8.1 are listed in the VMware Compatibility Guide.

• Native support for regulatory compliance packs 

• Integrations. Integrations. Integrations

• Network-based monitoring and troubleshooting with vRealize Network Insight with a launch in context.
• Native integration for NSX-T infrastructure monitoring
• Slack Integration for better collaboration
• Support for Virtual Volumes
• Full support for vRealize Automation 8.1
• Certified for VMware Cloud Foundation 4.0

Easy Management

• Simplified alert management with the ability to create and modify alert definitions and manage changes across alerts, symptoms, recommendations, policies, and notifications using a simplified workflow. This includes enhanced notification filters as well.

• Simplified and intuitive summary pages for key objects to simplify monitoring and troubleshooting with quick access to key information.

• Enhanced Dashboard Management capabilities such as search, arrange, maximize widgets, etc. to simplify the user experience.

• Consistent navigation via improved menus across the workflows to enhance ease of use.

 Application-Aware and Multi-Cloud

• Application Operations enhancements with the ability to do credential-less discovery of services leveraging VMware Tools.

• Telegraf Agent lifecycle management with public API support for application monitoring at large scale.

• Native Support for Azure via Cloud account and monitoring Azure services.

• Enhanced Public cloud monitoring support with an introduction to Google Cloud Platform (GCP) integration in addition to existing support of AWS and Azure.

• Enhanced cloud costing with Cloud Health perspectives.

• Out of box dashboards, Reports and Alerts to manage VMware Cloud on AWS, Public Clouds, and vSphere with Kubernetes

For a detailed list of enhancements refer to the release notes.

·       Download Link: vRealize Operations 8.1 Download Landing Page

·       Release Notes:  Release Notes

Everything you need to know about the vRealize Operations 8.0 release

The wait is over. vRealize Operations 8.0 is NOW Available. Here are my prescribed next steps:

1- See What's New - https://blogs.vmware.com/management/2019/10/whats-new-in-vrealize-operations-8-0-2.html

2- Read the release notes (PLEASE🙃) - https://docs.vmware.com/en/vRealize-Operations-Manager/8.0/rn/vRealize-Operations-Manager-80.html

3- Download the bits from here - https://my.vmware.com/en/web/vmware/info/slug/infrastructure_operations_management/vmware_vrealize_operations/8_0

4- Ensure to run the Upgrade Assessment Tool, available on the download page:

      5- Check the compatibility with management packs you might have in your environment - https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vrops

    

      6- Check product compatibility matrix (just in case) - https://www.vmware.com/resources/compatibility/sim/interop_matrix.php

      7- Good news – Most of the VMware solution packs that go with vROps are already released and here are the links:

Solution Name	Release Notes & Download Link
vRealize Operations Management Pack for CloudHealth by VMware	https://marketplace.vmware.com/vsx/solutions/vmware-vrealize-operations-management-pack-for-cloudhealth-by-vmware
vRealize Operations Management Pack for Storage Devices	https://marketplace.vmware.com/vsx/solutions/management-pack-for-storage-devices
vRealize Operations Management Pack for VMware Cloud on AWS	https://marketplace.vmware.com/vsx/solutions/vmware-vrealize-operations-management-pack-for-vmware-cloud-on-aws
vRealize Operations Management Pack for NSX-T	https://marketplace.vmware.com/vsx/solutions/vmware-vrealize-operations-management-pack-for-nsx-t
vRealize Operations Management Pack for vCloud Director	https://marketplace.vmware.com/vsx/solutions/management-pack-for-vcloud-director
vRealize Operations Management Pack for Skyline	https://marketplace.vmware.com/vsx/solutions/vmware-vrealize-operations-management-pack-for-vmware-skyline
vRealize Operations Management Pack for Container Monitoring	https://marketplace.vmware.com/vsx/solutions/vrealize-operations-management-pack-for-container-monitoring
SDDC Health Monitoring Solution	https://marketplace.vmware.com/vsx/solutions/vmware-sddc-management-health-solution
vRealize Operations Management Pack for NSX for vSphere	https://marketplace.vmware.com/vsx/solutions/management-pack-for-nsx-for-vsphere
vRealize Operations Management Pack for vRealize Orchestrator	https://marketplace.vmware.com/vsx/solutions/management-pack-for-vrealize-orchestrator
vRealize Operations Management Pack for VMware Identity Manager	https://marketplace.vmware.com/vsx/solutions/vmware-vrealize-operations-management-pack-for-vmware-identity-manager
vRealize Operations Federation Management Pack	https://marketplace.vmware.com/vsx/solutions/vrops-federation-management-pack
vRealize Operations Management Pack for VMware Integrated OpenStack	https://marketplace.vmware.com/vsx/solutions/management-pack-for-openstack
vRealize Operations Management Pack for Cloud Provider Pod	https://marketplace.vmware.com/vsx/solutions/vrealize-operations-manager-management-pack-for-vmware-cloud-provider-pod

Happy Installing/Upgrading.

Hope this article helps you. Please share your comments in the comments section below or follow me on twitter to be updated on similar content - @sunny_dua.

Reset out of box content post upgrade or install of vRealize Operations

I have been several times about this and hence I thought I would quickly blog about this. With vRealize Operations, the out of the box content owned by VMware usually gets a refresh with each release. If you wish to get the new updated content, then you can leverage the reset out of box option during upgrade. I wrote about this more than 3 years ago here and this is still relevant.

I have been asked that if one needs to reset the out of box content later on then it is possible. The use cases around this are things like, you have modified out of box symptoms or alerts and you want to restore back to factory settings, or you have accidentally deleted some out of box content which you want to recover.

You can rest out of box content easily per solution. A solution is also known as a management pack which helps you connect to various endpoints to collect data and run analytics.

Here is an example on how you can reset content for vCenter solution:

1- Login with admin privileges or the built-in admin account.

2- Click on Administration -> Solutions.

3- Find the solution for which you want to reset out of box content.

4- Click on the Configure option.

5- On the configuration page click on the option highlighted below to reset out of box content.

6- Read the warning on the dialog box, check the option to confirm and click on ok to take this action.

Note - This process can take up to 30 minutes and will affect the data collection as stated in the warning.

Hope this helps.

Share and spread the knowledge.

How to embed vRealize Operations dashboards for easy information sharing.

One of the most 💖loved💖feature of vRealize Operations 7.0 has been the new capability of sharing dashboards with URL. The following blogpost explains the options around dashboard sharing. In this post, I wanted to share a quick example on how you can use the embed feature of dashboard sharing.

Use Case - The use case is pretty obvious. In organizations where vRealize Operations data can be consumed by different departments, BUs, groups etc, the Operations admin can provide embed code with specific dashboard URLs to the consumers of this dashboard. This embed code can be used to embed the dashboard in an external webpage, confluence page, sharepoint, through a simple iframe.

Let's look at an example. We will use a Billing App dashboard which I have created to monitor the CPU Contention and Disk Latency on all the virtual machines which make up the billing application in my organization.

Let's click on the share button and get the embed code for this dashboard. In case of embedding dashboard, I would recommend that you keep the link expiry option to Never Expire for obvious reasons :-)

In my case, I want to embed this dashboard in this blogpost itself. See for yourself. This is a live dashboard from our lab :-)

P:S - I am able to publish a live dashboard on a public blog because my vRealize Operations instance is running on VMware Cloud on AWS. 😉😊😋

Note - You would require to accept the certificate for my vROps instance to see this dashboard in your browser. Click on this link to access the dashboard directly once and then it will be available to view in this blog post - https://35.161.24.145/ui?t=faasnilnyk#b45vkg0q2f

Go ahead and hover over the metric charts and see the live data of all the billing VMs. You can also see historical data by selecting the calendar icon in the widgets.. 👍👍

Hope this posts helps you undertsand the power of dashboard sharing with vRealize Operations and also allows you to share meaningful data with your peers in your oragnization.

Share and Spread the knowledge.

The return of the dark theme with vRealize Operations 6.7

With the recent release of vRealize Operations 6.7, one of a most awaited feature which has been re-introduced is the concept of themes.

In the past releases of vROps, you could change the entire user interface into a Dark mode. This feature has been re-introduced and can be set by a user through preferences.

To enable the dark theme:

a) Login to vRealize Operations.

b) Click on the user icon and Preferences

c) Under the Color Scheme select "Dark Theme" and click on Save.

d) The screen will reload and within seconds your entire vRealize Operations screen will be in the dark mode.

Here are a few previews of vRealize Operations 6.7 with dark theme:

Click here to download your copy of vRealize Operations 6.7.