Using scripts to automate VMware Site Recovery Manager workflows & recovery steps!

While I am trying to cope up with projects, new VMware products, training and new initiatives for the VMware community, I quickly thought of coming back to the audience and talk about something which I have discussed before in one of my post and share my experience around that.

Well, I am talking about an article which I wrote on Guest IP Customization on VMware SRM. I would suggest you read that first here before reading this to make more sense out of what we are trying to achieve with this post. Here is the link -  Things to Know About Guest IP Customization in VMware Site Recovery Manager!

Coming back to where I left. We know that certain operating systems, mostly Unix based might not support OS Customization, because of which we are unable to use the Guest IP Customization option in the SRM workflow. This feature allows us to insert a new IP address using the OS API's on a virtual machine at the time of Test Recovery or Disaster Recovery workflow execution. If the OS does not support IP customization then you would have to manually change the IP address on the recovered virtual machine on the DR site, this will impact the overall RTO (Recovery Time Objective) to resume services on the DR site. Though this should not impact a Test Drill much, however this method is prone to human errors and can be painful to execute at the time of disaster.

VMware SRM gives a great feature which can be used to achieve the desired automation and agility for recovering VM's with the required IP settings for Guest OSes which do not support customization. If you ever notice the recovery workflow, you would see that you are able to add pre-power on and post-power on commands to a recovery workflow. These commands can then either talk to a central repository of scripts which can be created in the SRM server or on the recovered virtual machine itself. These commands can be called if you select the respective recovery plan which recovers the virtual machines in question. The screenshot below shows how this window looks like in the SRM recovery workflow:-

In this post we will learn how to use the option "COMMAND ON RECOVERED VM". Here is the situation.

SCENARIO==> You have a virtual machine protected using SRM which is running Oracle Linux. Once this VM gets recovered on the DR site during test recovery or disaster recovery, you want the IP address and other settings to change as soon as the VM comes up on the DR site. Since Guest IP Customization will not work on Oracle Linux you would get the following error if you try to use that method - "Error: The Guest operating system "oracleLinux64Guest" is not supported." How will we solve this issue?

SOLUTION==> To solve this issue we would use the Command on Recovered VM feature of SRM workflow. Here is how you navigate to this window:-

1	Click Recovery Plans in the left pane, and select the recovery plan which has the VM in question.
2	Click the Virtual Machines tab.
3	Right-click the virtual machine and click Configure.
4	Select Pre-Power On Steps or Post Power On Steps in the left pane, and click Add.
5	Select Command on Recovered VM.
6	In the Name text box, type a name for the step. "Command to call IP Address Change Script."
7	In the Content text box, type the commands for the step to run. /bin/sh<space>/etc/network-scripts/changeip.sh  (I will share the content of the script changeip.sh in a moment)
8	(Optional) Modify the Timeout setting.
9	Click OK to add the step to the recovery plan.
10	Click OK to reconfigure the virtual machine to run the command before or after it powers on. Here is the screenshot from my server:-

Here is how my script chnageip.sh looks like:-



#!/bin/bash
#################################################################
# Injecting network files for DR environment
/bin/cp -p /etc/network-scripts/network /etc/sysconfig/network 
/sbin/init 6
#End of script.

Let me describe this script. I am copying the files which I have already kept on the Virtual Machine on my primary site with the new IP settings on the following location - /etc/network-scripts/network. These files are replicated to the DR VM image automatically as a part of my replication plan, whether storage based or vSphere Replication.

With this script I am copying the 'network' file which has all the new settings to the /etc/sysconfig/ which will replace the existing IP settings of the Linux machine to the new IP address which we have defined in our DR Network file. After that the VM would reboot to power on with the new settings.

Similar to this script you can do a bunch of things by calling such scripts on the recovered VM's, such as:-

a) Change any other OS settings such as name, IP, network settings etc.
b) Run cron jobs or schedule tasks.
c) Run configuration files to make changes in an application configuration.
d) Do wonders ;-)


Well now when you run this recovery plan, your VM will power on on the DR site and then as soon as the VMware tools is up on the VM, SRM workflow will call this script and the script will change the network settings of the VM as scripted by you automatically. Automation and agility at its best. :-)

You might have questions about rights to run this script etc.. Remember VMware Tools is used here, so the script should have the rights same as the user account which was used to install VMware Tools, normally its domain admin on 'Windows' and root on 'Linux' OSes.

Well that's all for this time, I hope this would help you get things going on SRM and use the amazing options which SRM provides you to automate stuff. In my upcoming article, I would speak about how to use the option of using "Command on SRM server".

Do let me know if you have any questions or comments. Hope this helps..

Things to Know About Guest IP Customization in VMware Site Recovery Manager!

With the introduction of Site Recovery Manager 5.0, the customization workflows of the product were improved tenfold as compared to the previous versions of SRM. SRM 5.0 Guest IP customization was a great revolution as this helped reducing the RTO significantly. In the earlier versions of SRM, it use to take a long time for Virtual Machines to get a new IP address during a Test Recovery or Actual Recovery since this was achieved by using utilities like sysprep.

However, with SRM 5.0 the Guest IP customization feature allows you to inject a new IP address to the virtual machine powering on at the DR site within a few seconds. This is done by using the guest OS APIs which are used to push the new IP address as soon as the VM is ready to be powered on. In-fact to achieve this, the virtual machine is first briefly powered on to inject this IP and then it will power on again with the new settings as per the power on priorities and dependencies which you have created in the recovery plan.

The amazing thing is that this can be configured for hundreds of virtual machine by using a pre configured xml file or a convenient GUI option can be used, if you have a smaller environment. To learn about how to configure Guest IP customization refer to the This blog article from VMware Blogs.

Now that you understand the feature, its important that you use and apply this in an environment which can support such a feature. The reason I say this is because of the fact that Guest IP customization does not work on all the Guest Oses which are supported on a vSphere Platform. This will only work on Guest OSes which support Guest IP customization. Though, this feature is supported by most of the Guest Operating Systems, however it is better to check before hand to ensure that you do not face any roadblocks during the implementation. You can get the list of Guest OSes which support Guest Customization on the following link.

In case you get into a situation where you have a Guest OS which is not supported for customization, you would get the following error message in the Recovery Workflow as shown in the screenshot below:-

Error: The Guest operating system "oracleLinux64Guest" is not supported. The value in the "quotes" will be the OS which does not support Customization hence if your recovery workflow has the step to customize the IP of this OS, then the recovery plan will STOP with this error. The end state of the Virtual Machine at the DR site would be in registered mode, however it would be in a powered OFF state.

You can manually power on this VM in the DR site, however the IP address of this machine will remain unchanged from the primary site which could result in a catastrophe, hence please be careful.

Now, let's talk about the remedies and how we can take care of such situations till the time the Guest OS advances and starts supporting the customization option. I believe we can 2 methods to take care of this issue.

Method 1 - We can make this change a manual option, which means that we can easily Add a Message Step to the recovery plan for such Virtual Machines. This message step would be added before the Power On step in the workflow. This message will display that the Virtual machine vNic needs to be disconnected at power on and the virtual machine IP address needs to be changed by the Administrator manually at the time of TEST or Actual RECOVERY. You should also disable the IP Customization step for this machine in the workflow so that the workflow executes successfully.

Method 2 - The second method is more ADMIN friendly as I am going to ask you to script this change and add this to the Post Power on script option in the Recovery Workflow. For this you would need 2 Ethernet configuration files, one for Primary and the other for DR site. Please research and create these files on the basis of the operating system which you have. This would mostly be a Unix flavored OS as most of the Windows Guest OS support Customization. Once you have these files ready, add a script on the Startup of the OS to replace the existing network settings with the new one at the time of , Failover, Failback or Test.

Hope this will help you to tackle such a situation if you come across one.