VMware Site Recovery Manager - Accessing Test Network during Disaster Recovery Drills!

Like most of my blog posts, this topic was also a question which was raised by one of my customer during an SRM Plan & Design engagement. I did not find a blog or a document which speaks about this topic and hence I thought of documenting this on vXpress and help the community use this solution if they face a similar situation.

Well, the topic is pretty much self explanatory, however let me go ahead and dissect it for those who are wondering what is TEST Network with respect to SRM. 

The most popular feature of VMware SRM is that, it allows you to perform DR drills, using the Test Recovery option which allows you to Test your DR side Virtual Machines, Applications, Networks and the Workflows which you define in the Recovery Plans. These Recovery Plans are created during the configuration of SRM and they are modern day DR Run-books which execute as soon as you run a Test Recovery of Actual Recovery from the SRM Console. Lets look at the difference between the TEST & RECOVERY highlighted in RED in the screenshot below:-

Once you have created a Recovery Plan, which defines the workflow which need to be executed when you press either of those buttons, you are ready to either perform a 

a) Test Drill - Just a test of your DR site virtual machines, to see if your DR solution is actually working. In this process, the Production Virtual Machines keep running on the Primary Site, while copy of these machines in the DR side are mounted on the ESXi servers and are powered on in a snapshot mode (This snapshot is deleted when you cleanup test recovery, so that you do not save any changes on the DR VM's while testing). The replication of data whether Storage Based of vSphere Replication (host based) is not impacted with this Test Drill at any time.

b) Recovery - This button if used, means you actually had a bad day at office... It means you met a disaster, and finally decided that your production site is Down (due to a fire, power outage, earthquake, floods etc). Once you press this button and agree to the warnings, you force the DR machines to power on based on your Recovery Plan and start operations from your DR Site.

Now there is a minor difference in both the cases. In case of Recovery your primary VM's are down, hence you power on your secondary VM's to continue business operations. The Secondary Site network can be an extended network from the Primary site or can be a different sub-net as well. You would not have duplicate Host Names or IP issues since the primary machines are DOWN.

In case of the Test Drill, since the Primary machines are still UP, you power on the DR machines in a ISOLATED TEST NETWORK. This can be created either by choosing the AUTO option while defining DR and Test Networks in the Recovery Plan or by provisioning an ISOLATED VLAN with IP addresses which can be assigned to these test machines and Testing can be performed.

So far I hope it was easy to understand and implement...

Now,since the product has this capability of Test DR Drills, you would want to Test your Recovery Plans, which include, Virtual Machines, Operating Systems, Data, VM Interoperatbility etc, which can be powered on in a bubble environment and tested as and when needed. This can be done even when your production is up and running so this is COOL. However, you need to understand that this testing needs that all the elements which you need to perform a test should be a part of this ISOLATED network, hence anything outside this network cannot be tested or included in this trust zone to avoid DNS conflicts which could lead to data loss/corruption etc. For eg. If you are testing a 3 tier application which has a Web VM which is virtualized and protected via SRM, an application VM (virtualized and protected via SRM), and a database which is PHYSICAL and is not protected via SRM, then you cannot really test the application completely as the physical database cannot run in the Test Mode like VMware Virtual Machines.

Even if you have the capability in your database to run on a snapshot mode, it is not recommended to include that DB in your Test environment unless you are changing the DB networking to the isolated Test Network. Do not create any routes between your Test network and LAN as this can cause trouble which is irreparable. 

Phewwww.... Alright, now since you would follow the right rules, lets talk about accessing this test network. Lets say you are capable to test these Applications, VM instances etc and you want your testers to access this environment from your Primary Site (in most of the cases here is where the application teams, users etc would be sitting). You have a couple of options here:-

a) Jumpstart Terminal Server - You can provision a W2K8 R2 VM on the DR site with RDS ( aka termial server) license and allow your testers to access this machine and use the web browser to access the application. This VM can be used without a Terminal Server License if you do not want multiple Testers to access this VM via RDP. This VM would be provisioned with 2 vNics. One connected to your TEST Network Isolated port Group and the other to your DR Site LAN. Needless to say that your Primary site users should have access to the secondary site LAN via MPLS cloud etc.

b) VMware View Desktops - VDI is another way of making this possible, since you can provision desktops in this network PG and ensure that you create a seperate pool for DR testers and allow them to connect when needed. 

c) vSphere Client Access - You can allow the Testers to Login to the DR site vCenter with limited access and then can directly launch the console of the Test Virtual Machines and play around. This should be very well planned and tested to avoid any unauthorized access.

d) VMRC Weblink - You can generate a Virtual Machine Remote Console weblink and give them to the Testers to use in case they need to, however this will also give them direct access to the virtual machine files and data which you may or may not want to share.

I am sure you can think of other ways as well, but remember that you think and freeze a method during the planning phase to ensure that you can test your deployment in a pilot before going live in the production environment. 

Here are a few screenshots from a PPT which I prepared for explaining this scenario.

SRM Setup between Primary & DR Site using vSphere Replication of Storage Array Based Replication

Performing a Test Recovery which will continue the Storage Replication and Bring the DR Machines up in a Test Network in a Snapshot Mode

The Primary Site has gone down and the Recovery is executed. The business has failed over to the DR Site and the Virtual Machines are connected to the DR Network

Well, I know this might bring up more questions in your mind and if it does then feel free to use the comment column and I will be happy to discuss these options. Choose the best for your DR environment and I can ensure that you would never face any issues whatsoever.

Using vSphere Replication with VMware Site Recovery Manager !!

Today was the SRM day at office, so I thought lets get all the discussions together and put them in an article which would help others as well. I will do it in a question and answer format to make it easier for the audiences of this article...

Note: - This is primarily focused on how VMware Site Recovery Manager along with vSphere Replication helps customers create a BCP/DR solution for them with some amazing and easy features available in this product from VMware.

----------------------------------------------------------------------------------------------------------------------------------

Question - Will SRM 5 work with different hardware at the DC and DR site? As I understand, in case of failover with SRM VMs cold-boot. So there shouldn’t be a hardware compatibility issue.

Answer - You can have different models of x86 server hardware on 2 sites as the Virtualization layer of vSphere would make it seamless for the virtual machines to boot up on the DR Site and then failback as well.


The identical hardware comes into question when you are implementing Storage Array based replication for SRM to use. In this case, your storage arrays should be identical for the replication technology to successfully replicate Luns from Protected to Recovery Site and back.

However, if you are using vSphere Replication, you would not have to worry about that as SRM then uses the host based replication instead of Storage array based replication.

----------------------------------------------------------------------------------------------------------------------------------

Question - If total replication data size is around 20 TB how can we achieve this using host based replication? It is huge data.

Answer - That’s a great questions and probably a challenge which any organization would face, whether they use, Host based replication or vSphere Replication. Now if you are using vSphere replication, there are 2 ways to solve this issue:-

      a)    Use the Full Replication option, which literally means that you start the replication on each VM and then let it finish before you could start creating and configuring your recovery plans on the SRM interface. So for a total of 20 TB of data it could take days (depending on the available bandwidth, distance and latency) before this would complete. I do not recommend this method to customers who are using vSphere Replication as they have an easier way of making these images available on the Recovery (DR) Site.

      b)    Let’s talk about the easier way now. Instead of replicating the entire data over the wire, we can use Physical Couriering method which would save time and bandwidth for the customers. It is as simple as taking a backup / clone of the Virtual Machines which need to be protected on the Protected Site, dump this copy on a USB Drive / Tape or multiple drives/Tapes in your case. Create a MD5 Checksum on these images to ensure consistency(optional step) and ship them across to the Recovery Site. Now, you can seed these images on the target LUNS. Once you are done with this process, configure replication on the Protected Site and point to the seeds as vSphere replication gives you that option during configuration and you are done. Now vSphere Replication will run its magic and just replicate the Delta changes to the Protected site which would be a minimal amount of data as compared to what you would do in the first approach.

----------------------------------------------------------------------------------------------------------------------------------

      Question - Suppose we replicate 10 VMs to the DR site and 5 go down at DC, in this case do we have to manually make DR VMs up or it does that automatically?

    Answer -The SRM solution primarily looks at providing you a framework to design your BCP/DR environment, hence you would use it in a case where you see either a Disaster coming or a disaster which has already destroyed your primary site(Protected Site). This will allow you to either do a planned migration (if you know that disaster is coming) or a site failover (if the damage is already done). Hence, it is not a VM level recovery solution, but a Site Level recovery solution. However, in your case, you can create 2 recovery plans with 5 VM’s each and execute only one recovery plan instead of failing over the entire site. So you can actually have a recovery plan for each VM and execute the one which you want to recover on the DR site in case of a disaster. However, I would revisit my statement here and recommend that we should first look at high availability solutions such as VMware HA, FT or third party clustering solutions for recovering from VM failures. If it is impossible to recover VM’s at a site then we should look at SRM as a Site failure solution.

----------------------------------------------------------------------------------------------------------------------------------

     Question - Suppose customer has  a VM in which we do some configuration like configuring IP of DB server so that it can be in sync with DB server. Now if that VM goes down then at DR it will come up and will try to contact the DB server which is in DC. So in that case how can we utilize SRM feature effectively.

      Answer - SRM gives you the automation to recover virtual machines at the Recovery Site, however it is important that you provide all the components which are required for the applications running on those virtual machines. Taking the database example, you should either have the database available with the same IP address on the DR site or if the database has a different IP address then you can either script the changes using VB Script to change that setting in the ODBC connection after the VM powers on or you can do it manually. If you script it, then SRM Recovery Plan workflow can accommodate that script for you and execute it. For the IP address of the virtual machines, we can use Guest IP customization and that would do the IP changes on the fly using API’s on Windows. You can use Bulk IP customization if you want to change IP’s on a number of virtual machines at one go and this can all be configured while setting up SRM for the first time. Regarding the DB, you can either Virtualize it so that SRM can replicate it and make it available at the recovery site or you can use DB replication technologies to have the database available at the recovery site. 

----------------------------------------------------------------------------------------------------------------------------------

These features really differentiates vSphere Replication from other host based replication technologies and helps the customers implement DR which they were unable to do before.

Alright, I know that might open the "Can of Worms" and make you ask more questions, if that's the case, feel free to use the comment field and we can have some more discussions around this topic.

In addition to this, I would highly recommend the following links from VMware Technical Marketing teams who have done a great job to deep dive into the vSphere Replication technology and discussed "Behind the Scenes" of this feature :-

Advanced vSphere Replication Options for Single VM Replication Performance - by Ken Werneburg

How to Track Replication for Bandwidth Estimation?

Hosting.com shares data on bandwidth, sync times, and RPOs with vSphere Replication