Using Secondary Management Network for vSphere Replication!!

In the past I have written a lot about using vSphere Replication and the best practices around it. This time around I wanted to share an experience of implementation of SRM and vSphere Replication in a brownfield Virtual Infrastructure.

As we all know that vSphere Replication uses the management interface for the ESXi servers to send the replication traffic to the DR site vSphere Replication Appliance, it is important that we understand the network flow clearly before I do a deep dive into configuration of the networks. The diagram below illustrates on how the data flows. I have tried to include all the objects in the diagram, involved in the implementation to avoid any confusions.

Let's see how the traffic flows in generic sense and then we will add IP addressing to it:-

1- Changed Blocks are captured by the VR Filter on the ESXi server in primary site.

2- This data is sent to DR Site VR Appliance using primary management interface of ESXi server.

3- VR Appliance in the DR Site passed the data to the ESXi servers in the DR Site using the NFC Service.

4- This data is then written on to the designated DR Site datastore.

Note -  Just reverse this sequence when you will do a reverse replication while doing re-protect in SRM.

We will now look into a real life setup and see how this replication will flow. Let me give you a quick view of the setup which I have along with the IP addresses:-

Let's look at each component one by one:-

1 - This is the IP address of the vCenter Server. Notice that the IP sub-nets are different in the Primary Site and DR Site.

2- This is the IP address of the SRM Server. Notice that the IP sub-nets are different in the Primary Site and DR Site similar to vCenter Server.

3- The IP address of the VRA server is not in the same range. The reason is that we do not want to use same ip segment as the management network. In this case we have a Point to Point connectivity between site and the IP configured is on that 10.12.12.x sub-net. This is configured on both sites as the VRA server will receive the traffic from ESXi servers on this interface. Remember this would be a Virtual Machine port group on which this appliance will connect. 

The default gateway for this Subnet is 10.12.12.1 at Primary Site and 10.11.12.1 at the DR Site.

4- VMK0 is the primary management network interface. This is used to manage the ESXi servers in the primary site. If you notice ESXi and vCenter are on the same sub-net. 

5- VMK1 is configured for vMotion on a Non-Routable VLAN. That is the reason you have a completely different IP segment here. Not our concern anyways.

6- VMK2 is the third VMKernel interface I have configured. This is to use the Point to Point connectivity for vSphere Replication. I want to the vSphere Replication traffic to go out of this VMK interface and reach the vSphere Replication appliance on the DR Site.

7- Last and one of the most important thing to note that in case of ESXi, the Default Gateway would always be the one which is defined with VMK0. Hence you will notice that all the VMKernel port-groups will have the same default gateway.


The last point here is the problem for me. Since I do not want the vSphere Replication Traffic to hit that gateway (172.16.3.1) in the DR site, when the traffic is sent to the vSphere Replication appliance in that site. I want it to hit the gateway configured for 10.11.12.x sub-net. The default gateway is 10.11.12.1 to be precise.

Now this is not possible until you define a static route which would force the vSphere Replication Traffic to go through the vSphere Replication Interface(VMK2) and then hit the vSphere Replication appliance on the DR Site with that default gateway. Remember you will have to just reverse this action and add a static route on the ESXI servers in the DR site for (10.12.12.1) Default gateway in the primary site.

Here are the commands to do it.

~ # esxcli network ip route ipv4 add --gateway <Gateway for vSphere Replication Subnet> --network <IP range for vSphere Replication Network in DR Site>

So in my case I will run the following command:-

~ # esxcli network ip route ipv4 add --gateway 10.12.12.1 --network 10.11.12.0/50

You would also need to add this line to the rc.local to make this setting consistent across reboots.

~ # vi /etc/rc.local.d/local.sh

Add the following line just before exit command in the script:-

~ # esxcli network ip route ipv4 add --gateway 10.12.12.1 --network 10.11.12.0/50

Save and Exit from this file and you are done on the primary Site. You need to do the same on the ESXi servers in the DR site for reverse replication to work. The command for DR Site ESXi servers would be:-

~ # esxcli network ip route ipv4 add --gateway 10.11.12.1 --network 10.12.12.0/50

Do remember to add this to the local.sh script as you did in the primary site.


Now let's see how the traffic would flow in this case diagrammatically:-

Here is KB articles from VMware which might help you with this setup.

Configuring static routes for vmkernel ports on an ESXi host (2001426)


Hope this makes things easy for you and allows you to setup vSphere Replication on your preferred network interface.

Share & Spread the Knowledge!!