Changing Swap file Location of VMware vSphere Virtual Machines

While working on a SRM project for a customer, I came across a situation where the customer wanted to minimize the amount of data which we need to replicate at the storage layer for the virtual machines to be available at the DR Site. We are using Storage Array based replication and not vSphere Replication in this case.

The best bet is to ignore the files which are not required to be a part of the DR VM. In a virtual machine a file which you can readily ignore is a .vswap file as this is a resultant file which is used by Virtual machines for swapping memory pages. To read more about virtual machine swap file, refer to this post by Frank Denneman. This would give you a good insight about Virtual machine Swapfiles.

Lets quickly look at what we need to do and consider to perform the relocation of the Swapfiles to a shared datastore (You can keep it on a local datastore as well, however this will increase the time taken to vMotion a Virtual Machine from one host to another since a full copy of the swap file would happen, the post above by Frank explains the same.

Changing the location of Virtual Machine SWAP file to a Shared Datastore

Note:- To change the location of the SWAP files of all the virtual machines in the virtual infrastructure follow the steps below:-

      1-     Connect directly to your host or to your vCenter Server using the VMware Infrastructure/vSphere Client.

      2-     Configure the setting on the Cluster level first by Right Clicking on the cluster in question and click on Edit      Settings, select the Swapfile Location and select the option “Store Swapfile in the datastore specified by host”.

(screenshots courtesy - SOS Tech - Wordpress)

The next step would be to set it up on each and every host in the cluster

      3-     Click the Configuration tab for the ESX host.

4-     Click Virtual Machine Swapfile Location and click Edit.

(screenshots courtesy - SOS Tech - Wordpress)

      5-     Specify the datastore where you want to store the virtual machine swap files.

      6-     After restarting your virtual machines, ensure the swap file is located on the specified host datastore.

      7-     An important point to note is that you would have to redo this setting on a host every time the HOST   is removed from the cluster or else the Swapfile location will default to with “Virtual Machine Files”.

Please ensure that we size this datastore appropriately for performance and Space. The space should be greater than or equal to the total vRAM assigned to the all the Virtual Machines in the Environment. Please ensure you size for future growth as well.

The performance of this datastore is important as the SWAP files help a VM swap out memory pages when the host is memory constrained, hence this datastore should be backed up by disks which have good performance characteristics. With vSphere 5.1 you can actually select the datastore for storing the swapfiles as a SSD based location which will improve the performance drastically.

You can also configure only specific virtual machines for redirecting there SWAP files, however this would incur management overhead. If you need to do this, you can follow the steps below (Reference VMware KB article 1004082)

To change the swap file location for a single virtual machine:

1. Power off the virtual machine.
2. Unregister the virtual machine. Right-click the virtual machine in the Inventory and choose Remove from Inventory.
3. Connect to the host:
• For ESX, use an SSH client. For more information, see Connecting to an ESX host using a SSH client (1019852).
• For ESXi, use Tech Support Mode. For more information, Tech Support Mode for Emergency Support (1003677) or Using Tech Support Mode in ESXi 4.1 and ESXi 5.0 (1017910). 
4. Change directory to the folder where the virtual machine resides:
   
   cd /vmfs/volumes/datastore_name/virtual_machine_folder/ 
5. Edit the virtual machines configuration file with a text editor.
6. Add this line:
   
   sched.swap.dir = /vmfs/volumes/datastore/ 
7. Register the virtual machine again. For more information, see Registering or adding a virtual machine to the inventory (1006160).

Hope this helps people who have to take similar decisions for their Virtual Environments.

*********************************************************************************

Update to the article (20th Sept. 2012)

*********************************************************************************

Happy to share that this article got a lot of traction in the last 4 days and there were a number of discussions which happened over the web, however unfortunately, those discussions were not updated here automatically. I wish there was a feature to do that... 

So for the benefit of all... I am updating the discussions in a question and answer format..

FAQ's

Ques: How does this affect performance of the VMs if the swap is on another datastore? We are re-architecting the vm environment.

Ans: The Performance of the VM's would be affected if you locate them on a storage LUN which could be under-performing due to the characteristics of the LUN. For ex- if the LUN is carved out of slow SATA disks. 

Just relocating the swap files to a different LUN does not result in performance issues. 
Another issue could be a slowness in the process of vMotion if the swap files are not located on a share storage (instead the are on a local storage.) 


Ques: From a architecture view, if we have all the swap space setup on a single lun for the production environment, would we need a corresponding same size lun setup for the dr site? Or would it be better just to have the vm take care of the swap at the dr site? 

Ans: I would not suggest you to dedicate a lun for Swap files in DR site, because this storage space would be kept idle for most of its life. Storage being expensive you might not want to keep it idle. Instead, let the VM's boot with the swap file with the virtual machine file. 

However, if you ever get into a failover situation and run VM's from the DR site, at the time of failback the Swap files would be replicated back to the Primary site, creating a similar situation which you have in hand today. Still the cost to replicate this during Failback might be much less as compared to idle storage space of this capacity. 

Chose what suits your environment the best. 

Ques: After reading your blogpost, need your inputs 

- As you might be aware storage vmotion creates a temp second vswp file on the source, so how did you size the .vswp datastore to incorporate storage vmotion functionality 

- In the scenario you shared how did you addressed a memory overhead vswp file "VMX swap file" created with each VM post ESXi 5.0 

- If I assume business critical VM's part of your SRM scenario do not undergo frequent power cycles and the solution architect correctly for configured and memory reservation the .vswp file will be static with no changes to storage blocks post first full copy would you still recommend a dedicated .vswp datastore

Ans 1 (By Samir Roshan of - http://thinkingloudoncloud.com/)

: By default now you have two vswp files in a VM directory one is old good friend swap file and the new VMX swap file. More details here http://pubs.vmware.com/vsphere-50/index.jsp?topic=/com.vmware.vsphere.resmgmt.doc_50/GUID-51767DC5-9A03-4B41-A385-9A11F6BD36F1.html

There are also chances of the .vswp storage block changes as the .vswp file is not a static file although it may seem so (due to it's size calculation on the datastore) but remember this is used for swapping and it is not guaranteed that the same memory pages are swapped all the time.

Now the SRM and swap file connection, in SRM your VM is going to have a reboot any way in the DR site in that case the vswp file is an additional load that you have to carry even for the initial copy. As we know the .vswp file gets created when the VM is powered on and it gets deleted when the VM is powered off.

So in SRM it's good to have the .vswp files stored in some other datastore than the one participating in the Protection group.

Local swap files with SSD is a good design consideration which can give better performance if there is swapping.

Ans 2: To start with I would not setup dedicated datastore for virtual machines unless I have to implement SRM in the environment, as this adds some complexity in the environment. 

As we all know that when we look at Interoperability with Storage vMotion and Storage DRS 

"Due to some specific and limited cases where recoverability can be compromised during storage movement, Site Recovery Manager 5.0.1 is not supported for use with Storage vMotion (SVmotion) and is not supported for use with the Storage Distributed Resource Scheduler (SDRS) including the use of datastore clusters. If you use SVMotion to move a protected virtual machine from a datastore that is in a protection group to a datastore that is not protected, you must manually reconfigure the protection of that virtual machine." 

The above extract from the SRM Release notes clearly shows that its not a great idea to use storage vmotion for vm's which are protected by SRM, since it would result in a reconfiguration of protection group, or a resync of entire data in case of vSphere Replication... 

To your second point, in case you need to do a storage vMotion, you would have to have extra space on the data store as the storage vMotion would create a new swapfile for the period of storage vmoition... Since you would not perform a storage vmotion on all the vm's at the same time, a 20% head room on the Swapfile LUN should be enough to help any environment.... 

To your third point, I agree that your super critical machines would not Swap out as much, however the files will be created... So if you have around 50 machines which you are protecting with 4 GB of swap each, you have to replicate around 200GB of the data which is of no use.... Companies can burst out bandwidth to do so, however it comes with a cost..... 

Ques: Very helpful note. BTW i have this suggestion if you want to keep the Swap file locally(its just idea only i have never tested it, so pleas feel free to feed back) 

Why not we plan future esx hosts with at lease 64GB SSD for Swaps. As i think this this solves 2 issues 
1. improve performance of VM due to swap out. 
2. improves vmotion if the swap is on local DS. 

I understand this is not for all environments but can used for most of them and its justifiable from the cost factor also. 

Ans: The suggestion is great and I completely agree with you on this... SSD is the way to go for local swapfiles... Infact it is a Performance Best Practice by VMware. Refer to following link for more

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf

*********************************************************************************

Update to the article (22nd Sept. 2012)

*********************************************************************************

Alright, this discussion is heating up and we have got some great comments adding a couple from @amitchell01

Placing swap on SSD won't help vMotion performance - in fact it will hurt it as we now need to 

copy the swap file from one host to another in order to move the VM. If vMotion performance is 

a concern keep the swap on shared storage so it doesn't have to be moved when the VM is 

migrated. 

Ideally and if your VMs and hosts are sized appropriately, swap thrashing shouldn't be a 

problem so the placement of the swap file should have a negligible effect on performance 

anyway. 

What I generally do if dealing with replicated storage is place the swap on a shared datastore that is not replicated. 

You can definitely do that but keep in mind that you will get no performance gains other than faster reboots of the ESXi host. Once ESXi is loaded it runs from memory so having a faster boot device provides no benefits.