Hope you are enjoying the What's New with vROps 6.6 series. With this post, I wanted to share a little gem of a configuration which has not been in the lime-light with all the new and cool things available with this new release of vRealize Operations 6.6.
This feature is about data retention which has not changed much since the initial days of vROps (infact it was called vCOps back then). If you remember, Time Series Data Retention is one of the key features of vROps which allows you to retain the metrics data in the FSDB for upto 10 years. This means that you can retain data without any rolls ups for months and go back in time whenever you want to.
While I have never seen a customer with a requirement of 10 year data retention, I have often come across customers who have a 2 year worth data retention policy. Specially in the healthcare or government space where historical data retention is a compliance requirement. In such cases, your storage requirement for saving all this data increases exponentially.
I have had a number of customers ask if we can purge or archive this data or even create roll ups to minimize the disk space requirements while ensuring compliance requirements of long term data retention.
This will also help with reduction of their backup and DR foot print which is key to enterprises running vRealize Operations Manager in a large scale production environment. With the release of vRealize Operations Manager 6.6, you now get a couple of options for data retention. You can find these options under the Global Settings option.
Let's find these settings and drill down into the meaning of these:
1- Login to vRealize Operation with admin privelleges.
2- Click on Administration -> Management -> Global Settings
3- Click on the Edit icon to view the settings in edit mode.
Here you can see 2 settings:
Time Series Data Retention : This is the existing setting from the previous versions and is set to 6 months by default for 5 minutes interval data retention.
Additional Time Series Retention : This is a NEW setting with vROps 6.6. This is set to 36 months by default. This setting will ensure that after 6 months of normal retention (5 minutes), the 7th month data is rolled up into a ONE HOUR (1 Hour) roll up. You can set up this option upto 120 months for data roll ups.
With this setting in place, one can easily, setup the 5 minute interval data to 3 months which is enough for capacity predictions and performance troubleshooting and use the additional retention setting to 3 months or more based on the compliance requirements.
So go ahead, re-think your data retention policy and see how you can make the footprint of your vROps instances smaller with the benefit of longer term data retention.
More to come.. Stay Tuned!!
Thanks for highlighting this feature, I had missed it amongst the all the new stuff in vROps 6.6
ReplyDeleteLooking at the defaults it looks like there will be an increase in the amount of data stored.
Rolling up data from 5 minutes to 1 hour means storing 1/12th of the data. If 36 months of data rolled up at 1 hr is equivalent to 3 months of 5 minute data. So the overall data storage needs increased by 50%. Does this add up for you?
I'm thinking I need to either plan for a significant storage increase or reduce the Time Series Data Retention as you suggest.
I was told the trending/forecast for Capacity / Time Remaining was based on the entire metric history. If we retain rolled-up data will that still affect the trending?
Hi Mark,
DeleteWe have made significant changes to our data storage logic which will automatically reduce the overall requirement of data for the out of the box image. Just to give an example if 6 months woth of storage would take 250 GB as per the sizing calculator, you would now be able to save around 10 months worth of data in the same space. This is possible due to change in the page size with this release. Hence all the extra space is used for the roll ups without you having to add more space.
We are working on updating the sizing calculator to reflect the additional benefits on storage.
The trending/forecasting uses last 30 days of data. hence this would not use the roll-ups. The Time Range for capacity calculations can be changed from the Policy Settings.
Hi Sunny - great to hear those numbers, can't wait to see the savings
ReplyDeleteThe Time Range - Data Range policy setting states - This much data is considered in *non-trend* analytics.
I've seen fluctuating Capacity / Time Remaining and quite a few forecast trends that only make sense looking back the entire 6 month metric history.
Hi Sunny, Thanks for the the post. This is great info (sadly none of it found in official documentation). I still have a question about how the data in rollups is presented:
ReplyDelete-When a rollup happens does it simply take the last (or first) 5 min time slice value and keep it, dropping all others in that hour? or does it Average all the 5 min values in the hour and save that as the value for the hour?
In future releases it would be awesome for us admins to be able to control the rollups for additional data, for instance if we wanted to have full 5 minute data for 3 months, then 1 hour data for 9 months, then 1 day data for 5 years, or something similar... it would be great to have that level of control over our data and ultimately disk size.
It Averages all the 5 min values in the hour and save that as the value for the hour.
DeleteFeature requested Noted :-)
Thanks for the feedback.