Are you testing your Backups?

[-]

Studio_Two@reddit

Grab a battered old standby server with a large drive running hyper-v. Test restore a VM to it, and then disconnect it from the main network. At the very least, you should be able to see if the VM will spin up. It also gives you an idea as to how long a full restore will take.

Reply

[-]

NSFW_IT_Account@reddit (OP)

Is this safe to do on production network? Can I test the restore (but not power on the VM) and not worry about affecting production

Reply

[-]

Studio_Two@reddit

Hi. Just built your confidence step by step. Rather than using your DC, spin up a fresh Ubuntu VM or something. Back it up, then try a test restore to your spare server. You can usually set the restored server to recover is a shut down state.

Reply

[-]

Mr-Hops@reddit

We use the DATTO backup service/appliance (Kaseya now I believe). Snapshots are done hourly, then the appliance will automatically boot the backed up server nightly to verify the backup. If the backup boots to the Windows login screen, I receive an email notification with a screenshot of the login screen to verify. Monthly, I will physically test the backup. The appliance backs up the servers into a virtual environment. To test the backup, I'll just disable the virtual switch, boot the server, and login.

Reply

[-]

NSFW_IT_Account@reddit (OP)

This sounds awesome. We use Barracuda and have none of that capability (AFAIK). Does Datto provide an appliance and where does the appliance “test” the backup to?

Reply

[-]

Top-Perspective-4069@reddit

Live Boot doesn't exist anymore? I used to use that a lot for client backup testing when I worked for a Barracuda reseller.

Reply

[-]

NSFW_IT_Account@reddit (OP)

Is that what that does? Its not enabled on the unit i checked and their support said they have no testing environment.

Reply

[-]

Top-Perspective-4069@reddit

Yeah. It might be a licensed feature or only available with a certain size appliance, I don't remember exactly.

Reply

[-]

NSFW_IT_Account@reddit (OP)

So I'm showing live boot as greyed out when I click "add". My sources are Windows OS based, even though they are technically VMs. I may need to add them as a VM as well in order for liveboot to work.

Reply

[-]

Top-Perspective-4069@reddit

Ah, yes, you definitely need to back them up as a VM from the Hypervisor.

Reply

[-]

NSFW_IT_Account@reddit (OP)

That leads me to the question: should I be backing these up as VMs or Windows OS type? I'm guessing I can get more granular with having the agent installed on each device, but for restore purposes VMs is going to be easier?

Reply

[-]

Top-Perspective-4069@reddit

It'll probably be a mix of both, depending on your RPO and RTO. In general Windows OS backups are fine for things like file servers. For application servers, I preferred VM backups. SQL backups are another thing entirely. Depending what the databases are, I've run those 3 times a day before.

Reply

[-]

MidninBR@reddit

I switched from it to Cove. They have automated recovering in their system and email me a screenshot of the login screen or if it failed every 15 days. This saves me money and time testing the recover process

Reply

[-]

trueppp@reddit

Datto has both, local and cloud. For all our clients, we have hourly local backups and daily cloud backups. Both do the snapshot tests. RTO on our last "full nuke" test (Ex: Client's office is wiped out of existence) was around 12 hours from notification to getting cloud infra running and 30 laptops ready and supplied to client employees. Most of the time was taken getting the laptops out the door and into employee hands.

Reply

[-]

NSFW_IT_Account@reddit (OP)

Ok this sounds awesome. Will be scheduling a demo with them I think.

Reply

[-]

Mr-Hops@reddit

Yes. It is their appliance. I believe it’s just a Dell server with Datto branding. The appliance tests it on itself. It’ll spin up the virtualized backups on its own hardware. Weekly it sends the backups offsite.

Reply

[-]

NSFW_IT_Account@reddit (OP)

Sounds pretty nice. I’m gonna look into it.

Reply

[-]

Mr-Hops@reddit

Good luck! We’ve had for about 5 years now. O oh 1-2 minor issues over that span. It’s not a cheap service though.

Reply

[-]

1z1z2x2x3c3c4v4v@reddit

> How do you test them? In an isolated test environment. > Is it possible to restore a production server to another machine without affecting anything in production? If it's an isolated test environment, then yes, else, no. > I'd like to start testing system state backups to make sure they work. Good idea. We do full system state testing once a year during our DR testing.

Reply

[-]

NSFW_IT_Account@reddit (OP)

What does the isolated test environment look like, and how can i set one up?

Reply

[-]

Snowmobile2004@reddit

If you have Veeam it has something called Surebackup you can setup, pretty easy to do on VMware. I have it setup in my homelab (well i did before my storage blew up), and luckily the tests worked and all the mission critical VMs i had backed up were safe. [https://helpcenter.veeam.com/docs/vbr/userguide/surebackup\_hiw.html?ver=13](https://helpcenter.veeam.com/docs/vbr/userguide/surebackup_hiw.html?ver=13)

Reply

[-]

Ok-Magazine-1507@reddit

yes also using Surebackup for recovery testing, if you select the option to not shut it down after it restored, ypu can use it for other testing, ( examle restore a VM , apply a patch or application update to see the behaviour before doing it in the prod systems)

Reply

[-]

frosty3140@reddit

for VMs -- this \^\^\^ -- Veeam SureBackup -- once you work out how the complicated networking config works in the SureBackup Lab it's a breeze -- we have all our VMs (about 30) restored in automated SureBackup jobs each weekend, to verify that the backups work and the VMs will boot when restore is required

Reply

[-]

GhoastTypist@reddit

I suspect they have a server thats typically idle not connected to the environment running the same software as their other servers for backups and once a year they mount their backups to that system by an external drive, then perform a recovery.

Reply

[-]

its_FORTY@reddit

You build the necessary infrastructure you need to test your backups but completely air gapped from your production network.

Reply

[-]

marklein@reddit

You're confusing me. The question answers itself. It's "isolated" from your normal network so they won't talk to each other. It looks like whatever it needs to look like in order to text your backups. You set one up by being a sysadmin.

Reply

[-]

1z1z2x2x3c3c4v4v@reddit

If someone with such a great Reddit name as /u/NSFW_IT_Account doesn't know that answer, I feel that I /u/1z1z2x2x3c3c4v4v am unqualified to properly answer it.

Reply

[-]

MidOrMeepo@reddit

As an MSP we leverage Veeam's SureBackup feature for most of our customers with automated reporting back to us. VMs run in a sandbox and are accessed through Veeam's proxy appliance. Automatically tests heartbeat and ping or for some VMs a little more sophisticated DC or database test scripts. No risk of affecting production and very little maintenance required once it's up and running.

Reply

[-]

No_Adhesiveness315@reddit

Veeam data labs ftw. Provides a lot of flexibility if you want to spin up a VM or VM’s reliant on one another and test a change before committing in prod.

Reply

[-]

NSFW_IT_Account@reddit (OP)

Where are the labs hosted?

Reply

[-]

frosty3140@reddit

You can get this stuff as cloud services -- or -- you can stand up a SureBackup environment on-prem (I suppose the licensing dictates the features available?) -- we have used the on-prem SureBackup in Veeam for the past 10 years -- works great -- I have just moved us from vSphere to HyperV so I have to rebuild the config in the next few weeks -- it can be complex the first time you see it, but fortunately I have good documentation from the first time we deployed it, so I can just copy that setup I hope.

Reply

[-]

No_Adhesiveness315@reddit

This was a few years ago at a previous job, but no - right in our colo. When you set it up, it deploys another Linux proxy (if I remember correctly) that acts as a fence between production and the “test” data lab. I had a jump box I configured some routes on into the test environment, worked out nicely for us. You can also take backups of machines in the data labs, then stand them up next to their production counterpart if the circumstance dictates it.

Reply

[-]

NSFW_IT_Account@reddit (OP)

Would love to learn more about this. Unfortunately our solution does not offer any sort of automated testing or sandbox environments to restore to.

Reply

[-]

MidOrMeepo@reddit

Check out the Veeam help center, their knowledge base is top notch. https://helpcenter.veeam.com/docs/vbr/userguide/surebackup_hiw.html?ver=13 If you can't do automatic testing, the bare minimum to alleviate the Schrödinger's backup problem would be restoring the VMs to a sandboxed vSwitch/Hyper-V switch and spinning them up manually every once in a while.

Reply

[-]

ScriptThat@reddit

Do you even have backups if you don't know if you can restore them?

Reply

[-]

general-noob@reddit

We wait for someone to break something, run the restore, and say “o, shit that worked?!?”

Reply

[-]

Glittering_Power6257@reddit

The moment where I believe I’m dreaming because the all important DC/DHCP/Print Server/File Server restoration “just worked”, and going from an ESXI to a Hyper-V compatible image at that, onto some spare desktop because the hosts were not operational. The moment I plugged the network cable into that desktop, my life flashed before my eyes as everything came back online. Working on splitting up all those roles btw before this ends up in R/shittysysadmin.(I’d inherited this setup on very short notice during said disaster).

Reply

[-]

NSFW_IT_Account@reddit (OP)

Can confirm

Reply

[-]

Valkeyere@reddit

If you aren't testing your backups you don't have backups. Whatever you need to do to pull a random file out and make sure it's able to be opened. Just because there is ~1TB sitting in some BCDR service or NAS doesn't mean it's usable data.

Reply

[-]

Background-Slip8205@reddit

I've only seen people rely on the backup logs to prove they were taken. What's far more common is for a business to do a yearly DR test.

Reply

[-]

linkdudesmash@reddit

Will do it live!

Reply

[-]

Unable-Entrance3110@reddit

We test them constantly because users are always deleting files and then asking us to recover them.

Reply

[-]

Inevitable-Room4953@reddit

We test our backups when someone deletes stuff they shouldn’t. Also have semiannual backup audits that we have to test specific systems.

Reply

[-]

NSFW_IT_Account@reddit (OP)

Lol, thats basically how often I use the backups too. I’d love to be able to fully restore a dc or file server from scratch though just to have that extra peace of mind.

Reply

[-]

Creative-Package6213@reddit

God I wish the company I worked for cared about testing their backups...

Reply

[-]

ipreferanothername@reddit

my company technically cares, but effectively does not.

Reply

[-]

occasional_cynic@reddit

It has stayed with me for almost twenty years when my boss (IT Director) came to me and said "we should have a test environment." I started researching (I was a level 1 HelpDesk keep in mind), and came up with the following after consulting with the sysadmins: * Two servers * One Switch * One Small firewall I think the total cost was like 12k. This was for a $300 million company. When I presented it to him his non-answer was along the lines of "we need to figure it out as we go...." I quit three-four months later, and that "project" never went anywhere. It is a lesson that stayed with me. If the company does not care, neither do I.

Reply

[-]

Creative-Package6213@reddit

Believe me I'm trying to leave as quickly as possible before this shit show collapses in on itself. But with the way the economy is right now who knows when that will be.

Reply

[-]

pangapingus@reddit

If you're not killing prod on scheduled days once/quarter and letting your Infrascale/Datto demonstrate their purpose you're playing hope BDRaaS, change my mind

Reply

[-]

Burgergold@reddit

Oh, yes, everytime I need to restore something...

Reply

[-]

malikto44@reddit

With Veeam and Commvault, I just set up a process that pulls a VM, "streams" the restore, fires the VM up, does some tests, if all tests pass, aborts the restore, and then goes around pulling abother one. I scripted something for file restores as well, making sure some random file at some random date can be pulled back easily.

Reply

[-]

Master-IT-All@reddit

FRIDAY NIGHT is the night. Not for party, but to sit there and test the Business Continuity & Disaster Recovery process. There's a big runbook for all services, and we would step through the entire runbook for BCDR, simulating an entire loss of the data center. Backups are for data restoration, so we test data restoration as well.

Reply

[-]

Vektor0@reddit

> Is it possible to restore a production server to another machine without affecting anything in production? I'd like to start testing system state backups to make sure they work. All of this info is in your vendor's documentation. Read that, and come back if you have any questions. Be sure to read about how they define a "system state backup." Some backup apps, like Intronis, support system state restores to the same system only -- meaning you wouldn't be able to restore the backup to another machine.

Reply

[-]

NSFW_IT_Account@reddit (OP)

Reaching out to my vendor is on my list as well. I always like having an open discussion on here as well and its an important topic.

Reply

[-]

Frothyleet@reddit

It is, but it's a very basic question and it's product and environment-specific. Not doing backup testing is barely a step beyond just not having backups in terms of IT architecture.

Reply

[-]

CloudLenny@reddit

Quoting my manager: "If you aren't testing your backups, you don't have backups, you just have hope!" Of course you should always check your vendors documentation, and you could also Air-gap it for a safety net**.** Many modern suites have a built-in 'Instant VM' feature for exactly this kind of testing. What backup software are you currently using?

Reply

[-]

Vektor0@reddit

Go away bot, no one wants whatever you're selling

Reply

[-]

idylwino@reddit

We have a whole BCM process that we perform annually. We are required to report on the gauntlet of restores, include file/folder from offsite tape backup.

Reply

[-]

Turbulent-Pea-8826@reddit

Luckily our backup system keeps logs and whatnot that do show if an anything failed backups. It’s fancy looking. If anything fails then I manually do a backup. To test restoring: Every week a pick a random server and do a full restore (via a clone so it’s not online). I also pick a server and do a file level restore. I also delete and restore a (non production) AD object. I also restore a server on our fail over site (again I do a clone, I don’t write over an active server). I record all of this in a spreadsheet that no one looks at. But we are required to ‘test’ backups but no one has ever given me a procedure so <shrug>. Once a year a send an email to my boss that we should do a live test failover and backup event which is ignored. I file those emails away for CYA purposes.

Reply

[-]

ifq29311@reddit

we have dedicated environment for automated backup restoring setup fresh VM from snapshot, run ansible to configure it, restore backup, run some predefined tests to verify whether given app/db/system works properly, restore snapshot to clean state, rinse, repeat

Reply

Reply to Post

60 Comments