When is the last time you tested your backups?

[-]

fatty1179@reddit

Today when a user deleted a file from a share yet again

[-]

WorkLurkerThrowaway@reddit

We are slaves to crappy software that runs our entire business. Not my team though so not my problem.

[-]

stephendt@reddit

They're still relevant in some situations.

[-]

JDH201@reddit

I use a feature in VEEAM that automatically boots backups into a test environment and runs a script to detect that things are running. I love this feature.

[-]

JDH201@reddit

Yep, for several years now. Works well.

[-]

darklightedge@reddit

Agreed, it works well for me as well. Veeam can handle it.

[-]

perthguppy@reddit

Like, I think surebackup has been in it for a decade now

[-]

JDH201@reddit

Yeah, I’m not a beardy UNIX guy but I’ve been at this a while.

[-]

Yep. Even has a virtual router that does 1:1 nat for all the test VMs so you can access them all and test yourself, and/or run automated testing scripts to access files and services and compare to production. Crazy powerful

[-]

SystemGardener@reddit

Datto also has had similar built in feature for years now.

[-]

easier2say@reddit

Yes, it does. Datto RMM feature for managing backups is really good and helpful.

[-]

iceph03nix@reddit

I still don't trust it. It's nice, but manually logging into a restored VM always makes me feel better

[-]

binkbankb0nk@reddit

But you can do that with that Veeam feature too.

[-]

perthguppy@reddit

It’s a great starting point, but that’s the purpose. From there you run your own scripts etc to access the servers and compare against production. The surebackup stuff also runs a network gateway to 1:1 nat the restore environment so you can just RDP etc yourself while it’s booted. And it’s just one button / powershell command to start a restore lab to test agaibst

[-]

D1TAC@reddit

I'm curious. Explain

[-]

JDH201@reddit

It is a standard feature. Comes with built in scripts or you can add your own custom.

https://helpcenter.veeam.com/docs/backup/vsphere/surebackup_recovery_verification.html?ver=120

[-]

Pyrostasis@reddit

Really wish Veeam for Azure had this functionality. Would be really nice. Far as I can tell it doesnt.

[-]

d00ber@reddit

Yep, I love this!

[-]

derfmcdoogal@reddit

It's called sure backup. It starts up the vms in a test environment then checks that services are running. But it doesn't mean a whole lot for data integrity. So you still have to do those kinds of checks.

Also not a bad practice to restore your backups every once in a while using whatever recovery plan you have.

[-]

BawdyLotion@reddit

Many moons ago I had a small script that dumped a dated file onto the various key backup locations daily so that when restoring I could check those files to confirm the restoration was really pulling restoring readable data. Is the sure backup system able to do simple checks like that as a (not perfect but nice to have) extra sanity check?

[-]

perthguppy@reddit

We have a script that looks in production for files that were last modified between the most recent backup time and the previous one before it, then pulls those files, and the same files from the backup test lab, and hashes them to check they match, puts both copies in a specific share for review, and emails us a report. Then we just go in and check ourselves if we want, otherwise we empty the folder out.

[-]

D1TAC@reddit

Wonderful. I didn't know this existed, was looking for a way to test backups if it's almost automated, where I don't forget.

[-]

perthguppy@reddit

With enough effort it can be fully automated including emailing you screen shots of the VMs and copies of files picked up from the machines to verify. You can also get it to compare files against production

[-]

jamesaepp@reddit

Not who you asked but likely the below.

https://helpcenter.veeam.com/docs/backup/vsphere/surebackup_hiw.html?ver=120

[-]

ZAFJB@reddit

SureBackup for Hyper-V too:

https://helpcenter.veeam.com/docs/backup/hyperv/surebackup_recovery_verification.html?ver=120

[-]

chum-guzzling-shark@reddit

test environment

A hwat now?

[-]

bbqwatermelon@reddit

They misspelled production

[-]

Spiritual_Brick5346@reddit

they both mean the same thing in the majority of companies, gogogo

[-]

Xidium426@reddit

Sunday, it's automated every week.

[-]

ReichMirDieHand@reddit

Nicely, did you use backup software for this?

[-]

CPAtech@reddit

What are you using for this?

[-]

Unable-Entrance3110@reddit

I test them all the time because our users are constantly deleting files and then asking for them back

[-]

sweetroll_burglar@reddit

wednesday

[-]

brads-1@reddit

Every 5 weeks on a Friday so that different media is used to test the restore. Tape drive for the the ultimate in air-gapped backups with 8 total Friday tapes.

[-]

GhoastTypist@reddit

How often do small teams check their backups and test their recovery plan?

I do this a few times a year but I can see this being a once a year thing.

[-]

TeamInfamous1915@reddit

About a months ago when a guy deleted the boot partition so they could extend a drive.

[-]

ZAFJB@reddit

many companies:

Two days after they got ransomwared.

[-]

MacMemo81@reddit

Every week. Recurring task to thechelpdesk.

[-]

luxlucius@reddit

Every day. Because I break stuff on daily basis :)

[-]

LeTrolleur@reddit

Management are stupid and take a very reactionary stance when presented with potential future issues, they'd rather save money now and not worry about saving more further down the line, regardless of my protests.

I enjoy it when shit hits the fan though, provided I'm not the cause of course, so when it does eventually go all tits up I'll be telling them I told them so and trying to hide the big smile on my face.

[-]

ZaetaThe_@reddit

Earlier this week

[-]

TheSpearTip@reddit

I can tell you that I deal with a lot of IT departments who a) never test their backups and b) never monitor their backups to see if they are even running as expected. It is a depressingly large quantity of both.

[-]

MBILC@reddit

"You dont have backups if you've never tested restores" right....scary how many people do not understand this and just assume because they got a notification that "backups were successful" that every is fine

[-]

TheSpearTip@reddit

The number of people who don't even bother setting up notifications and just assume they're running is just as bad. I've had people yell because their backups haven't ran in a year like it is somehow our fault.

[-]

TheEndOfEgo@reddit

Never, as god intended.

[-]

it4brown@reddit

On-prem - last week. Remote? Next week.

[-]

Bob_Spud@reddit

Checking the logs and emailed reports etc of backups have run and the backup system is healthy is not a valid backup test. You must have a proper plan for testing the system using data and machine recovery, not just adhoc stuff that users request.

Using user requests for data restores is not a valid backup test. This is often cited as being valid business test. Competent IT auditors will flag this as being unacceptable.

The article regarding Wexford County.

Backup/recovery systems are now important part of a cyberattack. Take them out first before going for the business systems.
Trying to clean up after a cyberattack using your backup system to recover could make things worse. The systems will become re-infected again. That's why you do a controlled recovery in an isolated environment.

[-]

UniquePossession9171@reddit

When prod RHEL upgrades failed and relied on Ami backup.

[-]

tbrumleve@reddit

Monthly, at the very least. It’s a requirement of our auditors.

[-]

d00ber@reddit

Today, and by test.. I mean a developer who demanded access to a server that I fought them on getting and somehow got accepted by our security and management team, destroyed a server to an unrecoverable state.

[-]

tbrumleve@reddit

Monthly, at the very least. It’s a requirement of our auditors.

[-]

Lost-Droids@reddit

All backups are checked daily when exported for errors by scripts and auto checks ensure that the files are backed up to s3, and of expected size etc, and a daily manual check of the output of these automations is checked daily (takes 10 seconds as its either a big green post saying everything good or red and something afoot at circle K)

Backups are then visually confirmed every Monday (which tests that the scripts are still doing what we expect), and monlty 1 random file from each dc is restored from something we have punted to s3

Daily checks takes about 10 seconds, weekly checks takes about 10 mins and monthly ch3cks takes about 10 mins

Data integrity of the backup source is confirmed every 5 minutes as its replicated drom production to backup and if any of that fails then we are instantly alerted.

Everytime we have needed backups we know 100% that they are OK and that feeling is wonderful

[-]