What do you do for air-gapped offsite backups?

[-]

cjcox4@reddit

Whatever you "store to", be that drives for tapes could merely be "sent home" with someone. Sure, not the best, but .... on the cheap??? (talking purely in the sense of "offsite" being "somewhere else")

Back in the "we have no money" days where I work, the backup system I wrote was capable of this "take it home" style of offsite with the ability to get at the data from the remote location.

Companies with budgets (most companies) usually have some sort of "backup system" they bought and today, likely has some sort of "offsite" to cloud storage option. But might not have a way to recover data effectively many years from now. You might say "that will never happen", but there have been many many backup companies that are "no more". Veeam, for example, is not immune from destruction in the same manner. Which may mean not only no more backup system, but effectively, no existing backup restore capability either. So... you start from scratch again.

Today, my company uses Veeam and wasabi.

For extreme physical hardware cases (we have few), a datacenter trip might be the only way. VM wise, with Internet (required by 99.9999% of critical life and death compute today, sadly), you can do full restores from anywhere. If something were to happen Internet wise, if it's "whole state or country"... probably have less than 30 days to live, regardless.

[-]

malikto44@reddit

I shudder at the thought of storing tapes at home. That would make the place easily visible to motions of discovery, and because the vehicle with the tapes is doing a commercial function, either one needs commercial insurance, or one may not have insurance bother covering if there is a wreck.

Storing tapes at home is a hard no from me. Even carrying company tapes, I much rather use a company vehicle for insurance risk, or let my insurance company know I'm doing a commercial use case, at the minimum... and be prepared to eat the added cost.

[-]

cjcox4@reddit

I shudder at the idea of companies that have zero budget for core infrastructure. But.... it happens.

[-]

malikto44@reddit

I've been at a MSP that had the boss say that "backups have no ROI", and "it is cheaper to have the redo their work than deal with backups".

Of course, when something happened and a restore was needed, with zero budget to obtain things, the first thing the MSP did? Fire all their UNIX guys.

Needless to say, once the MSP's clients discovered what happened, they had a mass exodus of customers, and a competitor of the MSP bought them up for chicken-scratch.

[-]

cjcox4@reddit

Yep. I always tell businesses that backups need to be at the top with regards to core infrastructure. And, making sure backups "work" (of course).

[-]

sdoorex@reddit

At a previous company we did it this way except instead of going to someone’s home, the drives went into a safe deposit box at our bank. Also provided a nice audit trail of when the drives were cycled.

[-]

Ssakaa@reddit

could merely be "sent home" with someone.

That... while valid conceptually, falls very flat when compliance and insurance enter the discussion. If Bob has your offsite backups, and Bob's house burns down due to an unrelated set of circumstances from the cause of your loss of your online backups, does Bob's insurance cover it?

[-]

cjcox4@reddit

Emphasis again "on the cheap" (read: free)

Sometimes you do what you can do.

[-]

sryan2k1@reddit

We don't. Immutable buckets in Wasabi

[-]

Fallingdamage@reddit

and if something happens to Wasabi?

[-]

thewunderbar@reddit

Then one of the other backup locations should have everything.

The odds of every on site backup we have burning to the ground at the exact same time that wasabi also suddenly becoming not a thing are not quite zero, but they are whatever the closest decimal point to zero is without actually being zero.

[-]

sryan2k1@reddit

We back it up somewhere else? Wasabi is just the offsite copy of what we have on site.

[-]

ThecaptainWTF9@reddit

This is the way.

Put buckets in immutable storage.

[-]

thewunderbar@reddit

This is our way.

[-]

plump-lamp@reddit

We don't. We use 2 separate immutable storage appliances at different sites and layer in an an immutable s3 target

[-]

Fallingdamage@reddit

How do you ensure they are immutable? If they're connected and powered on, they can be modified correct?

[-]

sryan2k1@reddit

How do you ensure the company handling your tapes doesn't accidentally throw them away or give them to someone else by mistake?

[-]

Fallingdamage@reddit

Do your backups yourself instead of trusting a third party company?

[-]

sryan2k1@reddit

We do. And then we pay a company to securely store them off site

[-]

Kuipyr@reddit

Why are you giving your tapes to another company?

[-]

malikto44@reddit

Offsite backups are always a good thing, provided tapes are encrypted, and there is more than one copy.

[-]

malikto44@reddit

This can happen, but this is why you use encryption on tapes. Just make sure to have a key management system. Some companies set a long pass-phrase as a master key, and that's that. Others rotate the keys out every so often. Some use specialty appliances to ensure each tape has its own key.

You can use sealed bins with barcodes, to ensure that tapes remain inside those, or you can have the offsite company scan each tape and place it on your shelf, and then every 6-12 months, have them audit their inventory and compare it with yours.

[-]

plump-lamp@reddit

they're locked down appliances. Not even administrators can delete locked data until the object lock expires.

For s3, the cloud provider's account team would need to be engaged to modify files inside object lock dates.

Two separate vendors/designs both with their own locking system

[-]

Fallingdamage@reddit

So you're saying theres a chance..

[-]

malikto44@reddit

Nothing is 100% immutable, but if the OS is secure, someone can't physically walk up to the box, boot a recovery OS, and one can't get to the OS generally, a box running something like a MinIO successor can be good enough.

I personally only consider "immutable" being optical (pretty much completely useless these days, capacity-wise) or WORM LTO tapes, but even that, the "WORM" bit is controlled by the tape firmware... but it is good enough. However, if one does it right with building an appliance to deny a foothold to attackers at the OS level, where they can't easily nuke object-locked items, it is good enough to withstand a ransomware event.

[-]

1fatfrog@reddit

On a separate identity plane?

[-]

plump-lamp@reddit

Local for break glass and sso for ease of use. No matter the access level you cannot delete data. Alerts generated for any notable actions

[-]

shimoheihei2@reddit

It depends on your scale. In most cases, having a backup server at a remote location that reaches in to pull backups from your main network is a very solid solution. No one on the network should be able to log into that remote server. If you really want immutable backups, several cloud solutions offer that feature, such as Object Lock on AWS S3. But at the end of the day, if you want a 100% guarantee, physically pulling the disk / tape out so it's offline may be the only option.

[-]

Icolan@reddit

We are using tapes for now, but that will be changing soonish. We have a project slated to start in January to implement Rubrik on-prem appliances and Rubrik Cloud Vault.

[-]

Excellent-Program333@reddit

Wassabi Immutable

[-]

charmin_7@reddit

Immutable and LTO8.

[-]

e_t_@reddit

There are desktop LTO drives. Some even have Thunderbolt connectors, which might be easier to use with workstations than drives with a native SAS interface. Cheap they are not.

[-]

malikto44@reddit

Definitely have tape software as well. They are not cheap, but LTO is the only local backup method in town these days. (No, backing up to HDDs is not really a good thing, due to how delicate HDDs are, and no guarentee of archive life of media.)

[-]

Schrojo18@reddit

My old work uses to have a tape library in each of their data centres (leased a rack in a colo) and then monthly took those tapes to one of our other sites. Now there is just one tape drive located at that secure site and tapes just need to be popped out and put in the cabinet and new ones put in to replace them. So if there is a catastrophic failure in A/both DCs then we can recover from tape there else if that locations has a fire etc then the DCs have prod/primary backup and can keep going.

[-]

Confident_Guide_3866@reddit

We dont, we have local block storage and immutable s3 store offsite

[-]

krattalak@reddit

If you have the scratch for it, we use Safemode on our Pure arrays.

[-]

sryan2k1@reddit

Doesn't stop a fire though. And yes you should have safemode on if you have the space for it. I think ours is set to 7 days.

[-]

cwm13@reddit

Active Cluster between 2 buildings across campus from one another and snapshots replicated off-site 3.5h north of us. Plus safemode.

[-]

bagaudin@reddit

We have customers who do tape backups remotely either to locally attached tape drive or to centralized tape storage where involvement can be minimal with tape library/autoloader setup.

[-]

Excellent_Pilot_2969@reddit

The USB drive is the best low-cost solution. Super reliable, as long as you don't have the 'human error'...

Here's an even more old school idea that works. You need a programmable power receptacle or 'smart switch'. Run a task that will send a command to it to switch it on/off. Use it for the external power supply of your USB drive or if it's a NAS, the network switch that connects to it. If you don't want to kill the power to the drive, use a powered USB hub and switch its power off instead of the drive's.

[-]