20TB, 20gb a day, onsite backup options

Posted by lemachet@reddit | sysadmin | View on Reddit | 65 comments

scenario is we have content being created, up to 20GB a day. This data is captured from the source media (SSD, SDCard, whatever) imported via a Macbook, onto an OWC Thunderbay 8. This data needs to then be de-identified, at which point it also becomes smaller, and once approved, we can ship it off somewhere else. Once this happens, the raw data on the source media can be deleted (but it remains on the OWC and will need to be kept for 7 years.) ONLY de-identified data can be taken offsite in any way, shape or form. I'm trying to find a way to back up this raw data on a daily basis. Whatever it's backed up to will then need to be stored in a fireproof safe. The idea being, if there's a fire and they lose all the equipment, at least the raw data is there. If they lose both the equipment and that raw data, well, that's a problem. Based on this, I'm pretty sure LTO is the best way to go, not just because it's easier for the people on the site to switch out a tape, but also higher capacity. My plan is to get a Windows PC with LTO8 or 9, run Veeam B&R and grab all the content off a file share on the Mac, stream it to tape, put it in the fire-proof safe. I have also found IoSafe which looks awesome, and OWC have a thunderbolt connected LTO which o could daisy chain to the existing das. Is there a better option, or alternatives I haven't considered?

Reply to Post

Reply

65 Comments

[-]

user3872465@reddit

So with 7 Years and 20gb/d you are looking at just about 51TB, Round it up to a nice 100TB. Just get an onsite and offsite NAS and push a sync for the new data accross them daily. Not much work, not to expensive and esier and cheaper than tape.

Reply

[-]

malikto44@reddit

I dealt with a similar item where footage had to be on prem, either in the office, or somewhere else belonging to the company. At one job, what I did was get the boss to rent out a small office across town, and build it out. Part of it was a mini data center with a couple racks in it. It connected to the main office via a VPN. At the office was a NAS and a small LTO autochanger. Every so often, someone would stop by the remote office, pull all used tapes out of the autochanger, place them in a nearby fire-rated safe, then label and throw fresh tapes into the silo. Since not many tapes were used, and there was a 1U server that did the tape backups from the NAS to the silo, this was effective for backups. At the main site, the data was generated and stored on the primary NAS, the backup program would back it up there to the backup NAS/SAN. Data was synced to the secondary NAS, so the NAS at the remote site's backups were separate from the primary site. This worked out well, but my worry was geographic separation... both locations were affected by electric grid or weather conditions.

Reply

[-]

lemachet@reddit (OP)

that would be a great solution and it was my original thought process "we'll just send it via vpn to your other office" but nope, it's a very clear lease condition that any footage which has not been de-identified cannot exit the facility in any way, shape or form. It's not enough to say "oh it's just shipped to our head office".  we can't take the raw data on tape, or disk, offsite. Only the de-identified data. (and backing that up isn't a problem)

Reply

[-]

malikto44@reddit

In that case, probably the best thing is to get a good, high quality media safe with 2-3 hour fire rating, sling data onto tape or disks, and use that. Chubb has a good reputation, and I've seen their safes go through a lot with zero maintenance.

Reply

[-]

spokale@reddit

A fireproof safe will do exactly zero good. You CAN put the data offsite. There are NO regulatioms so strict ad to preclude that. Sure, de-identify and encrypt it on your end. But a backup is not a backup if it is on site.

Reply

[-]

lemachet@reddit (OP)

.... Let's say you rent a car. The company which owns the car says "You're not allowed to smash it into a wall. It's a condition of us letting you rent the car." You CAN'T smash the car into the wall. Well.... you can.. but the penalties for breaking the condition of the car company letting you use the car are immense.  It's the same thing. Facility owner says (to ALL the tenants) "We can rent you space and allow you to use this facility. But it's a very strict condition of the lease agreement that any content which may identify anything undertaken by any other tenant of the facility cannot be removed from the facility. If you breach this condition, you owe us eye wateringly large sums of money and, probably the tenant whose project you identified, however inadvertently." I do understand the perspective of it not being a true backup if it's on site, howeve in this situation, there ARE regulations which prevent it.

Reply

[-]

spokale@reddit

That is not a regulation, that is a corporate policy.

Reply

[-]

lemachet@reddit (OP)

I never said it was a regulation. Just that it's a requirement of the people who own the facility. We aren't backing up other people's data but in the process of generating our data, there may be for example, a person in the background, who is wearing a corporate branded shirt and standing in front of, I dunno, a new Wheelbarrow they are preparing to take to market. But what they are doing is highly secretive and it's not generally widely known. So our data needs to not show that person's branding on their shirt, or their wheelbarrow.  If we send that data, with the person's branded shirt, offsite, we're in breach of the lease. So we need a way to reduce the risk of losing that data while it's onsite. Once we've removed the branded shirt, it's all quite straight forward.

Reply

[-]

spokale@reddit

So it's video data specifically? In that case, do you have any ability to physically alter how the video is captured to limit capturing other tenants? For example, can you limit recording where needed to more secure locations or add privacy screens around the work area, etc.

Reply

[-]

lemachet@reddit (OP)

Yep video specifically, and no it can't really be limited. We might be testing our shovel in a designated shovel testing area using a.drone to record. Joe might be queued up waiting to get into the shovel testing area and the drone can see him. It doesn't really lend itself to privacy screens

Reply

[-]

spokale@reddit

Corporate policy is not the same thing as regulation; I maintain, there is no regulation which precludes offsite backups. None exist.

Reply

[-]

lightmatter501@reddit

You need to automate the de-identification process. You can sell it as having less humans involved in the sensitive data. Then, you can automate network transfers out after you place a paranoid level of post-quantum encryption on the data. Use asymmetric cryptography so that retrieval of the data requires a different key or set of keys that is duplicated and kept in safe deposit boxes, a safe in the cto’s office, etc. If you go multi-key, you can make it so that at least 2 of the ceo, the cto, and the lead sysadmin need to provide their keys to unlock the data. Any auditor who is unhappy with this solution may be reminded that this is a paranoid way to store backups. The hardest part will be automated scrubbing of personal data, but in theory it should be doable otherwise 20gb per day would be impossible to review.

Reply

[-]

lemachet@reddit (OP)

automating the de-identification would be cool actually and i'll keep that in mind. Even once it's de-identified though, it needs to be approved for export by the facility owners, so we can't automate that part.  There's no personal data at play here, which is good

Reply

[-]

lightmatter501@reddit

You might be able to get the facility owners buy-in on the automated solution, and only need them to review changes to the solution. You can pitch it to them as a way to decrease labor costs on both ends. If it’s the exact same transformation every time, you might be able to get them to approve that series of steps, or even individual binaries, requiring re-approval with human verification on any change.

Reply

[-]

lemachet@reddit (OP)

nah it's not the same transformation So we might go and film, say, the process of making our new paint formula (I'm absolutely making this up!) in the background of out footage, is a guy wearing a dulux shirt, and he's putting a specific number of drops of whatever, into their paint mixture to make it slightly smoother or whatever. we need to blank his branded shirt and his adding those specific drops using a particular tool or process, out of our video. next time it might be a lady in a branded hoodie, making sure that when she tips over the newly developed anti-spill wine bottle, the wine doesn't spill out. etc  It's very frustrating needing to be kinda vague but also state a clear concept :/  appreciate your feedback

Reply

[-]

lightmatter501@reddit

Aha, so you need 3 PhD students and a GPU server to do some AI work.

Reply

[-]

RCTID1975@reddit

How long does it take to de-identify your data? What's the definition of "on-site"? ie that very specific physical location? Equipment you own? A building you own/lease?

Reply

[-]

lemachet@reddit (OP)

24-48 hours for a video editor to go through it. That very specific location. They have a lease agreement for a portion of the facility, a condition of lease is that the data must be de-identified.

Reply

[-]

RCTID1975@reddit

So realistically, you're really only talking about 20-40GB that needs to fall into this category? Once everything is de-identified, it can be shipped off site? If that's the case, I'd just go with a series of USB drives in a fireproof safe. The raring of that safe depends on the value and criticality of losing that data.

Reply

[-]

lemachet@reddit (OP)

Exactly - once it's de-identified (which can take 24-48 horus for the video editors to do) it can go to sharepoint, or glacier, or where ever. (It's also smaller!)  thanks for the feedback

Reply

[-]

mrcollin101@reddit

The data can be taken offsite. As someone who has dealt with some of the strictest regulations in the industry, there is a way to get this data offsite and be compliant. Microsoft has a GCC DOD cloud that is good enough for the USA Department of Defense to store classified information in, up to and including research generated by DOD contracts. Whatever data you have is not more sensitive than that, so look at the GCC High or GCC Low contracts with Microsoft, as the DOD cloud is reserved for the DOD and their contractors. There are some requirements that you prove your data is sensitive enough to store in those clouds, as they are strictly US based, and for the GCC High/DOD only US citizens are allowed to service the infrastructure, so they don't just let every Tom, Dick, and Harry store data on there. Now I am not saying you need to go with M$, there are plenty of very secure cloud providers out there, just that you can certainly put your super-secret data in a compliant cloud somewhere. Another option would be to purchase a dark fiber lease from an ISP and set up a satellite data center far enough away to account for natural catastrophes. A properly termed dark fiber contract will meet the strictest of regulations. Storing the all backup data onsite in any way is skipping a huge part of the 1-2-3 rule of backups, 3 copies, at least 2 storage media (This could be 2 NAS, as long as they are physically separate), and at least 1 offsite.

Reply

[-]

lemachet@reddit (OP)

They are tenants at a shared facility, with other tenants. All working on trade-secret stuff. It is a condition of the lease agreement that no photographic or videographic content may be taken of any part of the facility excepting the individual tenant's own projects. The tenants want, and agree to this process as well. The footage needs to be reviewed and ensure that there is no footage of any of the other tenants projects.  Given where the place is, dark fibre, or even lit fibre, isn't much of an option really.  THey'd have the budget to do offsite. It's just a condition of the lease, a requirement for, and of, all the tenants, that any content which may identify any projects under way at the facility must be de-identified before being removed from the facility.

Reply

[-]

mrcollin101@reddit

The DOD does this exact same thing, it is called a SCIF. Look at the framework they use and go from there. Good luck, this sounds like a fun client/employer. Challenging, but definitely unique!

Reply

[-]

lemachet@reddit (OP)

thanks, that's helpful!

Reply

[-]

exmagus@reddit

Just pay for Iron Mountain

Reply

[-]

lemachet@reddit (OP)

the facility that space is leased in, will not allow the unidentified data to leave the premises

Reply

[-]

exmagus@reddit

They suck then. Iron Mountain is VERY secure

Reply

[-]

LinearArray@reddit

Veeam

Reply

[-]

lemachet@reddit (OP)

Veeam B&R will do stream to tape, but I haven't been able to find data to support installing it on a mac (I can install Veeam Agent, but that doesn't support tape)

Reply

[-]

imnotaero@reddit

I think a better way to prepare for fire than a fire safe is to have a copy of your data offsite. That's either cloud or an LTO tape. If you can stomach a RPO of two days you can keep yesterday's tape offsite. I have, however, just necessitated an encryption requirement for your tape media.

Reply

[-]

lemachet@reddit (OP)

we can't take the raw data offsite, whether it's encrypted or not. We can't take the tape. We can't send it to cloud. We can definitely back up the de-identified data via cloud. That's fine. But we can't send the raw data offsite in any way.

Reply

[-]

GullibleDetective@reddit

Veeam with offsite immutable and in flight encryption

Reply

[-]

lemachet@reddit (OP)

the issue isn't encryption. The issue is that content which could identify other tenants or other tenants projects cannot leave the site before being de-identified.

Reply

[-]

GullibleDetective@reddit

It's encrypted content and thus not identifiable that is what encryption is

Reply

[-]

fkick@reddit

Is this video media? If so, and you can get the sign off, something like Sony CI or Frame IO are studio approved solutions. Sony offers varying levels of archive and proxy creation, as well as the usually security features (automatic watermarking, login tracking, user limits etc).

Reply

[-]

lemachet@reddit (OP)

Yes, it's video media, as well as telemetry data. I will look into those products, thanks.

Reply

[-]

zeroibis@reddit

I hope you never have a fire and that they understand that a fire safe is not a magic box. They are rated for X time. Do you contract with a private fire service in order to have a guarantee the fire will be put out and the safe secured before the time runs out? Or are you hoping that your local gov is going to prioritize saving the contents of your safe over all other activities? Hope your company has good insurance because from the sound of it if your building is in a major fire especially something like a wild fire you have almost a guarantee that you are going to lose all your data.

Reply

[-]

salacious-sieve@reddit

Safes are rated for a certain amount of time but these ratings usually refer to paper. That is, they guarantee that the paper will not ignite if the fire burns for a certain period. Tapes are melting long before the ignition point of paper.

Reply

[-]

CompWizrd@reddit

You can get media safes that are specifically rated for the lower temp of tape. It'll cost a lot more of course.

Reply

[-]

TheDarthSnarf@reddit

A decent, small, true 1-hour rated fire safe for digital media, is generally around $2,000. The materials for the interior and seals are all different since they can't release steam to protect the contents., like they can for paper protection. To protect magnetic media, they have to keep the interior temperatures much lower than what paper can handle. You have to be careful too, many of the safes on the market make bold claims that are unlikely to be based in reality... so be sure to buy from a reputable safe-making company.

Reply

[-]

zeroibis@reddit

Actually this is a great point, you need to buy a safe to test out first. Then you can determine the rating yourself. A very costly endeavor for sure.

Reply

[-]

CompWizrd@reddit

And in case someone's wondering. No, you can't reuse your safe after a fire event.

Reply

[-]

NorCalFrances@reddit

It's like an ablative heat shield.

Reply

[-]

zeroibis@reddit

" 2-hour fire rating verifying that it will withstand an external temperature of 1700 degrees Fahrenheit while maintaining an internal temperature of less than 350 degrees Fahrenheit " So I guess if you want to test you could always bake your tape at 350F in the oven and see if you can still get your data off it after removing it from the oven after 2 hours. However, lets keep in mind that your not going into that safe after just 2 hours. In reality lets say the fire is out after exactly 2 hours so the box stops heating up but it will take time to cool down so the actual heat exposure is going to be a lot longer than just 2 hours. I wonder how effective nested fire safes are lol.

Reply

[-]

friedmators@reddit

Your oven will prob oscillate between 300 and 400.

Reply

[-]

zeroibis@reddit

Might want to use a smoker instead as mine generally keeps temp within +-10F and if there is no wind it is within +-5F.

Reply

[-]

NorCalFrances@reddit

So does a real fire.

Reply

[-]

antiduh@reddit

What plastic tape in the world would survive 350 without decomposing, melting, or shrinking?

Reply

[-]

tripodal@reddit

even if the tape survives it may become demagnetized.

Reply

[-]

NorCalFrances@reddit

The tape coating has a Curie temperature of around 375 deg C. I think the plastic it's coating will be long gone by then.

Reply

[-]

RCTID1975@reddit

There are other options other than going down to Staples and picking just buying what they have in stock. There are options rated specifically for this purpose.

Reply

[-]

HolaGuacamola@reddit

FYI, if you delete data from the source system and keep it in your backup, your backup now becomes the main copy and you don't have any backups.

Reply

[-]

NorCalFrances@reddit

It's a backup in the sense that it's the original raw data, which they don't really care about unless something happens to the cleaned data. So in that sense it's a sort of backup to the processed clean data.

Reply

[-]

doweactuallycare@reddit

If you're backing it up. Then deleting it. Its not longer a backup. Do you understand that?

Reply

[-]

pentangleit@reddit

If nothing changes, at 20gb a day you're going to end up with 50Tb of data needing to reside on that OWC. Do you have a second one to mirror to onsite? or are you just going to hope that nothing ever happens that means your 7 year plan is toast?

Reply

[-]

eldudelio@reddit

local back up to disk with replication to offsite obs cloud for long term insurance proof..?

Reply

[-]

OZ_Boot@reddit

Get sign off to transmit data. Use Veeam, encrypt in transit to onsite repository. Offload long term storage to immutable S3 bucket(I recommend Wasabi S3) which is also encrypted. Fire safes are OK but usually rated for a specific time, 30 mins, 60 mins etc. I still wouldn't trust fragile media in them though as you're putting a lot of faith in something that can protect 'up to 30 minutes'

Reply

[-]

RyanLewis2010@reddit

Yes fire safes are not something to be considered as option in my books. Having been thru 2 fires recently the safe may physically survive but anything inside of it after that initial time frame will be melted. Had all sorts of paper turn so brown you couldn’t read them anymore

Reply

[-]

ChanceSet6152@reddit

LTO8 will not suffice, you must go 9.

Reply

[-]

RYU_1337@reddit

Damn, sounds like a lot of work.

Reply

[-]

lemachet@reddit (OP)

Yea but it's interesting. They do interesting work and the requirements are challenging and interesting.

Reply

[-]

ProfessorWorried626@reddit

Veeam and immutable repo.

Reply

[-]

ElevenNotes@reddit

Backup > Veeam > WAN accelerator > Veeam > Tape or Backup > Veeam > Backup Replication > Veeam > Tape Whatever you prefer. If you don’t have money: Backup > rsync > WAN > Target > cron to tape

Reply

[-]

lemachet@reddit (OP)

See, you said WAN there.... We can't ship it off-site. We can't take a tape or a disk. We can't replicate to another site they control. It cannot leave the site /in any way/ without signoff by the management of the location. This includes rsync or vbr to a veeam cloud repo. Even if we could, internet is super spotty.

Reply

[-]

ElevenNotes@reddit

Backup > Veeam > Rotating USB drives > Safe Backup > Veeam > Tape > Safe

Reply