I've been too afraid to ask but fuck it. Why not a snapshot, why always a back up. What's the big difference?

Posted by OtiseMaleModel@reddit | sysadmin | View on Reddit | 292 comments

Nearly almost every thread that mentions backing up before doing something there's a comment, a checkpoint is not a back up. But a back up takes much longer to do and much longer to restore. If you are just doing something like a minor update on a tool hosted on a server in your hyper-v environment do you really need to wait 8 + hours for a back up, run your update and then if you do meet a disaster have to wait all that same time to restore? What would you lose if using a checkpoint instead? Everyone always says it, can someone please explain it?

Reply to Post

Reply

292 Comments

[-]

Einaiden@reddit

A snapshot is for the oops, a backup is for the oh fuck.

Reply

[-]

1grumpysysadmin@reddit

Way more accurate than I care to admit. I had to have a backup pulled because when i reverted to my snapshot, the issue was still present. A clean backup from a few days prior fixed the issue and I was able to proceed. Stupid thing but it happens.

Reply

[-]

Curious_Property_933@reddit

Is that a result of it being a snapshot vs a backup though, or a result of the backup being taken at an earlier point in time than the snapshot?

Reply

[-]

1grumpysysadmin@reddit

The last one. I had a backup from a few days prior to my work that seemed to work a bit better. Thankfully all this server did was host some of the SCCM tools we had so it wasn't a big deal to go back a few days.

Reply

[-]

Kodiak01@reddit

There are three levels of mistakes: Little booboo, Big booboo, and OOPS. I know what I've done when I've said oops...

Reply

[-]

Lopoetve@reddit

Always done it as "huh, uh oh, and that's funny..." Any time I say the last of those I realize it's going to be a very long day.

Reply

[-]

renegadecanuck@reddit

For me, it's when I say "huh, interesting" that it's going to be a long day. Or "well.... that's not great".

Reply

[-]

DueBad3126@reddit

I say “that’s not ideal…”

Reply

[-]

Lopoetve@reddit

“Well, that’s not great” is the moment I start looking for cheap flights. What, forgot I had a vacation to… Omaha … scheduled? Sorry boss!

Reply

[-]

Adimentus@reddit

That's about the time I close the laptop, pray to whatever god will listen, grab a coffee and mentally prepare for the battles that lie ahead.

Reply

[-]

Xenc@reddit

Pray to Linus

Reply

[-]

TimotheusL@reddit

May the new scheduler be with you!

Reply

[-]

twoscoopsofpig@reddit

We called those RGEs, or resume-generating events.

Reply

[-]

WskyTngoFoxtrt@reddit

See Also: Career Limiting Decisions

Reply

[-]

AlwaysParanoid_@reddit

Related: Career Limiting Move

Reply

[-]

Drew707@reddit

Third Envelope

Reply

[-]

NASdreamer@reddit

Warms my heart to see this reply.

Reply

[-]

Drew707@reddit

Masochist

Reply

[-]

pnutjam@reddit

One place I worked had a checklist for procedures with WFYF next to some of the steps. "We're f'ed, you're fired" if you mess this one up.

Reply

[-]

SnooObjections3468@reddit

I like that, good way to get the consequences across.

Reply

[-]

zhaoz@reddit

Pray you never live to see the mythical Ooopsie Daisy!

Reply

[-]

Kodiak01@reddit

My GM had one of those back in the day. CDK Drive (formerly ADP) has a knack for omitting confirmation queries in the worst of places. Back around 2006 or so, GM was still doing shipping/receiving here alongside me. He went to cycle count an area, but changed one small parameter. At the top of the PSMS function, you have two options: (C) for Cycle Count, (P) for Physical Inventory. When you do a cycle count, no changes are made to inventory counts until the job is finalized; you can cancel at any time and nothing gets modified. OTOH, selecting Physical Inventory wipes the counts as soon as you start the session. ALL the counts. No confirmation prompt, just "bang, everything is 0!" Back then we were doing weekly tape backups. This Oops happened on a Thursday afternoon. They had to restore back to the previous weekend's tapes and re-enter 3.5 days worth of invoices, credits, AP/AR entries, etc. Everything. That was worse than when Eagle USA Airfreight (later EGL Global Logistics, later bought out by Ceva in 2007) had a complete extended AS/400 system failure back in early 1998. For the better part of two weeks, everything had to be done with hand invoices and faxes. They told us that when the system came back online, all hands were to be on deck to start entering things into the system. The system came up around 10AM... Super Bowl Sunday. We made kickoff. They told us to just get it done, neatness be damned. I was literally throwing invoices over my shoulder onto the floor as I was flying through the data entry. I don't even remember who cleaned them up as it technically wasn't my job at that point. Honestly, if a system of theirs was going to go down, I would have hoped it would of been Lotus Notes; we were going through ISO 9001 certification at the time and that is how they deigned to store the required documents. Old, slow and horrid to use.

Reply

[-]

Busy_Reporter4017@reddit

Sounds like the P function needed to be password and confirmation protected!

Reply

[-]

Kodiak01@reddit

There is a long list of things that they know is wrong with the software but have stated they have no intention of every correcting.

Reply

[-]

zhaoz@reddit

Lol wow, so you just press p accidentally and all the inventory counts are wiped? Great UX design work there CDK!

Reply

[-]

Kodiak01@reddit

Well, technically you have to hit P and press enter, [but yeah.](https://imgur.com/owMqZpE)

Reply

[-]

zhaoz@reddit

Is there ever a legit reason to ever select P? Like seems like having a sudo rm -rf button on your keyboard. Why in the name of Odin's butthole would you have such a button?

Reply

[-]

Kodiak01@reddit

For the rare time (typically once every few years for us) that the higher ups require a full physical inventory to be completed for accounting purposes. Out of all our locations, we're the only ones NOT on the schedule to do a full inventory because we actually do our cycle counts like clockwork. EVERYBODY has assigned areas, even delivery drivers to do during downtime.

Reply

[-]

zhaoz@reddit

Heh, I would just pop out the P button on the keyboard. Or put one of those plastic protectors over the button that needs two people to turn the key at the same time to open... [like this](https://cdn1.vectorstock.com/i/1000x1000/52/70/push-the-red-button-nuclear-vector-14545270.jpg)

Reply

[-]

Kodiak01@reddit

Unfortunately that would not work. This command can be used from pretty much any terminal and login.

Reply

[-]

zhaoz@reddit

Who needs hackers? They cant do more damage then pressing P, enter. Lol

Reply

[-]

mancer187@reddit

I call those "Not like this" moments, or just oh fuck

Reply

[-]

Still-Swimming-5650@reddit

And prepare 3 envelopes

Reply

[-]

AmbassadorDefiant105@reddit

At that point isn't it a full on Whoops!?

Reply

[-]

Kodiak01@reddit

It was a reference to an old Bill Cosby routine.

Reply

[-]

mpdscb@reddit

You can hide the little booboo and probably the big booboo too, but you can't hide the OOPS.

Reply

[-]

Kodiak01@reddit

Like when I drove a John Deere 3320 with a 3 point hitch backwards through the showroom window of a powersports dealer back ~2005? I felt bad about it for about 2 hours, then the mechanics told me I was actually the 3rd one to go through that window in the previous 2 years.

Reply

[-]

chrismholmes@reddit

Technically both work pretty similar, but I’ve often found, someone will shut the machine down for a snapshot which sometimes means this is better than a backup. (Obviously this should be done with clustered services and etc) People fail to forget that some VMs include running database scenarios that a backup may not fully get that locked file. It may not include the log replay events. It may not have gotten everything you wanted based on how bad the policies/procedures are, integration in to the OS/application and much more…. A shutdown vm snapshot has always been my main go to. Now I will say, if a vm version upgrade goes bad, you better have a good backup. You are going to need it…

Reply

[-]

Powerful-Ad3374@reddit

No need to turn the server off. Just stop the application services and SQL and take a running snap shot. YOLO. Genuinely had to do this on some of our servers that were in such bad states that reboots were considered to risky

Reply

[-]

PkRavix@reddit

That's why you use a backup tool that is application aware.

Reply

[-]

Highawk_@reddit

Ah fuck - a little whoopise dasiy Oh fuck - I've made a irrevocable error

Reply

[-]

moldyjellybean@reddit

I did both before any medium to major change. Always best to have 2 options when things go south.

Reply

[-]

UglyKidJoe1234@reddit

Usually in that order! lol

Reply

[-]

FredrickandNeval@reddit

Correct answer.

Reply

[-]

Charlie_Root_NL@reddit

Gosh this have me a smile

Reply

[-]

mpdscb@reddit

Exactly.

Reply

[-]

Training-Swan-6379@reddit

😆

Reply

[-]

compudude@reddit

This, x 100. We snapshot for change testing, but backups are the "oh shit it's gone" savior.

Reply

[-]

ycnz@reddit

This is the correct answer.

Reply

[-]

lutiana@reddit

A backup is a snapshot stored completely out of reach of the production network. If your prod goes up in flames, you need something completely off network to restore from. Snapshots would be gone (or unreliable in events like ransomware).

Reply

[-]

darklightedge@reddit

I believe that's described in the 3-2-1-1-0 backup rule: [https://objectfirst.com/guides/data-backup/3-2-1-backup-rule-and-strategy/](https://objectfirst.com/guides/data-backup/3-2-1-backup-rule-and-strategy/). Given that the number of cybercrimes dramatically increases, making backups without immutability is becoming risky.

Reply

[-]

Glittering-Camel4518@reddit

This is the actual reason. Also there are backup solutions with really fast recovery. There's the Datto BCDR suite that has everything running again in 20-30 minutes.

Reply

[-]

antiduh@reddit

With zfs this distinction dissolves a little bit, because you can send snapshots.

Reply

[-]

suglasp@reddit

NetApp can do this to. Offload older snapshots from production storage to cold storage tier in remote datacenter. Last week had to restored a vm from a few weeks ago from snapshot without problem. Cloning a snapshot of a few terrabytes to a writable volume takes just a few seconds to get it up and running. The only thing with this setup it there is no air gap possible if you want to able to restore fast from the cold tier, so you still need an extra backup solution for long term keeping those cold storage snapshots offsite.

Reply

[-]

arvidsem@reddit

If you are using ZFS, then you need a reason for doing any other than sending snapshots to an off-site server. (An already functioning backup system or regulatory requirements are good reasons). If nothing else, an on disk ZFS snapshot is generally dramatically faster to restore from than a tape or cloud backup.

Reply

[-]

thortgot@reddit

Having backups that have no requirements of existing hardware, configuration or environment is the primary point. Every production solution should have at least 1 level of offline or immutable backup.

Reply

[-]

randomman87@reddit

Not just that, but a backup should be a fully self-contained restore option. Like VHDX. A snapshot (or checkpoint) on Hyper-V is a differencing disk of the VHDX and a capture of those point-in-time settings. After a few days I've had them go tits up and refuse to rebind to the parent VHDX. Bye bye snapshot. You also need to make sure the virtual hardware is *exactly* the same. VHDX you can reattach to a different virtual machine entirely. Just make sure you weren't using a virtual TPM and encryption.

Reply

[-]

RemCogito@reddit

well then export the checkpoint and you'll have just he vhdx.

Reply

[-]

randomman87@reddit

Yeah but most of the people using snapshots/checkpoints as backups aren't doing that which is the issue

Reply

[-]

CARLEtheCamry@reddit

I was told by my VE team (but never tested/verified) that when you delete a VM from the host, it takes the snapshots with it. So - if you made the huge mistake of accidently deleting a VM from disk... that's it. Without a proper backup that is. Something top of mind as we are now deploying hosts to remote sites where bandwidth is a concern and over-the-wire backup is not possible.

Reply

[-]

computerguy0-0@reddit

So so so many people run their backups on the same infrastructure as their production network. Are they share something common like a domain with the same admin users. I lay into them constantly on Reddit, Facebook, and real life. It's amazing how deeply committed they are to being fucking stupid. Entire MSPs I've called out for this that are much bigger than mine. Storing client data on infrastructure with the same RMM tool. For using the same hypervisors with a production server and a backup server. It's just so disappointing.

Reply

[-]

Netstaff@reddit

Backup is NOT snapshot by definition.

Reply

[-]

Consistent_Chip_3281@reddit

Ya snap ahots are coo but what if ya know? I feel snapshot before making change but god also add backups to

Reply

[-]

Pelatov@reddit

Snapshots are ok as a temp measure if you make sure to delete them. Otherwise you get a delta vmdk that grows exponentially and then fills your datastore and crashes your sql database server Ask me how I know.

Reply

[-]

Busy_Reporter4017@reddit

You want also backups and offsite disaster recovery options.

Reply

[-]

memoriesofanother@reddit

During testing we found that we lost over 90% of iops on busy vms when running on snapshots. VMware also documented similar experiences. 30k iops to roughly 3k post snapshot.

Reply

[-]

dopefish23@reddit

Interesting, any chance you have a link that documents some stats around that?

Reply

[-]

avaacado_toast@reddit

I am very surprised at how many people on the sysadmi n sub think backups and snapshots are analogous and it is only the use case that is different.

Reply

[-]

rs217000@reddit

I've had to clean up the mess of folks who think snapshots are backups (e.g., 1.5 TB snapshots on 200 GB VMs). Merging SQL servers with 5 year old snapshots is a nightmare if you don't have the space handy.

Reply

[-]

skidleydee@reddit

If snapshots are being used correctly and removed on time. They have a really good track record. Of the tens of thousands of snapshots that we have taken every day, maybe once every several months one will fail to consolidate and even at that it works after the second or third attempt.

Reply

[-]

avaacado_toast@reddit

SNAPSHOTS ARE NOT BACKUPS.

Reply

[-]

thortgot@reddit

He wasn't arguing they are the same, he's saying if used appropriately they can be analogous to them. Use snapshots as appropriate. Offline backups are a hard requirement anyway. You can't rely on a snapshot as your primary backup, that's insane.

Reply

[-]

skidleydee@reddit

Not quite, I'm saying because they so often don't present any issues. People use them as the same whether that is appropriate or not.

Reply

[-]

skidleydee@reddit

My guy go read my other comment in this thread. I'm saying that because backups "just work" meaning they rarely fail people end up using them that way. They only look into what doesn't work.

Reply

[-]

JAFIOR@reddit

As a storage administrator, I'm still surprised how many admins in real life think the same thing.

Reply

[-]

hideogumpa@reddit

If SHTF then you're going to go back to last night's backup If you just want to be able to roll back to the state just prior to doing that minor update, then snapshot is what you want Just be sure to delete that snapshot if you don't need to revert

Reply

[-]

signal_lost@reddit

[Meh, snapshots are not as dangerous as they used to be (vVols and ESA)](https://youtu.be/VWVY1TRud_w?si=8dHv-3RS0NMVbSls)

Reply

[-]

LORRNABBO@reddit

Not me seeing a cluster of 300 TB going read only, with "only" 100 TB utilized by data/Vms, and 200 TB of old, patching snapshots never deleted for years.

Reply

[-]

signal_lost@reddit

On vSAN ESA or vVols? If it went read only that was likely from capacity exhaustion…

Reply

[-]

LORRNABBO@reddit

Yes, storage turned read only because it was at 95%, snapshots can indeed fuck up your space

Reply

[-]

zenmatrix83@reddit

thats assuming your using vsan or vvols.

Reply

[-]

signal_lost@reddit

NFS can also offload snapshot chains (and the entire chain as of the newer releases) for some filers. You used to have e to use: snapshot.alwaysAllowNative = "TRUE" NFS Improvements _NFS required a clone to be created first for a newly created VM and the subsequent snapshots would be offloaded to the array. With the release of vSphere 7.0 U2, we have enabled NFS array snapshots of full, non-cloned VMs to not use redo logs but instead, use the snapshot technology of the NFS array in order to provide better snapshot performance. The improvement here will remove the requirement/limitation of creating a clone and enables the first snapshot also to be offloaded to the array._ https://core.vmware.com/resource/whats-new-vsphere-7-core-storage#sec13398-sub5

Reply

[-]

xxbiohazrdxx@reddit

My main issues with vVols is that the VASA providers are touchy (although its gotten much better with newer versions where you don't have to deal with hours of certificate bullshit to get them to work). Vendors not building the VASA provider into the appliance itself is also fucking maddening. Having to run a Windows VM with some touchy bespoke garbage software so vSphere can communicate with the storage appliance is trash

Reply

[-]

signal_lost@reddit

Who’s still running vasa on windows? There’s a lot of decoupling on how vasa works to where it can span arrays plural and other stuff that will make resilience a lot better especially at scale and for coordinating things like XCOPY or vVols between federated arrays etc. I was initially skeptical of vasa outside of an array controller, but at a certain scale point, you do need a scale out VASA architecture.

Reply

[-]

xxbiohazrdxx@reddit

I guess I shouldn't have said Windows (although iirc IBM Spectrum Virtualize's VASA provider is Windows based), I just meant the provider in a VM period. Although I take your point about scaling and multiple arrays.

Reply

[-]

NimbleNavigator19@reddit

Look at you with your deleting snapshots. I have to harass people for weeks and still end up just doing it myself.

Reply

[-]

Pidgey_OP@reddit

10 days after every patch Tuesday we take about 400 snapshots and it's pulling teeth to get the person responsible for pre-patch snapping to also come through on tuesday after patch weekend and delete them all It's 2 lines in power shell. Please.

Reply

[-]

MrGuvernment@reddit

I got lucky and got to make the policy, you got 3 days (vmware recommendation for heavy I/O loads to delete after) If you have not tested thoroughly enough with in that period, explain why, or take a new one, otherwise, not my problem. because those same people come and cry about performance before when they 3 month old snapshot that is a couple TB's in size is still there from patching...and 2 cycles since then..

Reply

[-]

ipreferanothername@reddit

I convinced my group not to snap before patching, we have 40 maintenance windows... Not going to manage the snapshots. It's super rare a patch is so awful that we ever had to use the snap, and we get backups every night. They didn't love it but... In a year it was a problem maybe for 2 vms. So many people were saying they didn't snap for patching so I pushed in that direction.

Reply

[-]

Pidgey_OP@reddit

we don't snap for autofix patching since any server in that group is going to be largely replaceable without direct business impact (hence why theyre allowed to reboot whenever the tool wants to reboot them) Patch weekend is stuff we care a little more about and is just once a month (and creating the snaps is also just 2 lines of powershell)

Reply

[-]

Reynk1@reddit

We do forced snapshot removal after 7 days, bigger servers after 48 hours. Cut number of outages due to snapshots to 0

Reply

[-]

Dangerous_Injury_101@reddit

I wish there was auto-delete after X days option when you create the snapshots...its not like you ever want/need to revert to some million years old snapshot

Reply

[-]

RockinIntoMordor@reddit

I worked in Cloud Backup, so different stuff, and we had automated incremental backups, like snapshots. But it's amazing how often a user will say "Our business urgently needs this email/file from the 16th three years ago, and may have been deleted on the 17th"

Reply

[-]

AmDDJunkie@reddit

We run Simplivity and with that you can set a date to delete a backup. Very handy, I wish I could do the same elsewhere.

Reply

[-]

human8264829264@reddit

It's called a cron job

Reply

[-]

Reynk1@reddit

Ours is an Ansible job, but same difference

Reply

[-]

flattop100@reddit

What about snapshots causes outages?

Reply

[-]

courser@reddit

They're not static. They will continue to grow as long as they exist. Get a few growing big enough for long enough and down goes your datastore.

Reply

[-]

RegistryRat@reddit

Never knew this. I've never really done snapshots, just always worked off of backups. Good to know!

Reply

[-]

thortgot@reddit

The merge activity can massacre your IO as well. A medium size VM with a high rate of data change left for a week or two that needs to be merged can hammer lots of solutions. Merge your snapshots folks.

Reply

[-]

SoonerMedic72@reddit

I did this once on accident. Luckily it was in our DR site, not Prod, but still. Took a snapshot of Veeam for patching, forgot to delete it when I confirmed that it was working correctly again, VMWare support had to find and delete the delta file in the PowerCLI. Felt real dumb...

Reply

[-]

elemental5252@reddit

Linux gent here. Our VMWare gurus built the question into our ticket template for snapshot creation. If you don't answer it, they refuse to create snapshots for you. I believe a daily ansible job runs that performs cleanups of snapshots that are in the inventory. Refuse to cater to bad admins, engineers, and developers. It's your infrastructure - build it how you need it.

Reply

[-]

Pidgey_OP@reddit

It's the businesses infrastructure, not mine, and don't I dare touch it lest I hurt production. Id love to work somewhere that gave a shit

Reply

[-]

secretlyyourgrandma@reddit

> Refuse to cater to bad admins, engineers, and developers. It's your infrastructure - build it how you need it. Every time I read stuff like this it reminds me I need to get a new job.

Reply

[-]

NimbleNavigator19@reddit

I don't even get to use powershell to do it. I work for an MSP that offers private cloud hosting and the way our backend is set up means that unless you are on the team that maintains it you only get individual tenant GUI access. So every time I have to clean up after everyone I have to do it manually and jump from tenant to tenant.

Reply

[-]

Aegisnir@reddit

Why can’t you make a GPO script to delete them based on file path and age?

Reply

[-]

Vicus_92@reddit

I inherited a server recently with a 5 year old snapshot....

Reply

[-]

RandomSkratch@reddit

Ohhhhhhh fffffffff.

Reply

[-]

NimbleNavigator19@reddit

Who gets custody of it?

Reply

[-]

Superb_Raccoon@reddit

The State, someone called Child Protective Services.

Reply

[-]

ipreferanothername@reddit

I've suggested automating snapshot removal and maybe in 2 more years theyll let me They're only just now ready to let me manage vms that have been power off for months. Smh

Reply

[-]

GhostDan@reddit

Automate it. "After x amount of days your snapshot will be removed". You don't want to be in a situation where you have a billion snapshots and are trying to restore.

Reply

[-]

secretlyyourgrandma@reddit

i encountered an environment with daily snapshots of 1800 VMs with no retention policy. the cloud admins just rolled through and deleted stuff whenever it became a topic of discussion. pretty cool.

Reply

[-]

what-the-puck@reddit

Automate it. That is, automate the snapshot creation and removal process. Build in a method for people to extend snapshot lifetime, and an automatic removal if they choose not to do that.

Reply

[-]

Le_Vagabond@reddit

they usually learn after your entire VM filesystem dies after 6+ levels of snapshots and you can just roll back.

Reply

[-]

Hotshot55@reddit

> Just be sure to delete that snapshot if you don't need to revert Thanks for reminding me about the snapshots I didn't delete today.

Reply

[-]

TKInstinct@reddit

There was that post a year or two ago of the guy who had like 300+ snapshots and they were utilizing those as backups from years past.

Reply

[-]

Pure_Professional663@reddit

Spot on advice. Snapshot for quick, minor changes where you know if you've been successful very quickly. Once your change is verified, delete your unused snapshots.

Reply

[-]

ZPrimed@reddit

*laughs in storage that DGAF about snapshot chains*

Reply

[-]

autogyrophilia@reddit

Meanwhile me with sanoid snapshoting on 15 minutes intervals

Reply

[-]

triggered-nerd@reddit

Agreed

Reply

[-]

OrganicTruth4923@reddit

The age old question! Both have their pros and cons. At the end of the day a snapshot is a good ass covering measure for when things go wrong during patching and short term administrative items. The longer you leave a snapshot in place you could experience road blocks like being unable expand storage urgently to the VM completely locking up while a consolation of the snapshot is in progress. Backup will ensure you have that extra layer of recovery if you are needing to restore things like data consistent DBs, exchange mail files and so on. At the end of the day you need to ensure you are appropriately using Snapshot or Backup based on what the VM is hosting. The last thing you want to do is take a snapshot of an Exchange server that is a DAG member and revert the to Snapshot days later.

Reply

[-]

anonpf@reddit

I’ll ask you this, What do you do when the environment itself is lost? Having backups (especially offline backups is important for disaster recovery.

Reply

[-]

keystorke@reddit

Imagine you are skydiving and you have been carefull to pack ur shoot by the book this is ur snapahot, the secound parashoot is ur backup. Its verry good to have a secound parashoot in case the first one gets fuckd for some reason

Reply

[-]

pjmarcum@reddit

And backups should go offsite. Physically off site. One of my customers was hacked, the bad guys got into their backup vault and destroyed all the backups. This can’t happen with physical media that is gone offsite.

Reply

[-]

BenderMurray@reddit

You need to specifically check if you're snapshot supports the database of the server.

Reply

[-]

Kraeftluder@reddit

I'm all for explaining things to people who don't know but this is absolutely a question that falls into the category "If you need to ask, you do not understand this job or its requirements.".

Reply

[-]

chiefsfan69@reddit

If we did a backup before every update, nothing would get done. Backups occur nightly, snapshots are immediate, and restore immediately. I can't shut a system down for hours to back it up in a 24/7/365 environment. Yeah a snap can fail, but I've got storage snapshots as well, replication, and backups. And backups fail too. IMO backups are for offline retention, not update failures.

Reply

[-]

_Robert_Pulson@reddit

Snapshots are really meant for temporary/quick restores to a certain point in time. You create too many snapshots on a single VM, and it starts performing slowly. Plus, if you ever upgrade Exchange Server or the AD DB schema, a snapshot is not going to cut it if you have more than one DC. You'll want actual backups.

Reply

[-]

hurkwurk@reddit

a snapshot includes anything broken in run time. a proper backup has stopped all databases, etc, to get clean backups that are intended to be restored. most SQL databases do not support restoring a snapshot directly. There are many other products that have similar limitations, where you are not necessarily safe working from a snapshot unless you stopped those services yourself.

Reply

[-]

JAFIOR@reddit

Great point. If you restore from a snapshot of a vm that didn't quiesce, you can easily lose in-cache writes and jack up your DB.

Reply

[-]

XInsomniacX06@reddit

Depends on what you’re working on. If it’s a sql database or a domain controller you shouldn’t be taking snapshots. If it’s an app server with just a web front end a snapshot should be fine. But you should be taking normal backups of VMs regularly and then just taking snapshots for maintenance work as a rollback. You’d be covered both ways if somehow the vm becomes completely corrupted and a snapshot won’t work. There’s lots of reasons and someone will probably explain it in more details, but there are certain situations where snapshots are a big no no and restoring from backups are your only option.

Reply

[-]

-quakeguy-@reddit

Snapshotting DCs is fine, as long as one understands that the function of such a snapshot is not to restore the DC, but to backup the hidden volume where Windows Server Backup dumps your actual DC backup.

Reply

[-]

canadadryistheshit@reddit

I thought this action could cause a USN rollback if you reverted to the snapshot?

Reply

[-]

XInsomniacX06@reddit

Not if you do it in the proper way. There is a MS article about taking DC snapshots and how to do it but in practice it doesn’t save much time over just demoting old and promoting a new dc.

Reply

[-]

RandomSkratch@reddit

The process of demoting and promoting DC’s is super simple but man is it ever a pain in the ass when using integrated DNS and that one vlan that was never allocated a DHCP scope because “reasons before your time” and all those static addresses now fail DNS lookups because you demoted the primary and shit never uses the secondary in this instance. Whoops! That todo list never gets shorter.

Reply

[-]

always_salty@reddit

There's a cool read by VMware about this. https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/virtualizing-active-directory-domain-services-on-vmware-vsphere.pdf

Reply

[-]

canadadryistheshit@reddit

Thanks for this, this is a decent read so far.

Reply

[-]

-quakeguy-@reddit

You never revert whatever is running live to it.

Reply

[-]

StealthTai@reddit

Oh great, a new interesting thing I have to go read about before I can go to sleep. That is super interesting though

Reply

[-]

XInsomniacX06@reddit

Yeah there is a certain way to do it but most people don’t follow that and just snapshot and try to do some dumb shit like in place upgrade then it fails they restore and it’s a big ole mess. I just didn’t feel like getting into specifics. Senior AD engineer for the last 10 years. It’s not even really worth it. Maybe a couple niche reasons but large enterprise it’s a waste of time. In my opinion.

Reply

[-]

groupwhere@reddit

Repeat after me... a snapshot is not a backup is not a snapshot.

Reply

[-]

lnxrootxazz@reddit

A snapshot is not a backup! Snapshots work (for instance on BTRFS) using copy-on-write. A snapshot and the original from which it was created initially have the same data blocks. If this data is corrupted in some way both the snapshot and the original will be corrupted. Snapshots are useful for having local online 'copies' of the file system to fall back on or for implementing some form of deduplication, or for correcting the state of a file system to create a full backup without changing anything underneath Snapshots alone do not make data more secure. A proper backup must always be available externally and redundantly. It also has to be restoreable, which means that backups have to be tested again and again

Reply

[-]

VNJCinPA@reddit

Our backup system uses snapshots and sends them to the interwebs. We can pull files from a snapshot just as easily as a backup, and we have redundancy so we don't lose any of them. The only caveat comes down to databases, usually. Getting a good database backup often requires stopping the database briefly and saving the resulting file set. However, we back up databases to a file share then snapshot the share in case we need it. More specifically, a snapshot is everything that changed in the state of the system since the time it was taken. If you lose the files before your snapshot, you're out of luck. It's similar to a differential backup. If you only have the differential and not the base file, out of luck. Hope that helps a bit

Reply

[-]

usa_commie@reddit

Each subsequent snapshot is a new vmdk file. They must be overlayed on top of each other to present the disk. This is as I understand it: A) a performance hit the deeper you go B) risk of data corruption if any link in the snapshot chain is broken (accidentally, or by a 3rd party app) C) mgmt headache when snapshots are left on a different datastore, etc

Reply

[-]

Xzenor@reddit

Btw, good of you to ask anyway. If you don't know and can't find the answer, just ask! Feed your curiosity. It's how we all learn.

Reply

[-]

Kixsian@reddit

Snapshot is a the Delta from current state to a full backup.

Reply

[-]

KRed75@reddit

We use backup technology that uses vm snapshots. These then get replicated real-time to a different city across the country. Only changed data is backed up. As long as a previous snapshot backup has been taken, it's a very fast process. I still take a manual snapshot after the backup is complete as it's much quicker to roll back that way.

Reply

[-]

cacchip@reddit

How long would it take to do a full restore? I worked at a place that had very fast backups but then discovered that a full restore would have taken weeks with no guarantee's that everything would be good.

Reply

[-]

trueppp@reddit

What size? On decent hardware a 1-2TB is 4-5 hours.

Reply

[-]

cacchip@reddit

Our main site was next to a smoking crater in Downtown NY. We were trying to restore to the DR site.

Reply

[-]

KRed75@reddit

500GB full vm restore may take an hour or two.

Reply

[-]

dangermouze@reddit

Surprised I had to scroll down this far. Agent level backups and snapshot level backups both have their places and allow for file, service and VM level restores. Also why does it take 8 hours for backups to run? I don't handle backups now, but previously it was a few minutes for delta snapshot backups to run.

Reply

[-]

KRed75@reddit

Surely they aren't sill using tape?

Reply

[-]

ChipotleFriday@reddit

And don't call me Shirley

Reply

[-]

dangermouze@reddit

Tapes fine, but usually you go disk2disk, then tape copy runs later on when most d2d jobs are done.

Reply

[-]

Rattlehead71@reddit

Unrecoverable CRC error reading sector 24096 on DLT 19

Reply

[-]

NimbleNavigator19@reddit

Begone with your curses of the past

Reply

[-]

thegreatcerebral@reddit

D2D will go fast but not to tape.

Reply

[-]

winky9827@reddit

Probably backing up directly to the remote, instead of to a local staging area that syncs off-schedule. It's a pretty solid example of violating the 3-2-1 mantra which exists for a reason.

Reply

[-]

McGregorMX@reddit

I have a feeling that the 8 hours is the schedule time.

Reply

[-]

whythehellnote@reddit

> These then get replicated real-time to a different city across the country. Only changed data is backed up. What ensures that the replication process doesn't destroy your backup? If you want a real backup you need an offline backup which you have manually restored from onto a vanilla machine with no network resources.

Reply

[-]

TabooRaver@reddit

To be clear they're talking about new data on backup server a is copied over to backup server b. The protection is that the server software won't be overwriting backups, and any deletions are intentional. Outside of that it depends on what usecase the backups are for. If it's for hardware failures, accidental file deletion, unintentional IT fuckups or low complexity insider threats, some implementation of version control or differential/incremental is fine. If it's for cyber security DR events... it needs to be hardened. At a minimum it needs to be in a different authentication realm/domain, and backup agents should only have permission to push new data, not modify or delete. At the extreme you could implement data diodes and forward error correction, so that data can enter an air gapped network but nothing can leave. Preventing compromises. But that makes monitoring and ensure backup integrity difficult. A sane compromise is to have a 321 method, and the third copy be on tape, which will be physically disconnected and stored periodically.

Reply

[-]

D0_stack@reddit

> and the third copy be on tape, which will be physically disconnected and stored periodically. ~~Steel~~ Iron Mountain FTW. We have THOUSANDS of LTO tapes and the servers and drives in one of their mines. Servers and drives because it might be difficult to come up with the right stuff to read many years old archival data. And we do read many years old data occasionally because lawyers.

Reply

[-]

libbyson@reddit

I think this is referring to using checkpoints only as your backups. Always make sure that there is a recent backup.

Reply

[-]

Guderikke@reddit

Easiest explanation is that snapshots are still dependent on the source data, backups are not. If you lose your main data source snapshot is useless, backup can be restored and save the day. That being said, I use snapshots for changes all the time, upgrading software guess bad revert to snapshot prior to upgrade easy. It really depends on what you are doing. They absolutely still have their place.

Reply

[-]

HunnyPuns@reddit

A lot of good answers in here. I'd like to expand on backups and snapshots and RAID and all of these fun things that we get to deal with. I've always found it helpful to think of backups as the thing you go to when everything is fucked hard. Everything that isn't a backup is a thing that helps you avoid having to reach for the backup. RAID is not a backup. But it will absolutely save you from having to go to the backups if a hard drive dies. It seems like echoing what a lot of people are already saying. But I feel like this is a category unto itself. It's not disaster recovery, it's disaster mitigation. A pinch of prevention vs a pound of cure. That sort of thing. I feel like talking about these tools in these terms would allow us as professionals to better understand the differences in the tools, and where they would be used in your overall business continuity plan.

Reply

[-]

Leucippus1@reddit

Has no one yet mentioned databases? Databases. There, now you know. You *can* snap a database and you can normally recover it OK but the process for backing up a database does more than simply 'back it up.' Having said that, a lot of enterprise backup tools of today do a snapshot of the database and then set the backup bit after the fact and do a (ohhh, I forgot the word...) process against the snapshot to create a database backup in case you need to go get one row from it instead of the entire damned thing.

Reply

[-]

BraindeadGenius1054@reddit

Backup takes no time at all if you're on something like Backupify or SaaS Protect. You and your clients are perpetually up to date.

Reply

[-]

smftexas86@reddit

Nowadays snapshots and Backups work together. Have a couple of days of rolling snapshots to make it faster to restore your data, but keep backups on different storage that can't be messed with in case of bigger issue (ransomware etc). For updates etc. I'd argue that a simple Vmware snapshot or hyperv checkpoint is plenty, as long as you already have a good backup plan in place.

Reply

[-]

aussiepete80@reddit

Netapp guy here, snapshots ARE our backups.

Reply

[-]

inkonjito@reddit

One of the reasons would be. If you perform maintenance, updates of something that might affect multiple systems and or database. You can’t just restore a snapshot/checkpoint to the original point just like that, it might cause some corruption somewhere. For me it’s always been I create a backup. And I take a snapshot (if possible the snapshot of the machine in a powered off state). I once did an update which took a few hours, then it all was running smoothly, except for one config file that was changed in the past to meet some requirements, but overwritten to default by the update process. I knew roughly what was changed, but not exactly. If I had to restore the snapshot to get the previous config, I had to do the update again and could end up with some weird potential database issues. Because I had the back-up, I could simply browse the backup, copy the file from it using file level restore and make the necessary changes on production server. Creating the back-up up front saved me hours of work later. Also, if there’s a snapshot and for some reason you need to make changes, like expand a disk, that’s not allowed if there’s an active snapshot. (Yes I could’ve taken another snapshot after the update and restored to the first one to retrieve the config and then go back… but I wouldn’t want to take that risk). Not to talk about some potential corruption on your storage while removing the snapshot or restoring it.

Reply

[-]

godzillante@reddit

a snapshot is just a point-in-time state of the disk. you need the original disk in order to restore from a snapshot, which basically means “discard all changes from this moment on”, so you can’t restore if the drive goes deleted. a backup is a full image saved separately, and can be restored at any time

Reply

[-]

gruntmods@reddit

Literally had a snapshot fail last week in my homelab, wouldn't rely it in production

Reply

[-]

OtiseMaleModel@reddit (OP)

Okay but how perfectly managed is your homelab tho?

Reply

[-]

gruntmods@reddit

how perfectly managed is an enviroment where you need to roll back?

Reply

[-]

OtiseMaleModel@reddit (OP)

A lot more perfectly managed then my home lab lol

Reply

[-]

darrynhatfield@reddit

What you said is correct. to be more specific, a snapshot is a quick backup/full restore. A backup, can stay in your backup pool for any period of time you wish, it can also be restored granularly. As a result both have their uses but for patching etc, snapshots are preferred

Reply

[-]

whiteweather1994@reddit

Snapshots typically grow in size over time, thus making storage less tennable as time goes on. Backups, while larger, are static, thus easier to maintain.

Reply

[-]

Suitable_Box_1992@reddit

Snapshots are good when you fat finger something. Backups are for when your datacenter is on fire.

Reply

[-]

Far_Investigator9251@reddit

If you are using hyperv if you have a bunch of checkpoints and dont merg them in its a huge shit show to recover sometimes! Had a customer with 20 checkpoints and no hard drive space to roll them in, what a shit show!

Reply

[-]

badlybane@reddit

Yea think of it this way nested snaps are greate but the way snapshots work in vmware excetera is not meant to stick around. While a snap shot it active all the changes are backed up but the snapshot then runs. Your now using up more hard drive space and are depentdent on your previous snaps not having an issue. I have stumbled upon companies wondering why their vmware stack is backed up a or hyper-v and seen a snapshot that got stuck and was never cleaned up for over a year. If you tried to roll to a year old snapshot bad things would happen. Snapshots need to beconsolidated regularly. Also if your not backing up and someone drops a rat or microsoft releases a kb that explodes six months later. You won't have a clean system to restore from. Remember Replication is not a backup.

Reply

[-]

SilentPrince@reddit

We do daily backups so I just snapshot and go at it.

Reply

[-]

The69LTD@reddit

Por que no los dos? I do both, but backups are pretty fast when doing an incremental with a properly setup VMware/Veeam environment in my experience. I then delete the snapshot after a day or so when I get another auto veeam backup to roll my changes into the incrementals.

Reply

[-]

ssiws@reddit

If the workload supports VSS then you can use [Production Checkpoints](https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/choose-between-standard-or-production-checkpoints-in-hyper-v) in Hyper-V as a backup before doing something risky.

Reply

[-]

Bitter_Rhubarb9126@reddit

yeah, just leave those checkpoints there forever, like the dickheads who used to work at that one school i worked for, "wHy is tHE DIsK geTTiNG sO FuLl on the shitty little server i virtualized for no reason"

Reply

[-]

NameEnvironmental358@reddit

Why not both? They serve different purposes

Reply

[-]

Frothyleet@reddit

Also, the longer the snapshot exists, the more compute resources and time required to merge back into the original VM. If you roll the snapshot back, you're just nuking the temporary difference disk and you're back to previous state. If you delete the snapshot, your original VM has to pause and all of the "differences" get reintegrated into the original VM.

Reply

[-]

Spiritual-Stand1573@reddit

Snapshot saves your day, backup your life

Reply

[-]

meh_ninjaplz@reddit

Size matters

Reply

[-]

atroxes@reddit

This subreddit it about gaining knowledge. Never be afraid to do that.

Reply

[-]

SausageSmuggler21@reddit

A snapshot is considered a CDP, continuous data protection, copy. This is a very fast backup, stored locally, that can be recovered very fast. The near instantaneous backup times allow you to perform snapshots many times a day for more recovery options. However, snapshots are vulnerable to the same problems as the data: hardware failure, accidental deletion, encryption by malware, etc. Also, snapshots take up storage space on your storage arrays. If you are running out of space and are not able to procure more storage, you will almost definitely delete some or all of your snapshots. A backup is a point in time copy stored on non-local storage, e.g. Data Domain, the cloud, external hard drive, tape, etc. Backups take longer to complete, so you typically only perform a backup once a day, although there are many exceptions these days including database log backups. Backups are frequently performed at the OS or Application level, compared to snapshots which are frequently performed at the storage level. Thus, recoveries can be done for OS files, databases, and other applications directly. The big differences are the retention policies, the storage locations, and the DR/ransomware recovery capabilities. Backups are almost always cloned offsite to protect against a site disaster. Backups are frequently stored for at least 30 days, and frequently for a year or more, to meet compliance regulations. Backups can also be used to restore for a ransomware/encryption event. Proper backup software will include analytics of the backed up data to determine if a certain backup is clean or corrupt, which allows you to recover clean data to production much quicker than if you had to recover to a quarantined space, scan the data, then push into production. For many types of data, if you are only doing snapshots, you can be fined by the local government for not meeting specific requirements for certain regulations. Depending on what country your data lives in, Manufacturing design data, personal (PII) data, certain corporate financial data, R&D data, medical patient data are some examples where you could get into some trouble if you don't have proper backups. And, as a P.S., none of the above is all encompassing fact. As with everything in life, there are no absolutes.

Reply

[-]

DontTakePeopleSrsly@reddit

Do hourly storage snapshots and you don’t have to worry about snapshots.

Reply

[-]

FakeGatsby@reddit

Depends on the environment too, some backup technologies make snapshots exponential to the backups.

Reply

[-]

Miserable-Garlic-532@reddit

Have you ever tried to delete a snapshot after a couple months on a busy server? It can take a very long time. And the space required to store it can exceed the original vm.

Reply

[-]

ctrl_alt_lynx@reddit

I'll throw in a scenario I don't think I saw when browsing the comments. Clustered systems. One application I maintain is a cluster across about 80 physical servers. Huge database. Replicated so that there's 5 copies of any given record. More than required for redundancy, but we get so many queries that we need many read replicas. The data is also sharded, meaning that not every server holds every piece of data, it's spread around the cluster. A backup or snapshot of a single server is pretty useless in terms of the data. A snapshot will rescue a bad config change, but rolling back the stored data on one (or all) servers WILL result in an inconsistent state. The only "safe" way to back up the data is to export it from the cluster into one or more volumes that can be reimported into a fresh cluster. Individual servers don't need snapshots most of the time because if it doesn't go well you just redeploy and let the server get it's copies of the data replicated back. This is probably not the most common but not super uncommon either if you deal with any sort of HPC, research, or big data.

Reply

[-]

_BoNgRiPPeR_420@reddit

So a few things... Backups don't take 8 hours unless you have a really strange environment and terrible hardware. We finish backups on hundreds of VMs in less than an hour. This works out well because we schedule our maintenance windows/upgrades right after the backups finish around 9pm. Snapshots aren't meant to be kept around for a long time. Once a snapshot is taken, the hypervisor starts writing all new changes to it. This is why VMware screams at you to perform disk consolidation. If you let it go too long, consolidation can take a very long time and cause other problems for you. Backups are entirely separate from the hypervisor on different storage, in case things blow up. You wouldn't store your spare key on the same keychain as your primary house key, would you?

Reply

[-]

thortgot@reddit

Lots of environments can have backups that take 8+ hours for a full. 20+ TB file servers are not unusual.

Reply

[-]

_BoNgRiPPeR_420@reddit

I'd consider those unique environments. The 1% of businesses large enough to have that problem.

Reply

[-]

thortgot@reddit

Perhaps 10 years ago. Every mid size company I've seen in the past 5 years has similar scale servers. If isn't a giant file server, it's a giant DB or log server with logs into infinity. An incremental taking an hour is reasonable but if you are performing serious maintenance you want a full.

Reply

[-]

Longjumping_Gap_9325@reddit

I wish that was out case. Our ~4000 take several hours typically

Reply

[-]

_BoNgRiPPeR_420@reddit

At that scale, you're a bit different than most. I'd file that under "unique" environment.

Reply

[-]

cats_are_the_devil@reddit

Ideally you are backing up everyday... A snapshot is just a hotfix to get you back up and running if you screw up in the moment or need to rollback quick.

Reply

[-]

ThirstyOne@reddit

If left for long periods of time snapshots can cause VM corruption, in which case they’re useless. Backups are a definitive restore point, snapshots are for quick reversions.

Reply

[-]

trouphaz@reddit

For me, I've generally used snap shots right before making a change so I could roll back quickly if necessary. I'd have my regular nightly backups for major issues, but I agree with you. If I needed to do a full backup before making any change it would take forever both to backup and restore.

Reply

[-]

Titanguru7@reddit

Backup is a snaphost in vmware, But the snapshots get removed after backup solution does it job. If you leave snapshots they grow and cause issues.

Reply

[-]

jacenat@reddit

If your regulatory and operational requirements are low, Linux LVM snapshots **can** work at times if you know your shit. But I agree with others. As soon as you cross certain thresholds, best practices are where it's at.

Reply

[-]

bmxfelon420@reddit

It's the level of coldness so to speak. Your running VM is the meal the customer is eating, what's on their plate right now. The snapshot is there on the food warmer, just in case. But a fly could get on it, could get stale, etc. Backup appliance is ready to be cooked. . Could take a minute, but you can have it ready. But what if the chef gets sick, messes up, etc. Offsite storage is the freezer, not ready to go yet and will take awhile.

Reply

[-]

megasxl264@reddit

Backups aren't (shouldn't be) done on the same machine that's the difference. Generally speaking when someone says 'do a backup' it can mean quite a few different things since it became like a general term.

Reply

[-]

gregsting@reddit

That’s absolutely not the difference… a backup is a full standalone copy of the data. A snapshot is only saving the changes on the data, meaning to restore you need the snapshot + the current state of the data.

Reply

[-]

TabooRaver@reddit

Snapshots are actually the opposite (at elast in systems I'm familiar with). The snapshots is the vm put in read only mode, the running vm is the snapshot plus a differencing disk. Strictly speaking you don't need the differencing data to restore a snapshot, as the restore should discard that data anyway.

Reply

[-]

SpongederpSquarefap@reddit

Yeah this is right Normally you run from (in Hyper-V terms) the VHDX When you snapshot, you no longer write to the VHDX and you instead write to the AVHDX file which is chained to the VHDX file If something goes wrong with that AVHDX file and you have no backup, you're in for a bad day

Reply

[-]

gregsting@reddit

That might be a better way to look at it indeed, not sure if all snapshot systems are working the same way. The important part is that snapshot is a different concept than backup, not really related to the place where the info is stored.

Reply

[-]

wickedang3l@reddit

>But a back up takes much longer to do and much longer to restore. If you are just doing something like a minor update on a tool hosted on a server in your hyper-v environment do you really need to wait 8 + hours for a back up, run your update and then if you do meet a disaster have to wait all that same time to restore? I think that others have done a fine job explaining the broad strokes of how to differentiate between snapshots and backups. Backups take longer to restore if you're using snapshots properly. What happens when snapshots are used improperly, however, will take far longer than restoring from a backup. I inherited an infrastructure as someone else moved out of the role; the infrastructure responsibilities included Active Directory, SCCM, VMware, and a few other technologies. I began to encounter some creeping performance issues months in that I eventually traced back to VMware itself. The VM for the SCCM Primary Site server had a multi-year, multi-terabyte snapshot that was threatening to consume the datastore that it was on. Being a neophyte in the VMware space, I was familiar enough to know that this was a problem but the scope eluded me until I chose to merge the snapshot. Imagine my surprise when I began to monitor the progress of that merge. The effort was not measured in seconds, minutes, hours, or days...it took weeks to flatten that snapshot and there was no guarantee of success. The smarter play would actually have been to turn off the site server, build a new VM, and restore SCCM to the new VM. That option was still in my back pocket but attempting to merge the snapshot put me in a corner; if I had simply taken that VM offline and attempted the backup restoration to a new VM first and that had failed, I would still have the suboptimal system to fall back on. If the merge failed, I was completely at the mercy of an untested backup in an infrastructure I had just inherited.

Reply

[-]

DrAculaAlucardMD@reddit

Snapshots are for quick reversions if a change screws things up. An Oppsie button. Backups are for when things crash hard. So here is the dirty little secret. When was the last time you tested restoring either? Is your Disaster Recovery Plan up to date? No.... well you should check that too.

Reply

[-]

jfoust2@reddit

I'm sure that if OP described how much data and what kind and what kind of server and network and backup software and backup method, people wouldn't be saying "Why does a backup take 8 hours?"

Reply

[-]

klein648@reddit

In the environments where I work (VMWare) you can only have one snapshot. However, when performing automated backups with Veeam, Veeam creates a snaoshot, writes it to disk and then deletes it. If there is already a snapshot there, the backup fails. Thus, you are required to delete the snapshot shortly after. So if an issue arises 3 days later, you are fucked if you only did a snapshot.

Reply

[-]

ProtectionSubject615@reddit

Also, depending on how your VM drives are set, not all drives are included with a snapshot. RE independent drives and RDM drives.

Reply

[-]

DevinSysAdmin@reddit

...It takes you 8+ hours to run a differential backup on a single server? You shouldn't be asking snapshots vs backups at this point, you should be asking "How do I get my backups and restores within a reasonable time" Snapshots: * Short Term (Hours, Days) * Utilized before a change, sometimes on a schedule. * Do not allow granular item restores * Are on the same host/storage * Performance/storage issue with multiple snapshots Scenario: Server A has RAID failure, all data is now corrupt. How can you restore a snapshot if it no longer exists? Scenario: Server A is compromised, and all data is encrypted. How can you restore a snapshot if all data is encrypted? Backups: * Long Term (Months, Years) * Are (hopefully) not on the same host/storage * Offer granular item restores

Reply

[-]

Pristine_Map1303@reddit

![gif](giphy|gIqusaeYxgSiY)

Reply

[-]

Pristine_Map1303@reddit

Snapshots are for very short term. Don't keep snapshots.

Reply

[-]

Khue@reddit

Snapshot is a term that's not really consistent across the board but in the most broad sense of the term, it typically is just a pointer or a book mark at a specific time. For example, in the virtual machine world, typically a snapshot places a hold or a pause on the primary disk. Subsequent disk writes then get moved to a temporary space and kind of "journals" reads, writes, and deletes. The end result is that this list of subsequent disk activities on the snapshot file grows pretty ridiculously depending on disk activity because again, it's really a journal of changes to the underlying actual disk. What gets problematic is that the snapshot file usually exists in the same space as the underlying actual virtual disk for the VM and if you saturate that space, then you find yourself shit out of luck. In the anecdote above, if something goes wrong post snapshot, you just revert the snapshot and all the subsequent reads, writes, deletes get tossed and you go back to the point in time where you executed the snapshot. So if you take a snapshot at 8AM and then let the snapshot roll all of the subsequent activity done to manipulate the disk gets put to a snapshot file. At 5PM say you want to revert to the 8AM version of your VM, you simply revert the snapshot and the subsequent reads, writes, and deletes get junked. Alternatively, let's say at 5PM you are okay with all of the changes to the VM and you feel like you are safe to move forward, you then would want to "delete the snapshot". What proceeds to occur is that the snapshot then reads back the various reads, writes, and deletes to the actual virtual disk and merges all of those changes. Once the journalization of the disk activity has been committed to the underlying virtual disk, the snapshot is deleted and all disk activity resumes committing to the actual virtual disk and not the snapshot file. There are various nuances as well to merging the snapshot file that you should understand in the scenario above, but that's outside of the realm of this quick response. You need to know things about helper snapshots and current disk activities and how current activities impact merging of the original snapshot files.

Reply

[-]

night_filter@reddit

The basic problem is that a snapshot is stored in the same system as the live version. If you're doing a filesystem-level snapshot, then you could have a operating-system-level, filesystem-level, or hardware-level problem that causes you to lose both the live version and the snapshot. The idea of a backup is to get a separate physical copy, totally off of that system, and offsite. Now I've heard arguments that if you have all of your data replicating offsite and snapshotted in both locations, that can be considered a backup. I've also heard other people say that they're not satisfied with that.

Reply

[-]

GhostDan@reddit

Snapshots are great for what they are. Quick moments in time on your VM, easy ways to fix a 'oops'. Backups should have two major factors: -A FULL backup of everything, at a set time, preferably every day/week/month/whatever -This was important in the past, but in the current realm of encryption attacks it's so much more important, IT SHOULD NEVER be an easy to access piece of data, such as a snapshot being stored on the servers. A lot of why Tape is still popular (along with cost) is because you write it and it's almost impossible for a hacker to come along and encrypt it. If you are backing up to drives, you should do your best to makes sure the ONLY thing that has access to those drives is the backup daemon, and even then I'd highly recommend regular backups to tape (weekly/monthly) I've run into shops that just took snapshots, usually because they called me in to fix things after the fact ;)

Reply

[-]

hairypussblaster@reddit

I always do a snapshot before making a change, like you said it's so you can just flip back. Just get rid of the snapshot when you're done. A snapshot is not a backup, every change that is made to the system is recorded since you took the snapshot so it can roll back, this will impact performance, that's why it's important to not keep snapshots around any longer than needed. Backups are for when your building explodes.

Reply

[-]

wildfyre010@reddit

Depends what you mean by snapshot. If you’re talking a VM snapshot in vCenter, for example, this is a copy-on-write snapshot which means it’s dependent on the same disk subsystem and infrastructure. It won’t help you if the entire site fails. But also, copy on write snapshots introduce performance issues over time. They are fast because no data is saved at the moment the snapshot is taken; from that point forward, new or modified disk blocks are written to a second area of the disk, but in practice that means the snapshot grows over time as data changes and reconciling it with the original disk becomes more expensive. As a temporary point in time recovery mechanism, snapshots are great and the use case you describe is sensible. But don’t mistake them for traditional offsite backups.

Reply

[-]

bigfoot_76@reddit

Snapshots don't take into account things in memory not yet written nor do they provide adequate protection for things like databases. Snapshots are "crash consistent' as in the data you would only get data that is written to disk in an appropriate manner prior to pulling the plug on a server. Backups (properly configured with guest operations) are "application consistent" in a similar sense that you shut down the VM and then did a backup (there's no data to be written, no SQL database to screw up, etc.). Workaround: shut down the VM completely, snapshot, the power it on and complete the update. This gives you a very similar level of protection as waiting for the backup with a few caveats. Things could still go south without a proper backup if your hypervisor takes a dump during an upgrade. It'll be much faster to restore from backups than trying to move virtual disks around and hope that the machine configs are still functional.

Reply

[-]

skidleydee@reddit

It seems like most people are talking about VMware so I'm just going to go with that. Keep in mind snapshot isn't a technical term and can function differently in different stacks. 1. If you look at what a backup is defined as a snapshot technically qualifies. The issue is with how it's implemented vs a file based or VM based back up. Snapshots are like taking a picture of the current state of the VM and then building on top of that. The snapshot creates a delta where all data from the time of the snapshot is taken to the time it's removed. When removed the consolidation process writes the delta back to the actual VMDK file. So if you lose the VM but have a snapshot you're out of luck. 2. Performance hits are common with both because VM level backups will often use snapshots to capture the VM in the state it's in and removing the snapshot when it's done. If the snapshot fails to remove and the Delta VMDK continues to grow. The machine will have to split its reads across multiple VMDK files and even on an SSD when it gets to be 50g or so that will be noticeable. 3. You will fill up your data store where the snapshot has been taken. As all data is written to a separate VMDK file when the snapshot is taken, you are not using any of the existing VM disks. Filling up a data store has its own dangers and I encourage you to look that up. 4. Backups are often done in chains, this is possible to snapshots as well but with each snapshot taken and added to the chain, you will increase the likelihood of all the other issues on this list. 5. When training snapshots, each additional snapshot is dependent on the one before as each snapshot is just creating an additional VMDK file. While incremental backups do this as well, there is less danger because the consolidation process is happening to data that is not being accessed otherwise. 6. I would encourage you to read veeam's white paper on application Aware processing for this one as it's too large for the details here. Basically things like database servers do better when the backup process is able to truncate the database into many small files rather than one extremely large one. 7. Backups are independent of the VM you are working with after the snapshot has been consolidated. There are a ton of other reasons why backups are better but this is some of the reasons snapshots are t backups.

Reply

[-]

ThemB0ners@reddit

In addition to what many in here are already stating, keeping snapshots around significantly reduces storage performance.

Reply

[-]

identity-ninja@reddit

If your systems use anything Active Directory, Kerberos or SQL is in the mix, snapshots simply do not work. You have to use proper backup and recovery tools.

Reply

[-]

Camera_dude@reddit

Snapshots are not permanent. They can restore up to the point the snapshot was taken, but what if your file server gets hit with a cryptolocker type malware and the most recent snapshot taken was after files were encrypted? In many cases there's only one snapshot, but an offline backup will be safe from that type of issue.

Reply

[-]

InevitableOk5017@reddit

Most good backup software uses snapshots but please don’t leave one out there for very long. From my little understanding of things when you do snapshot it creates a new let’s call it a disk and every read and write is put on this new disk. Over time it will grow and fill up your storage. When you delete the snap it commits the read and writes back to you original machine. A snapshot is not a backup, keep it clean. Hope this helps.

Reply

[-]

Tx_Drewdad@reddit

A snapshot is not a backup, but it doesn't mean that snapshots are not useful. Backups should follow the 3-2-1 rule (three copies, on two different media, and one copy offsite). These are disaster recovery for when you lose your primary storage array, or some other disaster. Snapshots don't follow the 3-2-1 rule. They're more like 1-1-0. They're just a rollback point. So it's perfectly valid to use them as rollback points, but don't consider them backups for disaster recovery purposes.

Reply

[-]

CaptainZhon@reddit

If you have never been there then you will never know. Yes backup takes longer - but it is more reliable. Snapshots can get corrupted, application(s) in the snapshot can be corrupted. Relying on a (bad) snapshot only to find out it was bad when you revert back to it can cost cost you the night or weekend - so when we say snapshot, backup, close (if possible) it is for your protection :)

Reply

[-]

iCashMon3y@reddit

If you are doing nightly backups, you should be fine doing a snapshot before you make a minor change on a machine. The point is that the snapshot shouldn't be your only failsafe, you want to ensure that you have something else in case something goes wrong.

Reply

[-]

akerro@reddit

They're not always mutually exclusive https://mastodon.social/@chromakode/110936177254839251

Reply

[-]

nabarry@reddit

1. If your backups take 8 hours find a new backup tool. 2. Snaps are for exactly the use case you described “I’m doing something in guest that may go wrong and once it’s ok I will clear it” Backups protect against that, plus hardware failure. 3-2-1 rule protects against site failure too.

Reply

[-]

CrazedTechWizard@reddit

A Snapshot can (in most cases) ONLY be restored to the same exact VM that it was taken from. I can't take a snapshot of CompA and try to apply it to CompB, even if they are otherwise identical.

Reply

[-]

questfor17@reddit

Backups are not really the question. The focus should always be on the restore. Why are you restoring? How much are you restoring? Snapshots are a fantastic way of preserving stuff against all kinds of operational mistakes. Removed the wrong file by accident? Get if from the .snapshot directory. Nobody wants to go to offsite backups for that kind of mistake. Your storage array is completely dead and never coming back? You need another physical copy. Ideally this copy is on-site, as that is much easier to access and will restore your data more quickly. Your data center just burned down? Then you need a copy off-site. Maybe. Years ago I worked with a large well-known company who operated for years without off-site backups. Why? They calculated that any disaster that destroyed their one data center would also bankrupt the company. Eventually the grew to a size where that was no longer true and they invested in offsite backups. All of which is to say the gold standard, in my book uses: * Local snapshots for recovery from operational errors * A second copy on site to recover from major hardware failures. Ideally this copy should have some form of immutability to help defend against ransomware attacks. * A third copy off site to recover from major disasters. This copy also needs immutability. Not every application at every company needs all of the above, but skipping any part of this should be done consciously.

Reply

[-]

KiNgPiN8T3@reddit

Snapshots are good. But I’m VMware if you don’t get rid of them you’re opening yourself up to pain… I’ve lost count of the amount of times I’ve had some sort of issue with a server and it’s due to a snapshot that’s been left open eating up the disk space. Which then takes ages to delete.

Reply

[-]

sc302@reddit

If performing a change I will have last nights backup to fall to and take a snap shot prior to starting. I always have plan an and plan b and sometimes a plan c.

Reply

[-]

thegreatcerebral@reddit

Didn't see it but also, at least last time I was using it, software like Veeam will not work properly if you have a snapshot there because of what others have said. Veeam will use a snapshot to grab all kinds of stuff and then it will delete the snapshot when done. It will give you all kinds of errors if it cannot do that.

Reply

[-]

theborgman1977@reddit

We use Datto to bring a VM about 15 minutes from the hardware dieing. So it is not the same time as a normal restore. You should have 2 backups. 1. local device and 2 to the cloud. Datto is not the only one to offer spinning up a VM. What people do not get is a backup is good, but a backup with business continuity levels is better. This should be the minimum. I recommend not attaching them to your domain along with not attaching the hosts to the domain. This gives you layers or protection. Also, do not use the same passwords for hosts or backup. RAID - Business Continuity number 1. Backup - with replication to cloud or vice versa Ability to spin up a vm(Cloud or Local) Business Continuity number 2

Reply

[-]

mammaryglands@reddit

Snapshots are fine for what you described, but they aren't meant for long term. Aka backup

Reply

[-]

BuffaloRedshark@reddit

being able to restore only certain files if someone accidentally deletes one important file off a file server you don't want to restore a snap shot of everything, just that file

Reply

[-]

Professional-Bug2305@reddit

Why is your backup taking 8 hours? They shouldn't take anywhere near thst long.

Reply

[-]

OtiseMaleModel@reddit (OP)

I will admit I didn't check the last back up time before posting. Just an example my boss used. But it is cloud back up, each vm does take a bit of a while

Reply

[-]

Professional-Bug2305@reddit

Ah yeah makes sense then. But I'd seriously look into a solution thst backs up to an appliance on prem and then replicates to the cloud think about how many 8 hour restores it would take in a DR incident. You could be down for days on end.

Reply

[-]

gurugti@reddit

Snapshot has a much higher rate of getting you screwed. Backups are usually much more reliable.

Reply

[-]

westyx@reddit

* Every snapshot in a chain makes IO more difficult for the esxi host to action - it needs to keep track of which blocks are dirty at which stage * Most backups are incrementals and are stored on slower storage, either slower flash, spinning disk, or offsite on a private or public cloud. Every byte of every snapshot is on your primary disk * If you want restore points for a vm over a hundred day span using ba then that's one full backup and (probably) 99 incrementals. If you use snapshots then you blew past the support 32 levels of snapshots 32 days in. * If you're on slow storage and you're hammering it then you can get broken snapshot chains, causing a complete outage of your virtual machine until you dive in and fix things. A broken backup chain means .. your next backup will be a full and take a bit longer. * People forget about snapshots and they stay around. I've seen snapshots that are multiple years old - and they generally grow bigger the longer they're there. A backup does not grow in size. vVols and ESA make things a bit different in terms of snapshot capabilities, but it's still not a great idea.

Reply

[-]

FiredFox@reddit

To add to the other replies here: "You do not have a backup of your data until you have recovered data from your backups."

Reply

[-]

throw0101a@reddit

> **If** you are just doing something **like a minor update** on a tool hosted on a server in your hyper-v environment do you really need to wait 8 + hours for a back up, run your update and then if you do meet a disaster have to wait all that same time to restore? If that is what you're doing. But what happens if you're hit with ransomware: they will encrypt all your online files, and as part of that probably delete any snapshots (Windows Shadow Copy / VSS) you have online as well. The only thing that would be safe is copies of the data that you have *offline*.

Reply

[-]

valdecircarvalho@reddit

Snapshot is not backup.

Reply

[-]

Obvious_Mode_5382@reddit

This

Reply

[-]

jimmythewolf83@reddit

Snapshots create a temp file. The longer you have a snapshot the bigger that file gets and the more of a performance impact it has on your system. I once had to deal with VM sql server that had been snapshotted for over two years. When I went to delete the snaps, the server screeched to a halt and it took a full four days for the it to reconcile. Unfortunately it was a production database for the companies CRM and we had just taken over the company and were still in the process of deploying the DR plan. Snapshots just are meant for long term backups, but they are nice when you are making a change and want roll back quickly if it goes south, as long as you delete the snapshots within a reasonable period of time. I usually keep them for a week after.

Reply

[-]

malikto44@reddit

I knocked out a production VM for 8-12 hours due to the exact same thing. I have seen some really large snapshots, especially a high turnover database with encryption, where each snapshot is almost 100% the size of the current in-use process. I do a snapshot, do the upgrades, then a week later, remove it. If I need to keep the snapshot permanently, I will just clone the snapshot to a new VM and vMotion that clone to the archival tier.

Reply

[-]

ConstantSpeech6038@reddit

It seems like an overkill to do full backup before minor update, but there should be regular at least incremental backup in place. And that backup should be tested periodically. I would use snapshot anyway, but THE BACKUP must exist separately. What if the snapshot gets somehow corrupted. As they say in SEALs, two is one and one is none. That is healthy paranoia when stakes are high.

Reply

[-]

Burgergold@reddit

8 hours for a backup?

Reply

[-]

shultzmr@reddit

A few things. A snapshot is fantastic at taking a point in time picture of a machine for short term use (i.e before issuing updates). But take these following scenarios where a snapshot likely isn’t appropriate. As a user i have deleted my spreadsheet from a file share. Are you going to roll the VM back to a snapshot and undo everyones work? My VM has some corrupted files from a couple of days ago that have just surfaced. How do you go back far enough? Snapshots are stored alongside virtual disk files. Your storage fails. Now what? Snapshots work by creating additional drive files for changes to get applied to. This causes a performance hit, and it gets worse the longer the snapshot runs. Add multiple snapshots on top of each other and you can TANK performance. Anyone who has found a snapshot left for a few months then has tried to merge it can attest to this (it can even stun a VM and take it offline) Those a few examples of where snapshots don’t cover you. That isn’t to say snapshots are bad. Short term use (pre-patch) is a nice example. Test labs or sandbox - perfect. Snapshots are also vital for products like Veeam.

Reply

[-]

Agres_@reddit

Life is too short for a full backup. A snapshot is fine 90% of the time. The other 10% who gives a shit, just rebuild the shithouse server again. IT is a waste of time.

Reply

[-]

DistinctMedicine4798@reddit

I think the hesitancy with snapshots could be related to HyperV because if you forget to delete the checkpoint it can cause problems later

Reply

[-]

sobrique@reddit

The big difference is a backup is a complete copy of the data, where a snapshot just keeps track of what's changed. In a lot of cases? There's no difference. But _because_ snapshots only keep track of differences, they are fast and space efficient, but that means they grow as the number of differences do. And they _require_ the primary data source, and without that they're junk. So there's scenarios where snapshots break, where backups wouldn't. E.g. if there's a LOT of changes, you risk running out of disk space, and then your snap being unable to track everything it needed to. Or if you've a hardware problem. A snapshot on a RAID is no use when the raid is broken. A snapshot on a virtual disk is no use when the virtual disk gets corrupted somehow. Which is why 'both' is the recommendation. A backup is a lot more robust than a snapshot, in return for the higher cost of 'copy everything'. But both have their place, and are valuable as part of recovery scenarios.

Reply

[-]

603176911886936@reddit

Just throwing in a caveat (its what I do) - These two words "backup" and "snapshot" can be recycled terms when looking at cloud provider offerings - be mindful if you're taking advice from this thread what context you're applying it to. The one that comes to mind for me is AWS RDS. "Snapshot" is AWS RDS terminology for "backup". Its a full, self contained image of the block storage for the RDS instance at a point in time. It is either restord fully or not at all, and will restore all databases for that instance. From the same RDS instance, you can *choose* to create "backups", ie conventional MS SQL .bak files, that you can throw up to S3. These are database-level backups as people typically know them, and can be configured with MS SQL transaction logs too.

Reply

[-]

blnk-182@reddit

Obligatory RTFM. Yeah you can use snapshots for updates like you’re describing but if the disk gets corrupted you’re SOL. You don’t want to retain snapshots for too long for several reasons. And also you can’t restore single files from a snapshot.

Reply

[-]

sobrique@reddit

> And also you can’t restore single files from a snapshot. Depends an awful lot what sort of snapshot tech you use.

Reply

[-]

adept2051@reddit

Most people when they say backup, mean suitable precautions for the activity whether backup, snapshot or ability to redeploy.

Reply

[-]

Alsmk2@reddit

Why do you have to choose? Do both if it's viable.

Reply

[-]

Spacesider@reddit

Snapshots have a short retention period and are for when you need to make minor changes. That way you can pretty much instantly roll back any changes if needed should something break. Backups on the other hand have a long retention period and standard practise is to store it on another medium. It's not there for when a minor change fails and you need to restore the service, but for when the entire machine is in some way very broken. Think of the scenario if your disk dies or the entire machine is gone due to a power surge. Sure these things can be prevented with a RAID and a UPS, but lets ignore that for a moment. The problem is, this disk/machine had both the OS ***and*** the snapshot. So you cannot rollback, and you are now in a situation where you have potentially lost a lot of data and are looking at a long time offline. But if you had a backup, you would only be down for a little while and the data loss is not going to be so severe either.

Reply

[-]

FactorOk900@reddit

Depending on the software you use to take the backups, the time may vary sure. But having a timely full backup and use of incremental backups cut the time of backing up servers severely. It takes for a basic dc and such a few minutes to take a incremental backup if not loaded with loads of additional files. And restore also takes from less than 1 to same few minutes (depending ofc the method/state that you restore the backed up server online) basicly. So I'd say you should maybe check what software you'r using for the backups and enhance that experience. And as pointed out, snapshots are also useful, on certain things but personally I'd prefer a backup every time with proper tools cause it really does not take much time anyway, with proper tools.

Reply

[-]

One_Leadership_3700@reddit

As far as I understand it: \- Snapshot done while machine is running also contains memory and is not consistent for all database-related tasks. Active Directory is not that bothered anymore but a snapshot shall be done when powered OFF otherwise... I am not so sure when you revert to the snapshot how the system handles the "time-loop", depending on the infrastructure So I always go for backup But maybe that is old-fashioned now?

Reply

[-]

serverhorror@reddit

No one cares what you call it. Checkpoint, snapshot, backup, common sense, cover-your-ass technique, ... have a way to get back to a known good state. Do it in a reliable and tested way and know how to actually execute the rollback without data loss.

Reply

[-]

360SubSeven@reddit

Let me ask a question since its not my job i wouldnt know. Would you say for a production SQL DB it is necessary to backup the DB individually or would a backup of the complete environment via veeam or something suffice?

Reply

[-]

mobani@reddit

There is absolutely nothing wrong with using a snapshot for performing server maintenance or other updates, as long as your backup strategy have been implemented, and the risks have been evaluated. You should be running daily backups anyway.

Reply

[-]

dasookwat@reddit

Did you ever try snapshotting a machine with an active database? When you need to restore a specific file, backup is much more practical than a snapshot. When your certificates expire, your restored snapshot can be a pain to get running again. A backup is about backing up important data. A snapshot holds the OS and what's on it. Both have their uses and place.

Reply

[-]

gbsscc@reddit

a snapshot is the current active data. It hast to be written back from this point on. Starting with the snapshot you are writing all data to the snapshot not to yout standard vm disk

Reply

[-]

icebalm@reddit

You want to run a chain of snapshots on the hardware you're running on? And what happens if that hardware dies? If your building burns down? What happens when you need only 1 file from a couple snapshots ago? What happens when you want to restore one of those snapshots onto different hardware for testing? Snapshots are for doing something quick. Going to install a new CU on an Exchange server? Yeah, make a snapshot first so that when it fails you can rollback, but they're no substitute for backups.

Reply

[-]

OtiseMaleModel@reddit (OP)

Damn, I reread my post and I see I made out like I use snapshots only as a back up solution. We have a full back up solution but I basically have noticed every mention of a snapshot being responded at with "use a back up not a snapshot" and I'm just trying to do something small and felt like back up was overkill as.

Reply

[-]

BK_Rich@reddit

Especially on VMware, leaving a snapshot around for too long will degrade performance as well.

Reply

[-]

ApprehensiveBar7515@reddit

Ransomware, esxi brainfarts, you name it

Reply

[-]

greywolfau@reddit

Not related to your question, but never be afraid to ask a question. If the worst thing that happens is people call you a dumb arse for asking a question than it's still a good day

Reply

[-]

sssRealm@reddit

Snapshots are great and I often used them when do upgrades or other risky modifications. Though they should never replace regularly scheduled backups.

Reply

[-]

fadingcross@reddit

8 hour for a backup? Why does a restore take 8 hours? That's the real issue here.

Reply

[-]

people_t@reddit

They are 2 different things for 2 different requirements. A snapshot is good for a quick restore when something happens to the files/file system. A backup is for when you have a hardware failure or security issue.

Reply

[-]

MangoPanties@reddit

Backups are your daily task that the company can always fall back on, if the worst was to happen. Snapshots are for rolling back minor changes / system upgrades. That's how I've always used them and it's never failed me yet.

Reply

[-]

canadadryistheshit@reddit

What I have yet to see here is some reasons where you would NOT perform a snapshot and have to perform a backup. Some Virtual Machine appliances have very clear instructions to NEVER take a snapshot. Example here would be AKiPS, a network monitoring tool that uses a ZFS file system. My AKiPS instance monitors a lot of end point devices. If I lock up the VM with a snapshot the whole thing can go belly up. Tools like Aria Ops (vrops) and Extrahop also encourage users to not take a snapshot of the underlying VMs. So there are a two reasons: 1.) Performance hit. If you have an application running that is sensitive to even the slightest latency when it comes to disk read/writes, there will be complaints from users when snapshots are taken. 2.) VMs that host databases (single node deployments) its generally not a good idea either especially if its consuming a lot of RAM if you snapshot with memory. VMs will also be stunned for a short period of time just before the snapshot action completes, and also Ive seen a minor performance hit when the snapshot is deleted (consolidation of disks) All in all, it depends what application is running on the server. Most can handle it. However, like the rest of the folks are saying here, not only do you need a Plan B, it doesnt hurt to have that Plan C for when you go, "oh fuck this VM is screwed even after reverting to snapshot."

Reply

[-]

michaelpaoli@reddit

Context matters. Backups are much better for some things, backups for others. Let's compare two scenarios: * very busy active system, lots of activity going on, generally don't want to loose folks' work as they're chugging along, various database and filesystem changes, etc. You're going to make one change one configuration file for one service - it impacts nothing else ... but if the config doesn't work you want to revert it. You back up the one file, you make the change you test it - doesn't work or doesn't work as expected/desired, you change that one file back, and reload or restart that service, and it's back as it was before, and nothing else is interrupted or disturbed. But let's say instead you'd done a snapshot. Now you want to change just that one file back ... but not everything else ... and you did a whole entire snapshot for that ... depending how the snapshot was done, getting just that one file back as it was before, while not rolling back other changes, may not be so simple ... did you even note what file where, or did you just think/say oh, no problem, do a snapshot? Yeah, sometimes backup is the much better solution. * Somewhat different scenario. Say you're about to do a massive complex upgrade on a system. It's important production, but not being used at the moment, because maintenance window has been arranged, so services not in use on the host at this time. You do a full backup, you also do a snapshot. You work on the upgrade. And maybe an hour or so into it, things go seriously sideways. The time on the maintenance windows is ticking away, need to have it back up in production fairly soon - doesn't look like can just fix it and roll forward in time, so ... restore from backup or ... restore snapshot? Probably helluva lot easier and faster to just drop back to the snapshot that was done before one started the upgrade attempt. So, yeah, ... context matters. It's not a "one size fits all". There are other differences and pros and cons, but that's just at least bit of example scenario.

Reply

[-]

Bijorak@reddit

8 hour for a backup? i do full backups weekly and rollups daily and none ever take 8 hours. and i can also restore a vm within minutes using the restore. but if im testing a config change i will do a snapshot and rollback if it doesnt work. they are for very short term use cases.

Reply

[-]

EVASIVEroot@reddit

A vm snapshot is just a quick and effective recovery method for a short time span. Also, you cannot really snapshot a physical server in the same easy way. Let's say you want to test a windows hotfix or something on a vm and you sort of care if it works after or maybe you are testing the automation of deploying this hotfix and you want to unload the changes with out manually doing each one. Snapshot it. It works super well if you want to roll back to the snapshot point. However, it takes up a lot of storage as there is a mechanism for keeping changes and original snapshot state. It does not sound like a lot of storage but think of a production sql server and all the transactions that could take place. Since snapshots do not work long term and can be compromised in ransomware easily, they are not relied on for full on backups for DR etc. They are a great option to put into the change request for "roll back" option too. In a large environment, snapshots even become troublesome, especially if storage is a constraint. If anyone deals with this, I recommend learning PowerCLI and automating out snapshot reports or even auto deleting snapshots within a certain time period; you could tag with change requests if you needed to keep some long term for whatever reason.

Reply

[-]

Proper_Front_1435@reddit

Snapshots come with their own set of risks/complications/etc at the hypervisor layer that can fuck you is the short answer. Example, you make snapshot, even though ESXI did it, their isn't enough swap space in the data store to rollback or something. I use snapshots for OS upgrades, critical upgrades etc. Its nice to roll back quickly. But I also make a full backup, cause VMware has fucked up snapshots enough times for me to know better. My suggestion is start planning out your activities better, no reason it should be A or B. You know you doing work tonight at 2am? Schedule a backup 8 hours prior at EOD. Two plans. Always. Plan and plan for the plan to fail, whenever possible.

Reply

[-]

StConvolute@reddit

You could use a product like Veeam that basically uses snapshots to backup the servers. Their replication for DR is also just snapshot copies to another remote host. But the key difference, with Veeam and their _snapshot_ type backups, is that it's always going to another hosts or device. It's never stores with the production setup. So that's why I'd say backup over just a snapshot.

Reply

[-]

pertexted@reddit

Backup = long-term Snapshot = short-term backups are resilient to data corruption, presumably being not stored on the thing that is about to be tested on or broken. that doesn't mean you can't or shouldn't snapshot. it just means if something goes wrong-wrong, rather than inconvenient-wrong, you'll just be using your backup anyways.

Reply

[-]

have-you-reddit_@reddit

Auditing and ISO, especially with government contracts. That and a lot of snapshot entries for every directory plus added data until expired deletion to make way for new snapshots.

Reply