Pacemaker/DRBD: Auto-failback kills active DRBD Sync Primary to Secondary. How to prevent this?

[-]

vdvelde_t@reddit

Let drdb decide who is primary. I have implemented the below some time ago and this is still running https://learn.microsoft.com/en-us/azure/sap/workloads/high-availability-guide-suse-nfs

Reply

[-]

That guide includes implementing resource stickyness - it won't restore 'normality' in resource location after a failure. You'll end up with both shares resources on one node, instead of one on each, after a failure or system restart

Reply

[-]

srekkas@reddit

Maybe try linstor

Reply

[-]

DerBootsMann@reddit

linstor is a commercial orchestration layer for drbd . if the op has issues with his drbd setup , linstor itself won’t be of any help

Reply

[-]

srekkas@reddit

I use it on proxmox in my homelab. It doesnt require pacemaker

Reply

[-]

aieronpeters@reddit

It really helps to have 3 nodes if you can with pacemaker, even if only to are allowed to take resourced (location constraints). That way, a node dropping means the remaining knows it's not itself that's the issue, because it can still 'see/talk to' the remaining quorum node

Reply

[-]

posixUncompliant@reddit

This is fantastic advice.

Reply

[-]

Fighter_M@reddit

Yes, this is expected. See, Pacemaker has no awareness of active I/O, user sessions, or open files… If your constraints say ‘prefer Node 1’, and stickiness is set to ‘0’, it will immediately pull the resource back as soon as Node 1 is up again! DRBD also won’t delay a promotion just because someone is writing on the Secondary node, Pacemaker will simply demote/stop resources on Node 2, killing whatever is running. What should you do? Increase stickiness, which is ‘default-resource-stickiness=100’ or or even higher, so resources stay on Node 2 after failover. Use a ban constraint that is lifted only after you manually clear it or after DRBD is fully resynced. Don’t rely on a ‘prefer=50’ constraint for master selection and use DRBD Master/Slave rules or manual promotion logic. And absolutely enable STONITH, otherwise you will hit split-brain guaranteed! I mean with DRBD you’ll get either way, sooner or later, but the way you configured everything now it’s just a disaster waiting to happen. Good news is, there’s no filesystem corruption as long as DRBD is clean, just interrupted writes, which is an obvious data loss, but no corruption. But if a node flaps without STONITH, then yes, corruption becomes a real risk.

Reply

[-]

posixUncompliant@reddit

The only thing I'd add to this is that I do everything in my power to avoid an auto failback. My underlying assumption is that whatever fault happened to bring down the master will require an intervention to repair, but I can't depend on the node staying down to prevent failback to the compromised system. In practice, I consider a Pacemaker/DRDB set up one time use. Once it flips, a fair amount of intervention is required to make the set up reliable to handle the next fault.

Reply

[-]

LinuxLeafFan@reddit

Can confirm this is correct. In the scenario described above, you have your resource stickiness set to 0. In our env we normally set it to 1000.

Reply

[-]

aioeu@reddit

The concept of a "session" is completely outside of Pacemaker and DRBD. Pacemaker obviously has no idea whatb the resources actually *do*; it's just responsible for starting, stopping, promoting and demoting them. And DRBD doesn't intrinsically know that you've got a mounted filesystem on it — it's a block device, so all it sees are block reads and writes. It's been a few years since I last touched Pacemaker, but from memory I think what you'll want to look at is "resource stickiness". This adjusts the score for a resource so it is more preferable to keep it active on the mode that it is currently on than move it to a different node. You'll want to do this on the resource for the filesystem mount (or on the service that's using that filesystem, if that service is also being managed by Pacemaker). It doesn't make sense to adjust the stickiness of the DRBD resource since it remains active on both nodes all three time — i.e. it isn't being "migrated", just "promoted" and "demoted" as necessary.

Reply

Pacemaker/DRBD: Auto-failback kills active DRBD Sync Primary to Secondary. How to prevent this?

Reply to Post

11 Comments

vdvelde_t@reddit

aieronpeters@reddit

srekkas@reddit

DerBootsMann@reddit

srekkas@reddit

aieronpeters@reddit

posixUncompliant@reddit

Fighter_M@reddit

posixUncompliant@reddit

LinuxLeafFan@reddit

aioeu@reddit