Distributed File System

Posted by abqsysadmin@reddit | sysadmin | View on Reddit | 29 comments

Hey everyone, looking for some advice here I’m looking for a possible solution to a problem I have Currently we have a nfs server that serves shared libraries, stores and serves application related files(images, etc.), this all works fine except this is a single point of failure I have been searching for a POSIX compliant(single namespace) distributed storage solution, that can be accessed via nfs, and has non snapshot based geo replication, and preferably something that has synchronous geo replication although it’s not a hard stop on that. I’ve looked primarily at ceph for obvious reason, biggest downside is cephfs to my knowledge only supports snapshot based replication, I have also looked at ceph-rgw that’s exposed through nfs using ganesha nfs, I had some issues with the latter Any recommendations would be amazing, thank you.

Reply to Post

29 Comments

[-]

raip@reddit

Sounds like you're looking for GlusterFS. I wouldn't recommend synchronous relocation across WAN though - the performance hit would be pretty intense. Their async relocation is not snapshot based though.

[-]

DerBootsMann@reddit

> Sounds like you're looking for GlusterFS discontinued ..

[-]

raip@reddit

TIL

[-]

DerBootsMann@reddit

yeah , another good project ibm / redhat axed in favor of a questionable internal competitor … glusterfs vs ceph , ovirt vs openshift and so on

[-]

DasPelzi@reddit

Have a look at GlusterFS and Hadoop. Depending on the available Hardware and Network speed it might also be a possible to just mirror the device containing the nfs share with DRBD (block device replication over the network). Basically a small HA cluster (2 computers) with seamless failover to a secondary server if the primary one fails. (pacemaker/corosync)

[-]

darkvash@reddit

I’d stick with Ceph, to be honest. DRBD’s proper limited, you’re stuck with two nodes, active-passive setup, and it won’t even read from the passive one. If you want both handling I/O, you’ve got to slap on GFS2 or some other clustered file system, and that’s a faff to get running right. Doesn’t scale past two nodes either, and stability’s never been its strong suit. Ceph, on the other hand, got solid backing from IBM, rock-steady and scales beautifully, every node you chuck in gives you more perf. It's a pity, but IBM already binned GlusterFS a while back. Can’t comment on Hadoop, never touched it.

[-]

abqsysadmin@reddit (OP)

We have very fast intra data center speeds, and very fast hardware, across data center we have about 34ms rtt

[-]

Famous_Mushroom7585@reddit

glusterfs might be worth a shot. does geo rep better than cephfs and works with NFS. not perfect but less of a headache

[-]

the4amfriend@reddit

I think you mean GFS2, it's still supported by RedHat if I'm not wrong?

[-]

NISMO1968@reddit

> **glusterfs might be worth a shot**. Nope, that ship sailed ages ago. https://access.redhat.com/support/policy/updates/rhs Red Hat Gluster Storage 3.x End of Life : December 31, 2024 No commits, no updates, it's been radio silent for like two years.

[-]

ZY6K9fw4tJ5fNvKx@reddit

I'm running moosefs and am quite happy with it. If you think ceph is too complex i would recommend moosefs. Had lots of failures, none of them fatal. You could use fusefs and share it with your favorite nfs server. Or use nfsv4 replication directly.

[-]

darkvash@reddit

Yeah, MooseFS is pretty cool. JuiceFS is also worth a closer look if something more cloud-native is required. Ceph is still hard to beat though.

[-]

abqsysadmin@reddit (OP)

What do you mean by nfsv4 replication? And will definitely look into moosefs

[-]

ZY6K9fw4tJ5fNvKx@reddit

You could use the replicas argument but it looks like you still have to copy the data yourself between nodes somehow (rsync might be accaptable, depending on your usecase).

[-]

Fighter_M@reddit

If you don't need to mount the namespace from multiple places at once, just run ZFS on the source and use ZFS snapshot send/recv on the target. Dead simple, tons of folks do it daily. Tinker a bit further and you can even offload cold data using nested ZFS with dedup turned on, so only dehydrated content hits the wire, but that’s more for the hardcore crew with serious guts with painfully slow WAN links.

[-]

rejectionhotlin3@reddit

ZFS is love, ZFS is life.

[-]

rejectionhotlin3@reddit

Here's the long and skinny of it - Really, your applications should be HA native. The hypervisor and storage shouldn't be the source of this HA aside from shared storage either via NFS or whatever your preference. I'd personally prefer two storages boxes and put one of the HA VM's on each. But again, that's just my preference.

[-]

274Below@reddit

ceph inherently keeps multiple copies of data across different servers. The default "size" is three, which means that it's going to keep every block of data in triplicate, across unique servers. cephfs is an absolutely perfect fit for you.

[-]

abqsysadmin@reddit (OP)

Ceph does work perfect intra data center, but unfortunately not many options for geo replication for nfs, from my understanding at least.

[-]

274Below@reddit

You can tell it "this server is in this datacenter, this other one is over there, keep the data in both places." That's a basic function of ceph. https://docs.ceph.com/en/reef/rados/operations/stretch-mode/

[-]

abqsysadmin@reddit (OP)

Yeah I had looked into this, I thought there were strict rtt latency requirements, something around sub 10ms, I’m sitting at around 30ms rtt

[-]

Fighter_M@reddit

> cephfs is an absolutely perfect fit for you. Ever tried doing a 3-way replica in Ceph with one copy sitting miles away over a dark fiber link?

[-]

LostSailor25@reddit

Microsoft Windows DFS? You could fairly easily setup a two ( or more) server solution.

[-]

NISMO1968@reddit

>**Microsoft Windows DFS**? You could fairly easily setup a two ( or more) server solution. DFS-R, you mean? Honestly, it causes more problems than it solves. It’s painfully slow, only replicates written data, and because files need to be closed before replication starts, virtually nothing stays in cache. No transparent failover either, and it can’t handle open files. There used to be a paid workaround, PeerLock from Peer Software, but that’s long gone. Microsoft did plan to fix some of this with better failover and open file handling, but the whole team behind the major DFS update got reassigned to the Storage Spaces Direct project back in 2012, and that next-gen DFS-R never showed up. Basically, it’s been in maintenance mode for 15 years already.

[-]

raip@reddit

I've got no clue if it's any better but historically sharing out DFS Shares, which are always CIFS/SMB, over NFS causes all sorts of oplocking headaches and issues.

[-]

abqsysadmin@reddit (OP)

I’ve heard this same issue with DFS as well

[-]

chaos_theo@reddit

Amazing recommendation searched for ... gpfs and a Netapp can do for you

[-]

Love-Tech-1988@reddit

i\`d say hadoop

[-]

abqsysadmin@reddit (OP)

Will take a look!