BCacheFS is being disabled in the openSUSE kernels 6.17+
Posted by ehempel@reddit | linux | View on Reddit | 73 comments
Posted by ehempel@reddit | linux | View on Reddit | 73 comments
TheASHTening@reddit
0riginal-Syn@reddit
So likely never
Optimal-Procedure885@reddit
Well, Kent is an engineer so thinks he doesn’t have to work well with others. He’s single-handedly ensured bcachefs is stillborn…how foolish.
TRKlausss@reddit
I’m a safety critical engineer, and we throw those people into menial tasks the moment we see them. Bypassing the rules is a recipe for failures like the MCAS at Boeing.
koverstreet@reddit
What do you work on?
MCAS was a disaster brought on by business requirements over riding good engineering sense - specifically, Boeing cheaped out because they didn't want to have to certify the 737 MAX as a new type design.
TRKlausss@reddit
I worden on cars, airplanes and now industrial.
Most of what happened with MCAS was business, yes. But safety engineering played a role there on not blowing the whistle. If you don’t perform your validation correctly, that’s on systems engineering, not on any business decision. If they cut expenditure on validation from business, you tell them “whoops we are late because you don’t give us money”. I play this game daily.
koverstreet@reddit
I have a hard time recalling a major engineering disaster where if you dug down there wasn't someone who blew the whistle and was ignored. Remember Challenger?
I can't recall any specifics on MCAS (I saw some big case studies, but it's been awhile) but I'd be shocked if there weren't engineers saying "this is nuts" and managers saying "this has already been decided, make it happen".
Anyways, the journal_rewind fiasco that sparked all of this had similar elements. I've invested massively in QA for bcachefs - automated testing, test suite, building up a community so that there's people who build my git tree regularly and I have a decent idea of a number of their workloads, so I can actively ping people if there's something big coming down the pipeline that needs extra attention.
But in the kernel we've got management saying they know better and can decide whether a patch is for a critical fix and should or shouldn't go out without looking at any of that, without having worked on a modern filesystem, without the communication with the userbase - with nothing more than glancing at the pull request and patch and god knows how much they even look at that.
The journal_rewind patch needed to go out; there were no regression concerns (journal replay in bcachefs is drastically simpler than ext4, it's a very solidly tested codepath, the change was algorithmically simple and all the tests passed) - and it was for a critical issue that had bit users in the wild (you could lose your entire filesystem).
That's just nutty.
TRKlausss@reddit
The problem is that you didn’t follow due process. Yes it was an improvement. Yes it would have helped users. Yes it passed all your tests. No, you didn’t merge/got the patch ready when you should have and no, it wasn’t a bug fix.
Asking them for an exception is the problem. You want your own development pace? You can do it, no problem. But not in the kernel. Linus and others explained it to you better and longer than I ever would be able to. And that’s why he marked it as externally maintained…
Now you can do your own pace, Linus can maintain his stability. I only see winners in this situation. And if users are “losing”, my guy, your brought it to the table…
mdedetrich@reddit
This might be a shock to you, but that due processes gets bypassed and overridden all the time in Linux kernel development. Hell people have posted patches that add completely new drivers in the middle of the -rc cycle.
We need to stop spreading this bullshit that everyone else follows these processes (which in reality are more like guidelines) 100% of the time and Kent is the only one that broke them.
klyith@reddit
An exception made one time does not require an exception to be made every time. If an exception was made for devs who had a long history of being good kernel citizens who linus found pleasant to work with, and an exception was not made for someone with the exact opposite history who'd already been warned about it, is that hard to understand?
And ken didn't get thrown out after putting in a late feature, he got thrown out after exhaustively arguing about it.
That's the history you should be looking at. Not the number of times that exceptions got make, but the number of times that a late add was rejected and the dev had the good sense to say "ok". (And how few of them alienated everyone else on the kernel team by being relentlessly negative about others' work.)
mdedetrich@reddit
For starters, calling these cases exceptions as they happen all of the time, and by all the time I mean in pretty much every kernel release. In reality these are much closer to rough guidelines, not rules and everyone breaks them all the time. Heck some time recently back Linus broke the build in release staging for everyone else causing everyone else to fail build the kernel which is normally a massive no no
klyith@reddit
Exactly my point. The months of pedantic arguing have more to do with why bcachefs is not in the kernel as anything else.
mdedetrich@reddit
And that is a pyrrhic victory for Linux, so job well done.
TRKlausss@reddit
Nah, it’s a win. Objective is to not lose time of stretched-out maintainers. If someone is making them lose big time, off the kernel with them. It’s a huge victory for maintainers.
TRKlausss@reddit
It ain’t a shock to me, it happens all the time. On DO-178C, there is also a section for “what happens when the process is not followed”.
Long story short: you have to do a ton of paperwork. Why? You want to document every single step to know why something happened like it did.
We define our coding standards, we do code coverage. You can deviate and will in some cases, but you have to provide strong reasoning with considered alternatives as to why this time to you have to deviate. And there shall be a follow up later on to either change the processes or improve documentation to gather exactly under which circumstances the process doesn’t apply.
Which is what happened here in the end. If I recall correctly, the patch got applied at the end. Linus just went and improved the process: those modules that can’t play along with kernel’s processes please get out. It’s open-source: if you don’t want something the way it is, DIY.
mdedetrich@reddit
Right, which is a roundabout way of saying that the issues have nothing to do with following/not following a process so why are you bringing it up?
Clearly the issue is not with Kent not following the rules, as plenty of other people have done the same and it’s par on course for Linux dev
TRKlausss@reddit
We can go around in circles all you want. Luckily it is all publicly available. I’ll however repeat one last time: yes, exceptions exist. No, exceptions don’t mean your specific case shall be also granted an exception. If an exception is not granted, re-requesting without changing anything is a dick move.
And in this case, the resolution taken afterwards is something satisfying both sides.
mdedetrich@reddit
To be honest, calling these exceptions is a real big stretch considering they happen so frequently (like multiple multiple times every single kernel release cycle) that in reality they are much closer to rough guidelines.
The problem is that you are missing the big picture, processes are meant to achieve the goal, it’s not just for beuracratic checkboxing for the sakes of it. What is clear about the latest incident is that the current processes are not serving end users because at the end of the day what was so contentious was a fix for an end user so they could mount/recover bcachefs partition.
And when this creates so much drama there is something wrong with the process
TRKlausss@reddit
As per definition, an exception is a deviation from the laid-down process. Yes, they may happen often. That is a “process smell”.
Calling them “guidelines” is making a disservice to their purpose. It is not meant to fill checkboxes. It is meant to reduce 1. Defects flowing down to releases and (in this case surely more important) 2. Reduce workload of people involved. Having to review 5 exceptions per release cycle creates a stretch on Linus that has already very limited resources. With the added lost time trying to make people understand.
I agree with you partially with “the process doesn’t serve end users”. It doesn’t directly, in that the process is not laid down to make end users happy, but to ensure stability of the system (which maintains end users happy).
I’d argue that, if this creates so much drama, there is something wrong with the people not understanding why the process is what it is. Every developer thinks they can push their stuff and process are just “guidelines”, making life of maintainers way more difficult.
Look at the size of Linux. Look at the number of maintainers. Everyone cusses, no one wants to jump in and do the deed.
mdedetrich@reddit
If an exception happens just as often as the norm, then its not an exception by definition
I don't care if calling them guidelines is a disservice, thats what reality is, end of story.
In reality, if you have a strict process/rule, there aren't any exception and if there are exceptions they are incredibly rare and often require another process in ofitself to justify the exception.
This is not how linux kernel development works, its it not how it ever worked. The linux kernel development process is just a set of people writing emails on lkml, some people have power and they decide whether a patch gets pulled in, thats it. There are no forms, they just make a decision on a case by case basis. These people with power are free to pull in a patch that completely goes against the guidelines/rules without any real exception.
Linus himself has done this, a few months ago he entirely broke the linux kernel build in the middle of an rc (which is a massive big no no, much worse than anything kent has ever done).
So im sorry to be blunt, but they really are guidelines. They have always been guideliens. The way that lkml is that its puts absolute trust and power in a select few to loosely interpret these guidelines as they wish hopefully in the best interest of Linux. If you have something else in your mind you are just deluding yourself
That is true
Yes, and thats the reason why strict processes (like you describe) don't really work with projects as large and complex as Linux, unless you want to slow everything down to a crawl.
TRKlausss@reddit
I don’t know what is your objective here. Mine is to make people see that processes matter, and why in certain engineering fields are the way they are. Linux is so relevant that they have to be stiffer than a normal open source project, and I don’t like exceptions to processes.
You? Are you complaining about the kernel change control process? I got news for you: it’s GPL2. You can grab the ball and go play with it with your own set of rules if you so wish… But complaining about how a bunch of people play ball with a set of rules, no matter how biased the referee is, is something either a hooligan or a grumpy old man would do… In this case the referee saw it, got a ball for this man, and told him to go play by his rules, since he couldn’t play with the others. Great resolution IMHO.
There is a ton of stakeholders for the Linux project. I’d argue that most of the money-players like to have steady development processes. If asking someone to attach to those, specially for something shiny and new, is too much to ask for, it is better that it’s maintained outside of this set of rules.
And also since Kent may read this: Bcachefs is great work. And I don’t doubt it is thoroughly tested, and its development is neat. It is just too fast for the kernel, which has to balance other things. I think converting it to DKMS is a great solution, and if I had any idea about how different distributions pack different things, I’d even consider helping…
mdedetrich@reddit
My main objective is to state how linux kernel development actually works in reality and to also educate people on misguided notions of what process/rules means in linux kernel land. People seem to have an impression that the Linux kernel development is highly organized, process driven with strict adherence to rules and that Kent breaking them is what the cause of the drama is.
I am saying that none of this is true.
No I am not complaining at all, I am just stating how things currently work there, thats all.
Again not complaining about the current rules/processes at all, I am just stating that the real reason behind Kent's drama has nothing to do with him "breaking a rule" but rather a lot of bad blood and social communication between the filesystem maintainers and Linus. Kent didn't break any rule anymoreso than anyone else has.
Sure and there is an argument that with all of the money going into LKML that it is pushing the development to be more process driven. But you know who is the biggest obstacle and frequent breaker of rules/processes? Its Linus himself, much moreso then Kent. In fact this whole model of process driven development is in Linus's eyes the complete antithesis of how good software development works and he hates this way of working to the tooth. Linus has outright said that good software project management is not about process (because it turns into a bureaucratic middle manager micromanaging hell) but rather about trust. In his eyes, its all about the people you trust and the power you give them.
Thats a valid take, but I think the best solution that would have avoided all of this drama is simple. Just make it that if a filesystem is marked as experimental, it doesn't have to go through the same guidelines as long as patches are self contained to the filesystem and don't break the linux build. experimental is meant to have a meaning, and experimental code often has high code churn as the whole point is to quickly fix and resolve bugs for users so it can get to a stable state.
There is no reason why that cannot happen in linux kernel tree and to be honest I don't think linux kernel devs really want this. They usually hate things like filesystems being developed out of tree because long term it causes many more problems then it solves. And if thats the case, you can't have your cake and eat it too.
TRKlausss@reddit
Well that’s what a dictator is: it’s his ball after all.
I understand where you want to come with the experimental code bit. But the kernel has other mechanisms to interact with it, so if you want to make sth experimental, you can always do DKMS first, and when you are mature and want to follow that release… guideline… then you can make. Pull Request that it is maintained in there. I think however that wouldn’t have changed the end result in this case though.
Optimal-Procedure885@reddit
you still don't get it do you. You, your filesystem, and its users are not the centre of Linux kernel development. The 'critical fix' as you call it is relevant to perhaps .00000000000000000000000000000000000000000000000001% of Linux users at this juncture.
mdedetrich@reddit
If anything the MCAS failure at Boeing is completely disproving your point, as it was middle managers who twisted the processes/rules to appease shareholders to improve profit while completely ignoring engineer feedback.
Kent is the engineer here, not the middle manager.
InstanceTurbulent719@reddit
Goes to show how many boomers have undiagnosed, untreated autism
braaaaaaainworms@reddit
Autism doesn't make people into assholes
Mars_Bear2552@reddit
depends.
braaaaaaainworms@reddit
No, it does not. Autism is not an excuse to be an asshole.
Mars_Bear2552@reddit
never said it was an excuse. but autism can make people seem like assholes. definitely has an effect on your personality that other people can interpret as being malicious or rude.
minus_minus@reddit
The youngest boomers are 60.
mrtruthiness@reddit
Boomer? Kent is not likely a boomer.
0riginal-Syn@reddit
Technically, he's an X Gen, but the same certainly can hold true for my generation as well.
0riginal-Syn@reddit
Yep, thought he was a big fish in the pond instead of just a small fish that has to learn to work well with others to survive.
elmagio@reddit
It's insane that he thought (and probably still thinks) the Linux kernel, one of the most essential pieces of digital infrastructure, had to adapt to his project. That his filesystem, still in an experimental stage and not used by default in any major distro and not deployed at any sort of scale comparable to the other Linux filesystems, somehow deserved to get exceptional treatment with massives patches outside of merge windows.
BCacheFS is very promising tech. It really is. And I would have loved to move to it once it got to a more stable state. And now that may not end up happening ever.
0riginal-Syn@reddit
That is the frustrating part about it. BCacheFS showed a lot of promise. Yet he couldn't get out of the way and follow the basic rules for a project that the world largely runs on.
Middlewarian@reddit
"Down these mean streets a man must go who is not himself mean." True of Charlie Kirk and the (character) assassination of Kent. I've been walking down the streets for 26++ years.
mrtruthiness@reddit
It will eventually stabilize to the point that someone other than Kent will volunteer to be the maintainer. Two years is my best guess.
MarzipanEven7336@reddit
LMAO, this guy should run for president.
mrlinkwii@reddit
i see linus is being linus again
polongus@reddit
Hilarious that Kent is now "asking" for a depature from openSUSE release process :)
mrtruthiness@reddit
Specifically:
With some back and forth. OpenSUSE explaining that they don't use DKMS, they have their own process (KMP). And Kent basically asking for one more release. If what Kent is saying is true (that there are no critical bugs yet found in the 6.16 = 6.17 release or bcachefs), I don't see why they couldn't wait to turn it off. On the other hand, at this point the code is still there anyway, it's just a compile-time flag. I'm fine either way: It will only significantly impact people who are using bcachefs as their root filesystem ... and anyone who has an experimental fs for their root filesystem has volunteered for that challenge/impact.
minus_minus@reddit
Kent seems to implicitly refute this. He seems to want exemptions to standards and processes because it will save user data.
mrtruthiness@reddit
Nobody is going to lose "user data" because openSUSE removed bcachefs from their kernel. It will simply "fail to mount" ... which is no big deal unless it's their root filesystem. And I maintain that anyone who has an experimental filesystem for their rootfs has volunteered for a challenge (and those people won't lose anything either; most likely they have grub/whatever set up to be able to use an earlier kernel so that they can compile the newer kernel with the flag set to include bcachefs ---> one flag and a compile).
minus_minus@reddit
That’s my point but Kent seems deluded into thinking Herculean efforts should be made to protect that data.
nightblackdragon@reddit
Imagine using experimental file system and being surprised when you lost your data.
TheOneTrueTrench@reddit
There's a reason for that, he doesn't support LTS kernels or maintain anything except "the current version" of BCacheFS.
Meaning if you're on a stable release distro, and there's a bug discovered, he won't fix it in the major.minor version you're using, you have to upgrade to newest version.
mrtruthiness@reddit
Yes.
If you read the messages in regard to the Debian and Fedora dust-up that he caused, it's completely clear he doesn't understand the job and purpose of a stable distribution (i.e. non-rolling-release). And Fedora is IMO semi-rolling release since their stable support is only roughly 1 year.
I kind of wish the stable distros would just ignore him --- that's what he asked for. However, it looks like he's found a Debian dev to volunteer to maintain bcachefs-tools (assuming Debian approves). I will say that while most of the issue is due to Kent being clueless about what "stable" means ... some of the issue is due to the fact that the nature of the Rust language makes it very difficult for stable distros.
TheOneTrueTrench@reddit
Definitely the choir here. I've actually tried to explain to him that no, Debian is never going to ship updated versions of bcachefs-tools in stable (outside of backports, that's what it's for), because doing something like that would obliterate all trust in the distro as a stable release.
You don't ship updated versions in stable, EVER. You can fix bugs, but you never force-upgrade every user on your distro to a completely different version of a tool, just because its more usable. Even if the old version is just broken and unusable, you STILL don't do that. Ever. You don't know why people have that version installed, maybe it only works for one specific use-case, but you just cannot go changing Debian Stable users' software versions like that, it's incomprehensibly irresponsible.
But he just doesn't understand or care why people use Debian or actual stable release distros.
Which means that RHEL, Debian, and SuSE look at his filesystem and understand it's utterly unusable. Whatever version they ship, he's going to immediately abandon. No one except rolling releases would ever consider touching that.
kI3RO@reddit
what's that?
kinda_guilty@reddit
Kernel Module Packages.
kI3RO@reddit
Is this an opensuse thing? I've dealt with many servers in my lifetime but haven't encountered this before, and I'm very well versed in dkms.
Will read more of course but any tldr for the thread?
polongus@reddit
Yeah I don't think he's actually out of line here, just the irony of him asking to delay a patch until the next release...
UnassumingDrifter@reddit
His tone was reasonable, and the motive is actually quite understanding. I run btrfs on my Tumbleweed systems. I'd sure hate to update one day and have the system not come up because the filesystem driver has been removed. His penchant for pushing the limits in the kernel tree aside, I am saddened Tumbleweed isn't looking at this from their users viewpoint. Now, run an "experimental filesystem" and get "experimental issues" does seem fitting but still, it'd be nice to give more than a week or two notice. And while we're at it the notice should come up during
zypper dupactions with big red letters letting people know, making them acknowledge, etc. This will be a major issue for many people sadly.mrtruthiness@reddit
Why? Is openSuSE not set up to be able to boot into an older kernel if there are issues? Is it that hard on openSuSE to take the current kernel source, change one compile flag and recompile???
UnassumingDrifter@reddit
You clearly have a deeper understanding of the inner workings. For me, sure if it breaks I can roll back with snapshots. But that doesn't fix the fact I'm now struggling to figure out how to change the system up. Some of us run btrfs, or bcschefs, and know just enough to use it. But when it goes wrong (or need to compile special kernels) we don't know where to start.
Could I learn? Yep. Everything I do know I had to learn and I'm not a noob any more but still. Many of us are not at the "just set a few flags and recompile your kernel" level. I recall looking into zfs instead of having to use ext4/bcache for my raid array. Sadly yes I could find how to compile it into the kernel but I also found a lot of posts about breaking changes more than once. I can't always dedicate hours or days of time to sort an issue.
So while you wonder why know the answer is because not all of us have the same skill level. I'm learning, yes, but since this is for fun in the evenings I have time the amount I progress is much slower than someone who does this day in and out for a living with a team of skilled pros.
mrtruthiness@reddit
I don't know about a deeper understanding. Perhaps it's experience:
a. In the old days --- long before there were loadable kernel modules --- you pretty much had to compile your own kernel to support your hardware (e.g. you couldn't use your CDROM without recompiling ... unless you could afford a SCSI one, you had to enable your particular "sound card", etc.). It turns out it's not very hard. It's worth doing as a learning exercise for your distro.
b. Also, I've used Linux long enough to have a new kernel break my system. Most distros set things up so with each new kernel install, it saves a way to boot from several choices of older kernels (typically a grub menu choice). I've had to use these several times.
IMO: People need to have enough knowledge on how to restore/recover your system so you lose the "fear of change".
bcachefs is "experimental":
a. It probably shouldn't be your rootfs unless you know more than "just enough". You need to know enough to be able to boot back into the rootfs or recover from a livecd and backup.
b. Whatever is on a bcachefs fs should have good backups or be unimportant.
NinthTide@reddit
This (correct) use of capitalisation makes me sad
I’ve been enjoying reading it (knowingly incorrectly) for these last few months as
BCA Chefs
RealKleiner@reddit
Running a rolling distro, and users are surprised when said distro moves fast. shrug Hold off on updating, build the kernel for yourself or get involved in making BCacheFS work on openSUSE.
the_abortionat0r@reddit
Has nothing to do with a rolling distro and everything to do with idiots running experimental software expecting non experimental treatment
RealKleiner@reddit
Yeah, that's true. I was just suggesting that the crazies would be impacted faster/earlier compared to if it was Debian or similar instead.
JimmyRecard@reddit
At this point bcachefs could be the second coming of the computer Jesus, and I would not use it with such a loose cannon in charge of it.
the_abortionat0r@reddit
Yeah, honestly he was already annoying but if the FS was good and proven after a few years I would have used it but at this point he is just such an unstable human it's leaked into the FS itself
He thinks he is tech God and it wouldn't surprise me if a write whole condition gets found in Bcachesfs because of his arrogance (that and one has made its way into just about every FS at this point.
Drwankingstein@reddit
Very sad to see this happen, Bcachefs has been great as a rootfs on all of my systems, a couple arch, a fedora rawhide, and an opensuse system. opensuse making this decision is quite sad to see since it will mean the system will be unbootable with mainline kernel and what not.
oh well, I needed a reason to leave opensuse anyways.
the_abortionat0r@reddit
Anyone running an experimental file system as their daily driver is a fucking idiot. FULL STOP. If someone gets an unbootable system the blame lies with themselves.
throwawaymaybenot@reddit
not sure why you are getting down voted. Experimental software shouldn't be in prod. You should expect experimental software to break at some point.
the_abortionat0r@reddit
I swear the Bcachesfs cult are all like this.
If there's a bug then it's not bcaches fault it's still in development but then the exact same people freak out claiming that not getting patches approved or in this case removed from the kernel is somehow detrimental to end users.
It's like Debian users setting their repos to the newest ones possible then accuse packages of being unstable or in an unfinished state (because that's the ones they pulled).
This behavior and mind set is a mental illness.
Drwankingstein@reddit
tremendously bad take. You are expecting to hit bugs, not for a distro to just nuke compatibility on you.
the_abortionat0r@reddit
Lol, treating software in testing like it's in testing is a bad take?
You're insane. There no nuking compatibility because there never was any. It was only ever in the kernel FOR TESTING.
You kids have a lot to learn about software stages.
MarzipanEven7336@reddit
With the loudest of voices, haaaa haaaaaaaaaaa!
minus_minus@reddit
In b4 Kent shows up to say everybody needs to focus on maintaining data integrity in an experimental file system even if it means breaking any rule and redesigning complex systems in the fly.
the_abortionat0r@reddit
Nothing surprising. Just like Ken the code isn't stable or getting any better.
S1rTerra@reddit
So basically they took it out back and shot it