Has anyone experimented with creating their own File System?
Posted by ki4jgt@reddit | learnprogramming | View on Reddit | 8 comments
Think it would be cool to create something like:
```
ToC
file-x: seek: 2048
2048: file-x contents
```
Or, even better, if the FS was a merkle-dagg with self-healing and native encryption. The ToC could index where all the blocks started, by their individual hashes.
Something you could mount in your native file system, plug other devices into and expand/shrink as needed out of the box. 2 1TB USBs would just instinctively form a 2TB storage system, without any special software.
mxldevs@reddit
This is pretty standard for file archive formats. Which means if you change anything, you have to rebuild the entire archive to regenerate the offset.
I assume file systems need to be more robust
high_throughput@reddit
I haven't because of the intense amount of non-feature work that goes into a FS.
It has to be fast and efficient for heavily multithreaded loads, have strong durability in the face of power loss, and deal with hardware quirks. It has to follow all the userspace conventions related to sync behaviour, implement everything needed for various kernel subsystems like io_uring, and be resilient against abuse and poorly written applications.
To be able to focus on the features you describe, you might want to implement them in a layer above the FS.
ki4jgt@reddit (OP)
Why?
Why convolute something so simple?
Just begin at the start and end of the drive. Start your index at the top of the drive, with byte locations for each block.
Then build blocks from the bottom of the drove up, let the ToC and context meet in the middle. As the drive wears out, you just skip the damaged slot and pretend it doesn't exist.
If you use an intermediary language, like Python (probably something faster), the hardware and OS problems don't exist. Just create a mount point which interacts with your software, as if it's a real drive. You'd also be creating your own sync system over tcp/ip. One device wants a certain block, so it sends out a search broadcast into the LAN.
Then you can bake a memcache into your software, keeping the most frequently called blocks in active memory. All that should cut down on the various abuses.
You're taking the fun out of this, man. 😄
aanzeijar@reddit
I mean, why complicate it. Just write your blocks to twitter and index the links. Free memory!
ki4jgt@reddit (OP)
😆
John2143658709@reddit
i think everyone else is crazy in this thread. You should do this to learn, it’s not even that hard to get started. I was looking at mounting a custom filesystem for another project and used the “fuser” crate in rust. You only have to implement like, 5 functions out of the 30+ provided and you’ll have a working fs. You can just search up “FUSE” in any language.
high_throughput@reddit
How is this simple system going to handle deletions, fragmentation, sparse files, and efficient file indexing / random access?
It might be better suited for S3 style blob storage, which is also very useful but with way fewer operations to support.
dawalballs@reddit
So you’re implementing it above the file system like he said?