alt Hacker NewsStarted a guide to writing FUSE filesystems in Python
Comments
Just wanted to throw out there that although I'm a fan of FUSE, it's not the only option. I've had fun implementing a virtual filesystem via the 9p protocol not too long ago.
IIRC, I used py9p and the python experience was much nicer than fuse-python. You can mount a 9p service via FUSE (9pfuse) if you want. I just used the kernel v9fs client. If you're just looking to pass a filesystem through the network, I think I used the diod 9p server.
Overall, it's a nice little ecosystem to explore.
I wish I had known about this a month ago, when I had to go through the exact same process!
In a desperate attempt to find a less frustrating way to interact with Jira, I had the silly idea of starting a jira-as-filesystem project that uses our internal issue categorization to build a tree: directories represent issues, with files representing issue fields and subdirectories for linked issues. I ended up choosing fuse-python.
I haven't worked on it in a minute, but I was already bumping into issues (pun not intended) with the abstraction: using just the issue ID as directory name makes automation easier, but it makes it hard for humans to browse the tree, since a `ls` would just show you a bunch of inscrutable IDs. I ended up adding a parallel `<issue-type>-with-summary` directory type where the slugified summary is appended to each issue ID.
So many fuse mount options out there with varying tradeoffs, performance, and features (s3fs, goofys, seaweed, minio, Google drive, etc..). JuiceFS is pretty interesting for doing things like mounting an object store and accessing it via posix with all the metadata you would expect on a traditional filesystem. https://juicefs.com
Nice!
Adjacent question: lately I’ve been seeing people implement NFS base filesystems since that is a more widely supported protocol. I think rclone does this for Mac. Is there a guide, or even a comparison, for this approach?
If you're interested in seeing what a finished product looks like, check out azuline/rosé — a music manager with a virtual filesystem. Really good codebase with a lot of comments and explanations and types and tests, which should make it easy to learn from.
People interested in FUSE might also be interested in the CUSE companion (sub)project.
CUSE is userspace character device driver emulation. It allows you to emulate hardware without compiling a new kernel module. I just used it recently to write a hardware device supporting IOCTLs using Python. However I didn't find any good Python libraries that worked easily and documentation was lacking, but I found it easy enough that I ended up writing it using just the ctypes ffi library. The only part that wasn't immediately intuitive for me, as someone who has never written kernel drivers, is how IOCTL's require the buffers to be resized for each read and write which means the calls come in pairs, but luckily CUSE has a debug mode which shows all bytes in and out. CUSE was originally implemented to create userspace sound devices¹ but has also been use for things like custom TTYs. I used it for creating a virtual SPI device. Hopefully someone finds this useful and this project can get more attention.
Quite some years ago I created a Python FUSE filesystem[1] to to interact with dokuwiki (a wiki system).
It's built on hde llfuse[2]. But that required implementing a bunch of low level APIs that were not really related to dokuwiki. So I created easyfuse[3][4] as a wrapper, which implemented the things that were unrelated the dokuwiki implementation. If you're interested it in building a FUSE system it might be worth looking at.
[1]: https://github.com/JelteF/dokuwikifuse [2]: https://pypi.org/project/llfuse/ [3]: https://pypi.org/project/easyfuse/ [4]: https://github.com/JelteF/easyfuse
I like to think of fuse as a way of allowing Makefiles to specify DAGs over arbitrary resources. For instance, a fuse fs exposing the state of a k8s cluster might ease writing operators accessible for simpler minds like mine.
Or email, why not expose imap through a file system, so your RAG app (like gpt4all) can just access everything directly ?
Off topic but whenever I see a blog with some 90s/2000s vibes, I always go to their first page of posts. Never disappoints to sneak a peak into that time capsule - including gwolf.org!
I've recently discovered sshfs and learned about needing to have FUSE as a dependency for OS X, which spiked by interest. The code looks very clean and easy to understand, so thanks for that! Is there any guide/course you would recommend for the introduction to FUSE? It looks like all you have is to provide implementations to certains functions your filesystem will use but it's hard without knowing the details(ex. I wouldn't know I had to implement readdir without your code, and so on)
I did something similar and it was a really fun project! You can easily make a Google Drive FUSE fs, or something simple like an in-memory fs, an encrypted fs, etc... Its very interesting and a lot simpler than one would imagine. You basically fulfill an interface and FUSE isn't really aware of the implementation. Its more of a "contract" that X function returns a given result. You can implement a FUSE fs for a ton of cool stuff.
I wrote a little project to expose a bar git repository in fuse. Basically, you just have your .git folder, and fuse exposes every single commit (and branch etc) as its own folder at the same time. Without actually having to checkout everything as regular files in a regular filesystem.
It's quite nice, and really shows that git internally is already half a file system. It's also quite simple, because everything is read-only.
I attended a talk at Devoxx yesterday which showcased how new Java language features can be used to implement a Fuse filesystem. Basically as simple as generating Java bindings for Fuse using jextract on fuse.h, and then implementing a couple of method calls using the new Foreign Function & Memory API, which is set to replace JNI.
I will link to the video when it comes online, should be later today.
An user-space filesystem running in an interpreted language? Is that as bad as I think it is?
The libfuse github has some good examples for C/C++ in [0] of increasing complexity:
- passthrough.c mirrors existing filesystem, "Its performance is terrible."
- passthrough_fh.c "performance is not quite as bad."
- passthrough_ll.c implemented with low level api and "the least bad among the three"
- passthrough_hp.cc high performance version written in C++
Some interesting fuse projects in my notes: [1] splitting large files into segments; [2] show ZFS incremental snapshots as files; [3] transparent filesystem compression; [4] and [5] options for mounting archives as filesytems.
- [0] https://github.com/libfuse/libfuse/tree/master/example
- [1] https://github.com/seiferma/splitviewfuse
- [2] https://github.com/UNFmontreal/zfs_fuse_snapshot
- [3] https://github.com/FS-make-simple/fusecompress
- [4] https://github.com/google/fuse-archive
- [5] https://github.com/mxmlnkn/ratarmount