Is it possible to mount S3 storage as a filesystem/backend? #6

New Issue

jadudm · 2025-12-30T12:45:01Z

jadudm commented

2025-12-30 12:45:01 +00:00

I'd like to use Minio or similar as a backing store to a typical filesystem. Is that possible?

jadudm commented

2025-12-30 12:45:08 +00:00

Yes. Multiple ways. A few include:

This is also handy looking, if you're looking to merge multiple drives:

https://github.com/trapexit/mergerfs

notes

JuiceFS is interesting because it provides a transparent layer through to object storage.

JuiceFS stores metadata about the filesystem in a transparent way: a database. It can use SQLite, Redis, or Postgres (amongst others) as the metadata store. This means your metadata store can be completely local, or it can be remote and shared amongst machines.

The metadata is backed up into the bucket regularly (hourly). E.g. if you choose an SQLite store, it will be backed up into the bucket. This means that, if the local client is lost/crashes, you can install the client elsewhere, download the metadata, and reconnect to the bucket. (The client may even have some recovery built in. TBD.)

It can encrypt data, so that if the underlying bucket does not have encryption, then keys can be managed at the client/JuiceFS level.

What is most interesting, in some ways, is the high level of POSIX compliance. ACLs, hardlinks, and other features often missing from FUSE filesystems are present. This should mean that it could be tossed on a machine, and rsync could be used to backup data directly to this mounted filesystem, and it would be backed to a cloud object store. In some ways, it provides a more transparent way to backup to an object store than (say) rclone or similar. Applications only have to be able to write to the directory, and the encryption and transfer to the object store "just happens."

things to think about

When using an object store with ingress/egress costs, you'll be paying for every filesystem operation. However, mounting it against a local Minio instance provides a mechanism to go from filesystem to object store, and then that object store can be periodically sync'd to a remote bucket. (And, that remote bucket could become a JuiceFS backing store, if needed. It just wouldn't be "realtime.")

So, kinda interesting.

Yes. Multiple ways. A few include: * https://github.com/juicedata/juicefs * https://github.com/awslabs/mountpoint-s3 This is also handy looking, if you're looking to merge multiple drives: * https://github.com/trapexit/mergerfs ## notes JuiceFS is interesting because it provides a transparent layer through to object storage. JuiceFS stores metadata about the filesystem in a transparent way: a database. It can use SQLite, Redis, or Postgres (amongst others) as the metadata store. This means your metadata store can be completely local, or it can be remote and shared amongst machines. The metadata is backed up into the bucket regularly (hourly). E.g. if you choose an SQLite store, it will be backed up into the bucket. This means that, if the local client is lost/crashes, you can install the client elsewhere, download the metadata, and reconnect to the bucket. (The client may even have some recovery built in. TBD.) It can encrypt data, so that if the underlying bucket does not have encryption, then keys can be managed at the client/JuiceFS level. What is most interesting, in some ways, is the high level of POSIX compliance. ACLs, hardlinks, and other features often missing from FUSE filesystems are present. This should mean that it could be tossed on a machine, and `rsync` could be used to backup data directly to this mounted filesystem, and it would be backed to a cloud object store. In some ways, it provides a more transparent way to backup to an object store than (say) `rclone` or similar. Applications only have to be able to write to the directory, and the encryption and transfer to the object store "just happens." ### things to think about When using an object store with ingress/egress costs, you'll be paying for every filesystem operation. However, mounting it against a local Minio instance provides a mechanism to go from filesystem to object store, and then that object store can be periodically sync'd to a remote bucket. (And, that remote bucket could become a JuiceFS backing store, if needed. It just wouldn't be "realtime.") So, kinda interesting.

jadudm closed this issue

2025-12-30 12:45:08 +00:00

jadudm added the bash filesystems labels 2025-12-30 12:50:41 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: jadudm/how-to-do-computer-things#6