Choosing your storage path: how I decoupled my Kubernetes workloads from data using S3
Published: 2026-08-18
When I decided to leave Github, part of my plan was to also self-host my own S3 storage service for my Kubernetes homelab. I went through and tried out various open-source solutions, looking to manage my growing homelab needs and decide if self-hosting an object store was the right move.
How I got here
Why use S3 storage in the first place, and why even self-host it?
Personally, I wanted to start using object storage as a means to decouple my workloads from their data. This separation of concerns would keep my Kubernetes compute layer lean, focusing solely on running containers efficiently, while the heavy lifting of data management, archiving, and backups could be offloaded to a service cluster designed for it.
In addition, I was running a few services that I felt would benefit from using unstructured data as a storage medium:
- Immich Photo Library: While, S3 isn’t supported out of the box, I looked into using CSI driver solutions that could give me POSIX-esque compatibility.
- CI/CD caches: The API for dumping artifacts into an S3 bucket is more ergonomic than managing and mounting persistent cache volumes in Kubernetes which the get attached to designated workflows.
- Backups: Longhorn (my current distributed block storage solution) provides great tooling to upload backups to an S3 bucket!
My rationale for self-hosting came down to a question of ownership; I still wanted to own my data. For disaster recovery, the ideal situation might be to regularly replicate my buckets into cold storage somewhere; I’m still trying to figure that part out!
What I learned
Initially, I went all in on trying to replace my block storage solution completely. Solutions like SeaweedFS, promised broader functionality by also offering a POSIX compatible filer service on top of its S3 service. While these features were appealing, I eventually learned that it can’t replace using block storage for running databases. Object storage is built to handle large immutable files; the transactional nature of SQL databases and the high frequency of read and writes was not viable on SeaweedFS.
I ended up choosing Garage as the solution for my storage cluster. Mostly because it offered less but more focused features:
- Lightweight design: It ships as dependency-free binary that can run on machines with memory as low as 1 GB of RAM.
- High resilience: It’s built for redundancy and gives you the tooling for replicating data across multiple zones to ensure resilience against network and disk failures.
- Operational focus: Fast to deploy and safe to operate, makes it easier to manage for smaller teams.
At the scale I’m running at, it’s the right tool for maintaining my storage needs without any unnecessary overhead. Maybe in the future when I’ve got more nodes and bigger beefier hard drives, I can give running Ceph or RustFS a shot, but with the current resources at my disposal, that might be overkill.
Final thoughts
Ultimately, I found that neither object storage nor traditional block storage is an either/or proposition in modern infrastructure. The takeaway is that one needs to understand the specific limitations and strengths of each tool. With a good understanding of these boundaries, one can select the correct combination that provides the necessary resiliency and functionalities for your use cases.
In my case, I realized that block storage still has it uses, don’t try and get rid of it entirely. I got what I needed in the end, and have been pretty happy running Garage along side Longhorn for the past few months in my storage cluster.