Seeking Solution for Docker Orchestration and High Availability

Hello All,

I’m looking out to find a solution for orchestrating our Docker environment.

We’ve been using plain Docker compose for several years on a Ubuntu VM running under XCP-NG. However, we’ve realized the limitations in scalability and high availability.

In an attempt to address these issues, we set up a small lab with three servers, each equipped with six 2TB SSDs. We installed Proxmox and deployed Ceph as a hyper-converged system with dedicated 10GbE LANs for public and private Ceph communication.

Our initial plan was to use HashiCorp Nomad for Docker orchestration. However, we encountered complications in configuring Ceph client mounts for CephFS. This hindered our ability to establish a consistent location for dockers presistent data, and we haven’t been able to find a solution.

As an alternative, we’re would like evaluating TrueNAS SCALE. Since we haven’t found many examples or documentation, I’d like to understand how TrueNAS SCALE functions as a hyper-converged solution and whether it can meet our needs for Docker orchestration and high availability.

Thanks for the advice,

Hey @Tal_Bar-Or

At present, TrueNAS SCALE uses Kubernetes (via k3s) for containerized applications. We’re shifting to a native Docker approach in our upcoming 24.10 “Electric Eel” version (details available in our “Future of Electric Eel and Apps” thread) so if you’re intending to use Nomad/Swarm/etc that might be the ideal “out of the box” solution.

Our Community users have also documented the process of configuring Docker inside of a systemd-nspawn “sandbox” on the present 24.04 “Dragonfish” release, which would be manageable just as any other Docker installation. If Nomad requires a server-side binary or significant system customization, this might be a preferred option as opposed to trying to wedge the binaries into the native TrueNAS install (which would likely require “development mode”)

Hope this helps!