Implementing functionality for Docker Swarm

Problem/Justification
I enjoy owning my own personal data, especially in a world where more and more people seem to be interested in it. Some services offered by the TrueNAS app catalog, pihole, collabora, passlock, etc. are so critical to one’s own life that I don’t feel comfortable running them on a singular machine.

Impact
Currently, some users are given the impression that they can do these things, and whilst TrueNAS itself is a rather reliable OS, the hardware and environment that we put it in isn’t always very conducive to it staying online for long periods of time. This helps users brute-force reliability for critical aspects of their digital life by simply buying more machines.

User Story
Ideally, some user with a critical application could connect their TrueNAS system to another 2 by opening a page on the UI and entering their IP addresses (or domain names?). They could then select which applications get added to the swarm and rules for swarm usage (Certain apps can only run on certain servers, such as plex on my media server. Some apps might need to be ran on the same server that dependent apps are also running on) - Even better would be the options for the swarm to also sync changes made to the app’s storage/config datasets (although this could also be done manually via a replication/rsync job), we could even have this done automatically with a simple tickbox if the user selects the ixVolume dataset when configuring the application.

When a system fails or restarts for normal reasons, the applications that were running on it are spun-up on the other machines in the swarm, and an email is sent to the owner via the alerts that a system went down. The applications theoretically could experience some weird behavior as the sync might have last happened whilst the application was running; but from their POV that’s no different than recovering from a crash. If absolutely necessary, we could instead implement automatic update and restart rules - and sync only during such periods of downtime.