Any recommendations on an interim solution for Kubernetes Ingress?
I was in the middle of prep to deploy Traefik using Truecharts, when the catalogue was disabled/blocked from Truenas charts sync.
I’ve tried deploying Traefik using docker hub however had a bunch of issues with this. I had a look at the Truecharts documentation on how they got it working and it was apparent they had to combine ~ 4 separate Traefik docker / helm builds to their current setup.
My question is if there are any recommendations on an ISO VM that has Traefik pre-build for running as an ingress gateway for the existing Truenas Charts Apps running on Docker / K3S.
I was 95% complete setting up unifi controller and acme / cert manager when the Truecharts catalogue was revoked and would prefer to use something that is aligned with the planned changes for Truenas charts or it’s replacement.
A little bit of googling will find copies of their charts repo on GitHub. Add that to TrueNAS as another catalog and install the apps from there. From there I presume you’d be able to follow TrueCharts’ migration path to TalOS.
If you want to do this with iX’ charts, you’d be manually setting up the reverse proxy; I think Nginx Proxy Manager is the only option they have.
Alternatively, you can put Docker in a sandbox (I’m sure Stux will be along shortly with a link to his video), put Traefik in there, and use labels in the various compose files to handle ingress.
I looked at TalOS, and to be honest I could just use the TalOS metal AMD64 iso to build a VM on Truenas. Why I’d prefer to not do that is mainly resource management becomes painful as putting many containers under that host and then dealing with CPU / Memory isolation is likely going to be painful. (Plex container runs poorly if the process roams over 2 CPUs / Memory banks)
Like using dedicated docker/container Ubuntu VM scenario, yes its easy and simple to setup, the pain part comes with what happens next. I’m essentially replicating something that already exists in Truenas (it becomes one more thing to manage).
The other option I am looking at is to see if HAProxy Ingress is part of the HAProxy Plugin for my Opnsense firewall. I could just use HAProxy default with a bunch of complex external port mappings, but using the Ingress module seems to provide some of the benefits that Traefik has along with being one less VM/Container I need to run (since it lives in my Firewall VM (which runs on my Truenas Server)
At the moment it’s hard to make decisions without a clear understanding of what iX is planning to do with Apps, and some confidence if they will maintain the existing Docker style support.
I’m not a huge fan of Sandbox’s or Jails. I understand why some people like them, but honestly, they still need a Bridge group to my Virtual Firewall. At this point I’d likely just go down the TalOS path to be honest. I’m not sure if the TalOS path means I still need HAProxy Ingress or Traefik, in order to interact with the containers sitting behind TalOS, or if TalOS can operate as an ingress Domain Name proxy itself…
I’m also reluctant to use a github clone of Truecharts as the long term support for that is unclear. Getting updates seems unlikely making it a non-starter
…or use Caddy for that instead. I find it much easier to configure than HAProxy, especially now that it has a GUI.
Understood. My confidence in iX in this regard isn’t very high[1], which is one of the reasons I’m giving serious thought to TC’s VM-based migration. But that carries its own issues with it, which may include increased use of system resources (they’ve suggested this may not be the case, but I guess only time will tell), and quite possibly a very different UX.
There will be no updates; it’s just a copy of what the TrueCharts catalog was when they archived it. But it lets you stay in that ecosystem, and migrate to TalOS when that’s ready, where their charts will be (and are currently being) updated. It’s a temporary measure until you’re able to migrate.
I’ve pretty well concluded, though, that it’s best to separate “NAS” from “apps” as much as possible. A VM on the NAS is probably reasonably safe. Sandboxes, I’d guess so, but with less confidence[2] (though they allow you to use host mounts, while a VM requires a NFS mount). So in that vein, I’m not really planning on doing anything significant with iX apps, in their current or future incarnations[3]. If TC/TalOS ends up doing what I want, great. Otherwise, I’ll probably do something on Proxmox.
I expect the next step will be for them to declare that maintaining their own Compose-based catalog is “too hard,” just as they’ve done with all the others, and abandon it in favor of “paste in whatever Compose file you want.” ↩︎
iX have said they’re going to add “official” support for them or something like them with 25.04–what, if anything, that will do to existing sandboxes is unknown, at least to me ↩︎
Though I am using their Tailscale app to manage replication between two TrueNAS systems remote from each other ↩︎
I doubt TC can maintain what they were doing without ultimately need to charge a fee for access to their Catalogue in combination with a TalOS build. I don’t fully understand what the $$ relationship was with iX but it looks like it was initially based on some level of funding for their work as it benefited Truenas Enterprise licensing and their hardware offers.
Based on that I would just as likely build my own TalOS setup and if needed use their documentation if it’s available to figure things out by myself.
I agree with you that path that is more industry standard is likely better to just run a VM and run all the containers within it. TalOS seems like a secure option for that. By isolating the container Apps from the Truenas Scale OS, if iX ever decided to stop maintaining the free edition like Netgate / PFSense did, I could fairly easily port that to a new Hypervisor.
I’m not aware, and I haven’t heard suggested until just now, of any financial arrangement between iX and TC. As far as I know and have ever heard, they are and have been two completely independent organizations–I’ve never heard any kind of endorsement by iX of TC, and in fact they often seemed to be at odds with each other. I wish they had worked together–I think a lot of the drama we’ve seen between them would have been avoided if they had–but I’ve never seen any evidence that they have.
The inclusion of “True” in the name of TrueCharts has, I think, led some to believe there’s an official connection between the two organizations, but I don’t believe that’s ever been the case.
I actually use HAProxy in my pfsense firewall to reverse proxy to my containers
The benefit of having a reverse proxy at the actual container host is that you can use split horizon dns to redirect the traffic to avoid bouncing off your firewall.
Bridged Vlan, Dedicated Interface, macvlan interface…
none of these change the fact that the packet forwarding outside this still needs to go into a routing layer that I prefer to isolate from other parts of the network as some of these services are exposed to external access. Sandboxes and Jails protect the host from the application but they don’t protect other hosts, unless they are isolated from them.
The benefit of the containers, are that they essentially reside in an isolated virtual network. The only way into or out of that isolated network is via the Ingress Proxy or exposing the services via the container gateway host IP. In both of these cases you’re effectively running a stateless port based filtering function. Effectively fire-walling the container apps from the rest of the network zone they transit through for external access.
Using Dockge and watchtower (along with the TrueNAS alerts emails when system updates are available), my set up is now more automated than when I used the official apps and TrueCharts, since watchtower finds and updates things automagically. The only container I can’t do this with is nextcloud, since I’m using a build file.
TLDR; I’m no longer logging into TrueNAS just to check for updates, which means I haven’t logged in at all for a few weeks.
For me, I don’t want all those updates! I prefer rare but planned updates. Updates can bring bugs, and cause chaos at times. For this reason, I manage them myself, on purpose, sometimes even testing them before updating the live app. If, and I do mean if, there is some big benefit to some update, I might just do it, otherwise, don’t care. Stability means more than updates for me. YMMV of course.
Lots of apps will notify you of an update, but it seems Plex users in particular are pathologically concerned with having the latest version. Not quite sure why that is.
Must be the same ones who just had to do something after Truecharts removed their catalog, no more updates? Got to convert to something else today! Or, the ones who updated to 24.04.0.0 the day it came out or very soon thereafter. I don’t understand it either.
That really is a different category, I think. Even though we should all know better than to install a .0 release from iX, they use lots of electrons hyping what’s going to be in them, and then breathlessly announcing the release. So, while we should know better, I can better understand why people (including me on occasion) upgrade TrueNAS earlier than would be prudent.
But Plex, specifically, is a whole other animal. I remember threads on the old forums to the effect of, “Plex released a new version three days ago! When will the plugin be updated??!!”
Only if you know that a specific version has in fact enhanced that. Otherwise, wasting your time and risking issues. And that’s just one app, one may have a whole lot of apps. Assuming they publish good changelogs, no idea if they do.
When you automate updates, you likely have zero idea what is being changed, is it a breaking change, are there new bugs, etc. For something like Debian, that’s typically fine as they are very conservative within a version.
