k3s was meant to manage apps across clusters when SCALE could actually scale out. The end of Gluster meant that this plan can no longer realise, which then makes k3s unnecessary.
I suppose that iX has some kind of a long-term plan, at least in the form of a general direction (a “vision” in corporate speak). It’s just that the long-term plan keeps changing on a medium-term basis to adjust to Reality. 
Yes, we made the mistake of believing that IBM/Redhat would hand off Gluster to the community in a reasonable way… rather than abandoning it midway through a release. Reality can be nasty and we have to cope with it.
Unlike a proprietary system, TrueNAS uses the best of the Open Source world. We also contribute like with the upcoming Fast Dedup. However, our plans will change as new Open Source tools become available (or are deprecated).
I do not for a second believe that there was any Machiavellian plot here. It is easy with hindsight to say that K3s was a bad decision, and by changing directions like this ixSystems are effectively admitting that with hindsight it was a mistake. But hindsight is a whole lot easier than foresight - and I have no doubt that when ixSystems were considering how to deliver apps on Debian they thought that K3s was the way to go.
I feel the same - I am not a Kubernetes evangelist either.
For me the first rule of thumb is KISS - Keep It Simple - so why use complex Kubernetes when a simpler Docker / Composer architecture can deliver the needed functionality. If Enterprises want complex containerisation with clusters and failovers etc. they are NOT going to use TrueNAS for it - they will build out a dedicated Kubernetes environment. So Apps are essentially for the smaller users.
As someone who has managed IT from strategy through to nuts and bolts for several decades, the issue here feels like ixSystems forgot the KISS rule, over specified what they thought was needed, and ended up choosing complex Kubernetes rather than simpler Docker/Composer.
The second rule of Thumb is MISS - Make it Simple (to use) - i.e. in the spirit of the rest of TrueNAS (which takes ZFS and wraps it in usability functionality) you take the technology you select and wrap in a UI to make it easy to use. They did this for K3s - and had they chosen Docker / Composer from the start, they would have done it for that instead.
And this is exactly what is worrying me about the new Docker / Composer proposals outlined above - that they are NOT going to wrap the Docker/Composer functionality in something to Make it Simple.
I think that there is some element of truth to this, but this doesn’t quite hit the spot.
ixSystems, for whatever reason, backed the wrong horse. Admitting this and changing horses is going to cause pain for the existing user community, but for the long-term this probably still makes sense. K3s is just too complex with too much overhead for the use case it is aimed at.
The solution IMO is to invite greater participation by the non-Enterprise community to at a minimum validate strategic plans in much more detail and ideally to participate more in defining such plans and not just validating them. In the end ixSystems will make their own investment decisions, but at least they would have a broader base of input than they currently do.
As an idea for discussion / consideration within ixSystems, I would personally suggest a Community Strategy Group be formed, chaired by @kris and with community members who have a mixture of different experiences with TrueNAS or with NAS in general or with Linux / ZFS in general or IT in general or development / UX / Operations Management etc. and perhaps with some other key members of the ixSystems team who have a particular affinity with the needs of the smaller NAS Community.
I believe that had such a group existing when the technologies within SCALE were being selected, there would have been a much greater chance of Docker/Composer and possibly Sandboxes/Jails having been selected from the start (assuming of course that the technologies were available at the time).
This is good to know - now we have this input we should have more confidence in investing in implementing TrueCharts apps, and clearly should wait to see what form this will take before thinking about other migration approaches.
GOOD!! I believe that TrueCharts are genuine Kubernetes/Helm experts and will try to find and develop the best possible implementation as a strategic platform rather than picking a tactical solution solely as a survival approach.
BAD!! TrueNAS is an appliance - building a strategic solution may be difficult without breaking the appliance concept.
See above.
Sounds pretty reasonable. Except that if you want an App which is in the TrueCharts catalogue and not in the TrueNAs catalogue your choices are currently:
For most people this is not much of a choice - use TrueCharts now regardless that it is now based on technology declared obsolete by ixSystems and regardless of the rework effort required later.
As above, sound advice.
Very good to hear.
Or use the app-supplied Docker Compose / Docker Run command.
…which will not give you equivalent functionality to the TrueCharts app. To name one thing it appears nobody else but me cares about, it won’t do ingress with Traefik.
Docker works just fine with Traefik. I have it setup myself. I even got the A+ rating on SSL labs. Techno Tim over on youtube actually just put out a video on setting up Traefik with Docker a few weeks ago, which is fantastic and shockingly easy.
Of course it does. But “the app-supplied Docker Compose / Docker Run command” isn’t going to set it up for you.
Regrading Traefik
Traefik will become a TrueNAS Community App… setup via Docker Compose. Its worth a new thread to discuss what is needed to simplify the set-up required.
I feel like such a pariah in all of this.
Currently I use pkg install and pkg update within a FreeBSD jail to install software.
If I ever switch to SCALE (still not sure), I plan to use pacman -Syyu within a “Linux jail”.
That’s it. Nothing more.
For home users (and “power users”, if anyone still uses that term), why do we need additional containerization inside a jail?
I can enjoy quite a good amount of software in my FreeBSD jails, with as much control and flexibility as I would expect sitting in front of an Arch-based distro. (Before Jellyfin was available as a binary package, it was the only software I could not run; but that is no longer the case anymore.)
With a “Linux jail”, the issue of unavailable software is even less of a problem.
Case in point: yt-dlp
A program like yt-dlp is critical to keep updated frequently. (Just take my word for it. There are reasons why this is so.) What version is it available as a “container” / “Docker image”? Right now, as I type this post, is it version 2024.05.27?
If it’s not, then it’s a non-starter for me to even consider using “Docker whatever”.
Currently on FreshPorts (FreeBSD) and Arch Linux, the version is 2024.05.27.
That’s true… running Kubernetes or Docker in a sandbox is an option, not a prerequisite.
Nor do I.
Was it, though? On what basis?
It’s clear that iX now believe that Docker is what people want. That’s nothing new; people have been clamoring for it[1] at least since TRTMNBN[2], which even had half-baked support for Docker in the form of a semi-hidden Linux VM running Rancher and using 9pfs to access the NAS storage. But TRTMNBN was a disaster[3], so iX pulled the plug on it and ended its mercifully-short existence.
iX now appear to be convinced that “Docker” is what people really want. And maybe it is, and maybe my read of the user base is wrong. But I don’t have any confidence that the “app catalog” is going to last a year in any meaningful way. I suppose time will tell.
Maybe it’s just time to do what the more-conservative among us have been saying for years, and use the NAS as just a NAS.
Its important to note, that this has absolutely nothing on common with Traefik as users are used to with TrueCharts.
Traefik in Docker, is not the same thing as Traefik as a kubernetes ingress provider.
than picking a tactical solution solely as a survival approach.
This is indeed part of the reason why:
Yes while a sandbox-based approach might work “reasonably” well for now, we simply do not see the majority of our users maintain such a thing. Or at least, not without significant issues doing so.
It will likely be VM-based, to ensure we don’t break the appliance concept.
+ custom jails with a caveat that any software used is not endorsed nor supported by iXsystems.
Don’t take my jails away, @dan. Don’t take away my almost zero overhead software that can run directly on my NAS’s filesystem, downloads and all. (Especially for digital internet archivists.)
The reason people have been saying this is exactly because of the current existing app setup. Bringing in docker will allow people to use practically whatever apps they want without relying on proprietary third-party app catalogs and ecosystems.
100% accurate. With my docker setup, I don’t have to click “start” on my external apps when truecharts traefik randomly and frequently forgets all of my external ingress setups.
I don’t wanna seem like I’m yucking someones yum. If you love truecharts and haven’t had any issues, great! If you wish to continue using them into the future, you can spin up a Talos VM or any other Linux VM inside of Scale and setup helm proper. This is considered a higher tier vs using Scale, according to truecharts own documentation: Stability Tiers and Helm Support | TrueCharts Charts
I don’t have to click “start” on my external apps when truecharts traefik randomly and frequently forgets all of my external ingress setups.
Sadly enough that is a SCALE specific bug we’ve not been able to reproduce outside of SCALE. From a technical perspective the “start” button shouldn’t and doesn’t do anything of relevant from a Helm/Kubernetes perspective, but appearently the middleware does “something” we haven’t been able to bugtrace.
This is considered a higher tier vs using Scale, according to truecharts own documentation: Stability Tiers and Helm Support | TrueCharts Charts
This is 100% correct, it’s niche issues like the above that lead us to (re)classify systems in those different tiers. The more layers there are in-between, the more risk of something wreaking havoc or something like iX-Systems pulling the plug.
You know how TrueCharts does it? Do it like that. If you don’t know how they do it, well, it’s pretty well-documented on their website. If you can simplify it further from there, great, but that would do.
That was probably kind of snarky, but really, they’ve been doing this for a couple of years now. Each app has a section for Ingress configuration where you give it a hostname, that links into Traefik, Cert-Manager, etc. to issue a cert if needed (or use a wildcard) and set up an ingress route to that app. You pretty much never need to touch the configuration for Traefik/Cert-Manager/Clusterissuer once they’ve been set up; all the necessary configuration is done with the individual apps.
And just as I don’t really care what system is used to handle the containers (whether k3s, k8s, docker, whatever), I don’t really care what software is used for this–if I’m setting up my own reverse proxy, I’m using Caddy, so I’d lean that way if Caddy can be configured in a similar way. But I don’t really care if it’s Traefik, Caddy, Nginx, Apache, or something else, as long as the setup in an app–without touching the configuration for whatever reverse proxy you settle on–can get the route set up, terminate TLS, and all the related functions.
But I’ve been asking for this for eight years now, and nothing from iX to date has done anything like it.
Right now, as I type this post, yes:
https://hub.docker.com/r/jauderho/yt-dlp
