I would like to see these stats, we are not talking about just TrueCharts users. There is no way there are single digits users using the official TrueNAS apps, forget about @truecharts stats.
Also, if IX don’t want to develop their own apps, they can continue to support K3s only and discontinue their apps, not make K3s unsupported feature. That’s what bothers me the most.
yep.
“containers” is a concept/isolation/layer above “jails/namespaces”. You can have ‘containers’ in BSD as you can in Linux. Kubernetes is an automated configuration layer on top of containers.
The stats Kris reported in the other thread were that > 90% of SCALE users are using apps, and < 10% of those are using TrueCharts apps. That would mean that the remaining users are using only iX apps, which they promise will all be migrated seamlessly to the Docker-based system by the time 24.10 releases in four months. I’m skeptical, but what I think doesn’t much matter on this question.
I likewise think iX are making a bad decision, though for a different reason: by allowing third-party app catalogs in the product, they’re explicitly inviting third-party app developers to make their own catalogs. Pulling the rug out from under them, which is exactly what they’re doing here, is a crappy move.
Which is why I stated earlier that moving to Docker is a step backwards. I don’t remember when someone used the word “Docker” last time, everyone is using containers and Kubernetes.
Imagine the devs joy if Docker is dropped in two years, like recent CentOS 7 fiasco.
Again, seems like you made a back/sideward step already when you “side graded” to SCALE when ‘containers’ could have been implemented in CORE. So…chalk it up to bleeding edge?
I don’t understand why you keep repeating the bleeding edge keywords. There is nothing bleeding edge, Scale was not released yesterday.
I wish this could be a viable recommendation, but unfortunately, it’s also going out slowly…
SCALE is four major releases into its lifecycle. It shouldn’t be anything like “bleeding edge.” k3s has been out for over five years; it similarly isn’t anything like “bleeding edge.” Your continuing to repeat this doesn’t make it so.
Containers have been part of “CORE” since FreeNAS 8.0, some fifteen years ago–they’re called jails. But they aren’t any flavor of Kubernetes, and AFAIK no flavor of Kubernetes runs on FreeBSD.
Okay, the phrase obviously missed its mark. How about: “be patient and hope for the best because there isn’t much you can do about it if you don’t want to go native Linux/BSD”.
Nope. Jails are an isolation. On BSD something like Podman + Jails would be a container.
No (correct), the automation layer above containers, like Kubernetes, does not run on CORE.
Sounds like a good and scalable solution. We do see that many Enterprise level users need that flexibility and scalability and look forward to seeing the repository. We are doing two things in TrueNAS to support that:
However, we don’t see that non-datacenter users want that level of complexity and sophistication. Most of the complaints about Apps have been the complexity and bug reports of non-official charts and the inability to import well-tested Docker Compose Apps. So, the overwhelming response of the Community has been positive to this transition.
We’ve also promised that all official TrueNAS Apps will migrate over seamlessly and have built the infrastructure and process to do that.
So, the users impacted are those with custom Kubernetes Apps. We don’t like that either, but in the end we think its better for users to have standard Docker Apps or standard Kubernetes Apps. The quality of Apps and infrastructure with custom Kubernetes Apps was just not high … too few users & developers, too many bugs and configuration issues. We could only support dozens of Apps well. We want Apps to be easily portable to and from TrueNAS.
So, should TrueNAS have started with Docker and Compose? In hindsight, we would agree. We have seen huge benefits from containers, but not so many benefits from Kubernetes. We don’t need autoscaling and orchestration in the same way as Google.
Strike that! I guess there is a tool called nomad to do the automation layer on BSD.
I’m sure you’re aware that there are others who have already done this. For a couple of examples, see:
…and its credited sources. Which isn’t to say there’s no room for another implementation, of course. Personally, I think I’d be more interested in doing this with Talos than with k3s on some random Linux distro, but maybe that’s just me.
Documentation: AXIVO | K3s Cluster
I was hoping IX will also adopt Cilium with K3s, please let’s keep K3s as production feature in Scale, not unsupported nightlies only. I also strongly believe Longhorn is a robust choice for Scale and K3s, it will solve so many problems. Suse put a lot of work effort to make it a central distributed block storage in their software components.
Definitely not with the modern features that my repo offers. It is a production grade, complete turn-in-key solution. The documentation explains everything.
You can use Talos, but I still think Ubuntu Server is a robust choice and more flexible for the end-user. Not to mention K3s is a production-ready product, I think DigitalOcean use it as their main Kubernetes distribution.
At the end of the day, we’ve seen in our stats and general community feedback that there are very few users (Single percentage wise) that use K3s in any meaningful way on TrueNAS. The vast majority are using “Apps” in the sense that they just want to click through and install their favorite 3rd party software packages on their NAS with minimal fuss.
Since almost day #1 of when we announced SCALE with K3s the number #1 feedback has been “We wish you guys had used Docker, it was easier and had the right amount of flexibility for setting up my home-lab.”. K3s has also been the number #1 source of issues and user frustration since.
We took that feedback to heart, and so far the response to this announcement has been overwhelmingly positive across our social platforms. At the end of the day by using Docker, we will be giving users the flexibility to run as simple, or as complex as they like, even enabling tools such as Portainer or Dockge to be deployed as management engines as well. And in a far lighter-weight manner, since K3s was a bit of a pig (Compared to Docker) on the resource side.
In the end though, this is one of those topics where lots of folks have lots of different opinions and wildly divergent use-cases. By embracing Docker/Compose/JailMaker/KVM we think there are enough tools for home-lab types to build whatever type of bespoke solution they desire. If you didn’t like K3s you can run K8s proper in a Jail as well, or Podman, or LXC, or Incus, or Nomad, or whatever the next hotness ends up being. 
I personally think IX will regret this decision, eventually, and I’m sure other devs share my opinion internally. While I understand you like to offer Docker as alternative to certain users, I do not understand why you want to alienate the users who already use K3s since several Scale releases by transforming it into an unsupported feature, it is bad software development practice to abruptly change direction. Also, I strongly believe people who prefer Docker use Scale as a little hobby side project, with little knowledge what Kubernetes offers, as robust implementation.
Perfect example, AWS offers EKS and ECS, you don’t see anyone using today ECS. These times are long gone.
Docker compose is more suitable for the target which is simple apps for simple uses.
Industrial users surely wouldn’t use “apps” or even whatever is bundled to run the apps.
In that case a more robust/standard k3s or k8s environment would be called for surely?
And maybe that can be delivered in a maintainable way in either a sandbox or a vm, or the more usual separate node.
The only part I have difficulty digesting is turning K3s into an unsupported feature, out of the blue. Really not the right approach. I have no problem with IX implementing Docker for little gimmick usage, but please continue to support K3s, as it was done until now.
It’s currently supported until the next major release, and the currently supported version does not expire at that point.
So when it becomes unsupported it won’t be out of the blue.
This gives time for migration plans.
