I’ll admit that isn’t what I’d expected.
I hope you’re not including me in that “docker zealot” grouping… and I do NOT believe that raw docker-compose editing is a replacement for point-and-click “easy mode” apps.
2 Likes
Holy moly this is fantastic news! I think this will be a very positive step for TrueNAS. Also to be supporting compose natively is a real win leaps you past what is capable in competing products. This is fantastic!!! Super excited, I will absolutely be on board for beta testing.
Well done IX, this is incredible news!
4 Likes
I truly believe this to be a wrong decision, from a business perspective.
I am a happy TrueNAS SCALE user and I am running many ZFS pools, several VMs and 100+ apps with TrueCharts. I am not particularly concerned with the change outlined above, since everything aside apps will stay the same, and SCALE UI for apps in its current state is not necessarily a plus.
At the same time I strongly feel this to be wrong move. I’ve been using SCALE since it was in BETA and I can say it was a rocky road until Dragonfish. But now? I am pretty amazed at how stable the whole solution is: when I boot up the server I have something north of 150 pods spinning up and a couple VMs, and to think that everything works without a hitch all the time is testament that we are now reaping the benefits of going down this path. SCALE has matured pretty well, and minor issues aside, I started to like it more and more recently.
And now that we are starting to see the rewards, you really want to undo everything, and go back to a questionable and imo inferior solution?
Why do I say questionable?
- as already mentioned in the comments, having a universal UI for any docker compose is not going to happen, or at least it’s going to be a moonshot
- there is already mention of static hardcoded .yaml files as the main path for app deployments (to reduce breaking changes when updating), basically urging for standardization
- in fact, in the comments there are already calls for standardization of docker-compose .yaml files, pretty ironic since this is what Helm charts do
Also, I am pretty baffled by all the people saying that Docker is inherently better, because imo it really isn’t
- management hell for docker-compose .yaml file: you think it’s going to be heaven, but you know that managing all these different .yaml files is going to be a pain, especially once you start customizing them
- unified methods for app automation (updates, backups, etc…) with k3s, learn commands once and use everywhere.
- with Helm/k3s you get out of the box addons like ingress, cert management, split DNS, whole system restore, per-app backup and restore on S3, metrics, VPN addon, automatic Postgres support, Homepage integration and possibly even more. Yes, these are all exclusive to TrueCharts, but it goes to show that having such approach is superior, since now in Docker you need to deal with all the above per app through copy paste mainly. Imagine the smell.
Now, why do I say it’s an inherently wrong move, not even a bad move, but simply wrong: do you think in the future people will want more standardization, automation and ease of use for their apps, or less?
Even if Docker is a wild success on Electric Eel:
- users will start sharing “optimal .yaml files”, or "best configuration’, everything will be based on copy paste
- documentation will be hidden in some hacks posted somewhere
- backups and restore setups will be per app, databases will possibly be fragmented and different for every copy pasted docker-compose file
- the official catalog won’t be suitable for many use cases
Not only this, but security is another issue that is overlooked here: by adhering to best practices and using conservative approaches, security gained in the Helm approach is often security gained for every app. Are the apps running as root? Do the containers have vulnerabilities? Are permissions correct?
Now extend all of these considerations in the future for the next 5 years. Do you see a flourishing app ecosystem with this move? Is this platform going to serve you and your users well?
All the problems mentioned above is stuff that can be automated or conformed to best practices. And this is what I believe people want now and will surely want more of in the future.
Containerization, distribution, scanning and deployment of apps are all tasks that will be highly automated (and already are) and the benefits on the long term are obvious.
I planned to move my apps off SCALE k3s since quite some time, so again this is not going to change much for me, but at the same time I want to say that I believe this is a genuinely wrong move. Helm + k3s/k8s is the superior approach now and in the future, and we are starting to see the results now so I am baffled by this sudden change after IX-System poured so much work in it.
I appeal to all the cool headed people in IX-System: reconsider this. I believe this to be an emotional decision, not based on a rational SWOT analysis or similar: instead you should double down on supporting k3s/k8s, give users more freedom when using it and develop new cool UIs for apps and kubernetes stuff.
Don’t make SCALE the new CORE in 2024. Say no to jails and to the lazy Docker way 
By the way, whatever happen I wish the best for all the parties involved, hope you keep on shipping awesome open-source software 
2 Likes
A lot of your concerns about “copy and pasting” seem kinda unfounded, considering many app developers who ship Docker containers have a recommended/starter Compose/Run file anyway, with a notes section about what you can customize in the compose file. Not to mention, Docker has been around for a very very long time at this point, and none of what you mentioned has really ever been an issue.
1 Like
I think the plan is to keep Apps, and use something like templated docker compose in the backend: “Current plan is to keep a lot of the Apps UI intact with regard to how we browse for Apps in the catalog.”
This sounds simpler than Helms, and I think much more people will be able to contribute to the community repository.
I think most of popular containers will be available to be set up from a GUI as an app very quickly, with an additional benefit of an easier deployment of more custom stuff using a compose file.
I have a system based on a Supermicro A2SDi-4C-HLN4F, which is basically a TrueNAS Mini X in a different case, and the overhead of k3s is significant, this is a very welcome change for me.
I’ve been using all my own custom “docker” containers for a year or so now, they are called custom apps. It appears the EE release would give me compose capability as well, probably not terribly useful for me as I have everything already set up without it. I’d still want the UI for specifying my container parms, hostpaths, network, etc. so hopefully that is staying around as I find it pretty simple.
My kubernetes cpu overhead is under 1% so not a big deal on my hardware, perfectly fine as is. But I suppose I get it, the complaints have been many here. Certainly many will appreciate not having to “deal” with k3s any more. I guess I am surprised by this somewhat but probably not a big deal for me at least.
History suggests that the “power users” who can set up and properly maintain their own jails are a minority. For everyone else (that is: most users), containers and pre-build compose files for $WHATEVER_SOFTWARE looks like a fair solution.
2 Likes
Thanks for your feedback @pinoli
I think, inadvertently, you captured the dilemma we see in the market. Larger scale K8s users like yourself do move their Apps off TrueNAS and prefer to use K8s via its native APIs and then TrueNAS via the CSI driver.
Smaller scale users prefer the simplicity of Docker and Compose. The extra power and configurability of K8s turns into more configuration issues.
There is no simple way of running both with Apps. Networking is different.
So, our decision embraces both technologies:
K8s is still enabled via Sandboxes. It can be updated, clustered and configured free from any TrueNAS constraints. K8s users should make requests for better sandboxes.
Docker and Compose becomes the Apps vehicle with simpler configuration and UI. There’s still lots to do, but most of the stability will be inherited (after a couple of updates).
In the meantime, we will keep Dragonfish available to allow transitions only when users are ready for it.
9 Likes
So are you saying that ixSystems actually have a lengthy history of not listening to the community and that they have not learned from this history and so continue to make poor quality technical decisions?
I understand that Docker Compose is easier to use for most (homelab) users, but you lose out on a LOT of capabilities.
To name a few: I’m using ArgoCD to automatically deploy my apps from Git, ESO to load my secrets, Loki, Grafana, Alert Manager for monitoring and alerting, cert-manager with let’s encrypt, Trivy Operator for vulnerability scanning.
I hope k3s remains a first class citizen or at-least will remain easy to deploy.
I think it was @CheeryFlame who was saying that, not me. As to whether I agree with it, well…
Having users maintain their own k3s setups is the 2018 version of how kubernetes should be done.
But at the very least its not as solid as the current great unmutable solutions like TalosOS or even dedicated systems like Harvester.
So are you saying that ixSystems actually have a lengthy history of not listening to the community and that they have not learned from this history and so continue to make poor quality technical decisions?
The way they actively rephrased “we also want docker(-compose) support” as stated by the community, into “Lets nuke third party app catalogs next version already YOLO”, is pretty clear.
Literally everyone agreed that having docker-compose added would be a huge benefit to the community. Even we publicly and vocally agreed with that.
The statement that, somehow, the community wanted everything completely redone using docker-compose, buking third party catalog(s)… Is weird and crinchworthy.
The community will still end-up with the same “locked down” Apps, regardless of backend. But loose access to TrueCharts as a “price”, for their “zero gain” of a changed backend.
This is a loose-loose choice by iX-Systems.
5 Likes
So, in effect you will have a UI which then creates a static Docker composer.yaml file? And apps will have a way to define their configuration UI (like current K3s apps can) and apps will NOT start with a static compser.yaml file?
(If this is the case, it sounds great.)
I can readily believe that there might be technical challenges with running K3s and Docker/Composer alongside one another.
But I agree that literally dropping K3s and introducing Docker / Composer in a single release - thus making life difficult for all TC users - is a very bad move if it was avoidable.
Here is how I see my migration options as a user who has a small mix of TN and TC apps. I effectively have four migration choices with the current EE announcement:
-
I can wait for a TC technology that runs in a VM. This is NOT a solution to me as it will likely significantly increase the existing overheads.
-
I can upgrade to Dragonfish, install K3s in a Sandbox and manually migrate all my TC apps one-by-one from TC to K3S in the Sandbox, losing update capabilities. At a later date, I might then need to migrate again to an ElectricEel Docker solution. This is a massive amount of work x2, with a significant learning curve x2.
-
I can upgrade to Dragonfish, install Docker in a Sandbox and manually migrate all my TC apps one-by-one from TC to Docker in the Sandbox, losing update capabilities. At a later date, I would need to migrate again to an ElectricEel Docker solution. This is a massive amount of work x2, with a lower but still significant learning curve (because Docker is less complex than K3S) x2.
-
Fully document my Cobia / Dragonfish TC configurations (with screenshots or other means), upgrade to ElectricEel immediately losing access to all my TC apps, and then re-implement them as fast as I can using the EE Docker solution. Because at present there are no plans for a Community Apps Catalogue, we will be back to researching and manually installing all the TC apps. So this will also be a significant amount of work, and will result in a period of time when apps are not available.
None of these options is a win for me. They all involve unpalatable migrations, and none except the TC in a VM one continues to have a broad-based community app catalogue.
So, although I think the Docker/Compose direction is a good one, I think that the implementation has been poorly thought out, without asking the community about the impact it will have.
More worryingly, this appears to becoming a habitual behaviour - e.g. see also the lack of communication and then mis-information about the reporting changes in Cobia.
I think I speak for many people here when I say that I very much value the benefits I get from being able to use a great product like TrueNAS for free, and I am thus very supportive generally of what they are doing. But that doesn’t mean that iX is always perfect, and there is scope for improvement.
IMO ixSystems (@kris) need to start thinking about how they can engage with the community better and to work more closely with this community to plan solutions that work for both iX and the community.
3 Likes
- I can wait for a TC technology that runs in a VM. This is NOT a solution to me as it will likely significantly increase the existing overheads.
The overhead will not be too significant.
Maybe even less in some ways, as the middleware currently has so many significant issues leading to it hogging CPU and RAM, that a good vm-based solution might even be faster in some cases.
1 Like
Upon closer reading, explaining that Docker and k3s networking are incompatible implies that Docker will be in the base OS (which might be another issue if one is concerned about the “blast radius” of root-level docker) in place of k3s, and that there will be no sandbox by default.
Exposing jailmaker in the GUI is reportedly in the plans, but it may come with Electric Eel or later.
1 Like
This does sound like a return (for me) to having to do storage over NFS instead of native filesystem access for the containers. 
The problem with a VM is that it hogs a big chunk of RAM which can be an issue on smaller systems - and if there’s free RAM in the VM it can’t be used by anything outside of it.
3 Likes
Our current goal is to offer multiple options to suit multiple usecases of users.
4 Likes