Migrating from TrueCharts

TrueNAS General
,
102

To avoid any doubt, whilst I am generally supportive of iX’s long-term efforts and the resulting great TrueNAS software, I am on open public record as being critical about:

  • iX 's decision to migrate direct from Helm to Docker without any overlap; and
  • of them encouraging 3rd party Helm catalogues and only then to dump them without notice; and
  • their poor communication about the reasons for their decisions.

A few comments on your response:

Most people who were using the TrueCharts catalogue are probably not on Discord at all, or not on the TrueCharts Discord server or if they are then only in case they need support and are not active watchers/readers. From my perspective, and with the benefit of hindsight, Discord was probably not the best route to communicate your reasoning at the time.

I am on Discord for many many software products in case I need support, but I don’t have time to follow or read any of them.

Apologies - I thought I had read in your previous post that you had had some conversations with iX, but it appears that I was mistaken.

As you say, you cannot explain conversations which never happened. And the fact that they never happened was down to iX - who were intending to drop 3rd party catalogues and were therefore responsible for both ensuring that they understood the consequences and for playing nice with those 3rd parties who invested in supporting TrueNAS’ 3rd Party catalogue request.

Understandable. As is a decision to stop investing any more time in keeping existing apps updated or including new ones.

However (with the benefit of hindsight) there were other ways to discourage users from installing TrueCharts unless absolutely necessary and informed about the future issues such as adding a warning to each and every catalogue entry for TrueNAS Scale users to see.

Or if you were going to remove the catalogue entries, why not leave an empty catalogue rather than have thousands of users worldwide receive a completely unexpected error about a catalogue not being downloadable.

Put simply, IMO there were more user friendly, and nicer, ways to pull the plug that the one you took.

Perhaps I am also at fault here for not looking at Discord and communicating with you there once things started to go belly up for us users.

103

Its also explained on reddit a few times and it might even be referenced in 1 or 2 news articles.

It’s our primary means of communicating with our community.
Though it was also posted on multiple sources, but at least very many times on the discord.

On that topic, yes… One of the primary reasons our core-team (and actually most of the staff) is done with iX-Systems and TrueNAS, is the fact they knew in februari this was coming. And purposefully kept it silent while we spend hundreds of hours on the data/pvc/openebs migration for DragonFish.

We don’t blame or even dislike the move to docker, just the way its handled.
A good Shepard would’ve given notice.

Its not that easy, that would’ve meant significant modifications to our CI pipeline and we where not going to invest any effort into SCALE at that point any more.

Also, our numbers indicate that people, in all honestly, dont read jackshit.
So it would’ve meant complicated rewrites to change App display names separately from their names, which we don’t actually had any CI for anyway.

Simply put, this was not an easy solution.

Honestly, we assumed the system would’ve handled that more cleanly. But yes, an empty catalog would indeed have been a cleaner first-step.

No judgement on that, we cannot keep track of every community around every software product. It’s indeed the case that we’ve decided to not be super active in other associated communities much anymore and focus on our own.

104

The decision to move to Docker was entirely based on community requests. There was a desire for Docker Compose simplicity and for greater reliability/support of Apps. Loosely coupled 3rd party Apps were getting many complaints for complexity, reliability and support challenges. it was the least favorite part of TrueNAS and the most common request to fix. It was negatively impacting TrueNAS reputation and business.

We used a 2 step parallel running process:

  1. We supported jailmaker and nspawn containers (sandboxes) in Cobia. This allowed docker compose apps to run in parallel with Helm charts in Cobia and then Dragonfish. After these were successful, we made the decision to switch in Electric Eel to Docker Compose and alerted the Community well in advance. Dragonfish could run with Helm Charts for years as has been proven by 13.0.

  2. In Electric Eel we also decided to migrate the Helm Charts Apps we controlled to Docker Compose Apps automatically (after much sweat, that is working). We also committed to running Kubernetes within the nspawn containers. This is still possible. 3rd party catalogs could have used this to support existing Helm Charts.

3rd party catalogs had a choice, but we don’t control them. TC had been very adamantly against Docker Compose as a native capability. We both knew that mixing Docker Compose and Helm Charts was not feasible due to incompatibilities. They had to be treated as separate islands. Most users would have to choose one or the other. We were Ok with that approach. TC decided, as was their right, to abandon Dragonfish early. Admittedly, we were surprised by that since things were generally stable.

This was not an easy decision or migration process. The proof will be in the customer/community satisfaction at the end of the Electric Eel release process. We will see the results every day in Electric Eel uptake. There is still work to do, but so far we are on schedule and the quality is frankly better than expected.

There are already more systems using Docker Compose Apps in Electric Eel than systems using 3rd party Helm Charts in Dragonfish. The adoption rate of Electric Eel is 3X higher than previously, even with the teething issues of migrating Helm Charts to Docker Compose. We’ll keep on polishing. By January, the Community will be well placed to pass judgement on our efforts.

Through this process, TrueNAS growth has not slowed and has accelerated in the last month. We hope that continues. More than anything, we hope that deployment, migration and support issues for Apps become much less troublesome for the Community.

3 Likes
105

I should frame that and put it on my wall :sweat_smile:

2 Likes
106

I am not questioning the decision to move to Docker. I am questioning whether the implementation route and the communications were as sympathetic as they could have been.

  1. Since you admit that it is possible to run Kubernetes within an nspawn container, and it is also feasible to run Docker within an nspawn container, that means that is WAS feasible to run both the old Helm-based UI-driven apps infrastructure alongside the new Docker-based UI-driven apps infrastructure.

  2. A two step migration (Helm UI predefined apps-> sandbox CLI manually defined apps → Docker UI predefined apps) is much much much more complex and sudden-death than parallel running of Helm and Docker both with a UI. It is only for those users with suitable skills and the time to spare - and completely infeasible for non-technical users who installed TrueCharts Apps in good faith on the basis that iX had encouraged 3rd parties to provide alternative catalogues.

  1. iX provided infrastructure that allowed them to publish a Helm-based catalogue and then you pulled the plug. What exactly was their choice?

  2. iX encouraged 3rd parties to invest time and effort into creating a Helm-based catalogue because it was in iX’s interests to have a much wider base of apps which would encourage TrueNAS take-up, and after TrueCharts had invested that time and effort and given iX the benefit of a broader apps catalogue, you pulled the plug without providing either TrueCharts or their users with any realistic migration path (by which I mean an iX provided easy non-technical UI-based non-sudden-death-cutover migration path). iX could have supported parallel running for a major release or two (yes - there would have been a bit more engineering effort), but you decided not to.

Had you provided parallel running for a major release or two, then:

  • Migrations could have been done one-by-one whilst the server was in production, rather than as a sudden-death cut-over during the version upgrade to EE.
  • TrueCharts (if they wanted to - and if not an alternative Community effort) would have had time to use the same migration functionality you created for the iX Chart apps to allow for migration of TrueCharts apps to Docker.

This is NOT a reasonable comparison. Docker apps in EE include iX apps, 3rd party Helm Charts is just TrueCharts.

The users who have already migrated are likely to be at the more technical end of the user spectrum, and they are well placed to be able to achieve this smoothly.

But (as I previously warned would happen - but iX new better) we are already seeing examples of users who are upgrading and losing their apps, either because they were not aware that non-iX apps would simply disappear or because the iX-apps migration didn’t work for them e.g.:

So please let’s not crow about how great the EE take up has been when there are users whose production services have been disrupted by an incomplete and apparently flaky app migration process that resulted from a decision by iX to do a sudden-death cut-over.

Tell that to the non-technical (and admittedly perhaps somewhat naive) users who upgraded and lost their apps and now have to flounder around to try to get things working again. I suspect that they have already passed judgement on iX’s efforts.

107

It might be best to focus on your own company and development process.

In short:

  • Kubernetes and docker-compose aren’t inherentlyincompatible, supporting both at the same time just wasn’t worth the additional development effort (rightfully so) as confirmed by @kris on reddit multiple times
  • We are not and never where against docker compose native capability. We actually think it is much superior than “custom-app” without YAML.
  • There are more options than systemd-spawn, we picked a TalosVM and our own custom (to be released) ClusterTool
  • We dont have endless resources to keep pouring them into a, quite frankly, dead-end OS.
108

I think we can safely say, that removing third-party catalog support with a 6 months notice, after people poured thousands of hours into it, is something we did take into consideration when deciding on if we would continue supporting DragonFish.

The conclusion among our staff, was that it seems like iX was not acting in good faith. True or not, thats not the basis on which we wanted to continue with a platform.

To be fair, we all only share the good numbers.

More likely, not even technical users, is that people downloaded Electric Eel especially try test/try the new Apps features based on docker-compose.

Regardless of weither they would be continuously using/trying it.

For example most of our staff also did this, but none of us are really keen to keep relying on the SCALE Apps system either

To be fair to iX here, TrueCharts itself was also not yet (and is not even) considered “production ready”. At the current state we would say the common chart is “production ready” but the derived charts are considered "home-production or homelab only)

This is not fair towards iX-Systems nor their staff.
People are always quick to blame projects for their mistakes.
The less technical even more so.

Bugs and mistakes happen.
With us, with iX, with every it project.

2 Likes
109

To be fair, no one made them update to a .0 release practically the day it was put out. The advice I have is the same I always have, wait for the .2 release. Everyone wants every update to anything instantly. To their own peril. It is not surprising at all. I’m still on MacOS Sonoma, gee, I don’t have the bugs in Sequoia. I’m still on IOS 17. etc. etc

I am not agreeing or disagreeing with the rest of your post, I am staying out of it, lol.

1 Like
110

…or in many cases well before. For better or for worse, the siren song of Docker appears to be very strong.

1 Like
111

Exactly.

There is a difference between sharing the good numbers and making unsuitable comparisons in order to make your numbers look better than they are and especially so if you are doing so in order to deflect nuanced criticism.

Exactly - EE users are mostly new users, or existing users with good technical skills - the home / small business users without good technical skills have mostly yet to upgrade, and when they do we will have to see just how big a mess results.

I am using the term “production services” to mean people using their TrueNAS for live usage rather than purely for testing.

I was not using it to refer to being production ready for large scale deployment in enterprise Kubernetes clusters. I am not sure that this is a target market for iX because from my career in Enterprise IT I would say that Enterprises would most likely create a separate Kubernetes cluster environment on bespoke hardware rather than using a TrueNAS server - Enterprises tend to spin up a virtual O/S instance for each infrastructure service rather than e.g. combine NAS and Kubernetes clusters on a single box. The exceptions to this would be storage networking services which (unlike e.g. SMB or NFS) are not provided natively by TrueNAS e.g. Syncthing where is specifically makes sense to have the app near the data - but for this you don’t want Kubernetes clustering anyway.

Is it really unfair? TrueNAS is actively targeting non-technical users who were sold the idea of using TrueNAS because of the app catalogues including 3rd party apps.

To be fair to these users, then if iX decided it was going to put users through a sudden-death cut-over, I would expect:

  • A general warning about app migrations when offering an upgrade from Dragonfish or earlier to EE or later.
  • A pre-upgrade sanity check to confirm which apps will and will not be migrated automatically, and if there are some which won’t then an “Are you sure message?” with the list of apps that won’t and a link to a web page explaining what they will need to do.

Those of us who have had many decades in IT know better. The less experienced users simply trust and expect iX to provide a system which (from their perspective as non-technical users) doesn’t break.

Well more and more non-technical users get offered the EE upgrade now that the .0 version is going to start to be offered - and perhaps many of them have successfully upgraded from Bluefin to Cobia to Dragonfish on the .0 versions of each of these. And so IMO it is likely that many of these users will trust that the Dragonfish to EE upgrade will be equally problem free and go for it without realising the consequences.

So, my hunch is that we are going to see an increasing number of people saying that their upgrade app migration failed and their system is now broken and looking for help on how to fix it - at least until the word gets around widely enough about how this upgrade may break your system for them to start being more cautious.

112

I agree with that for sure, which I why every chance I get in the hopes some of them read it, I post about waiting for .2 release! You would think they would learn their lesson. When IOS breaks something, when Android breaks something, when Emby/Plex breaks something, when Nextcloud breaks something, when Windows breaks something, when Debian .0 release breaks something,. etc. etc. etc. When do you finally say, gee, I don’t need every update instantly? To answer my own question, apparently, never! I don’t take every docker container update when it comes out, no interest at all in doing so. I’ll update some specific app like Emby or Nextcloud some day, maybe 6 months from now when I am confident the bugs are out. I don’t update ANY of my apps without thought. And I certainly would never ever allow auto updates.

I would wager many of those users who updated to Eel are users who updated to Dragonfish .0 and had issues, lol.

Honestly, I think while IX maintains the software status page which gives some info, when they post we are proud to announce the availability of Electric Eel everywhere, it should include a disclaimer that there may be some issues with the .0 release and people who rely on absolute uptime should consider waiting or whatever good wording is. I am sure many would ignore, however, at least one could say did you read the release notes before updating. But I bet some might actually wait too.

113

Let’s put it this way…

As an Enterprise IT Manager (now retired), if I had designed an which was sudden-death cutover when it could have been parallel running, then if the upgrade went wrong for more than a handful of users I would have been dragged over the coals for putting the business at risk.

114

Speaking of which…

Setting up a Talos VM (whether on TrueNAS or elsewhere) is, of course, easy enough, as is setting it up on bare metal for those who might prefer that COA (course of action). Initializing the cluster using clustertool is likewise pretty straightforward, and your docs cover that pretty well. That gives a single-node (by default) cluster with a variety of services installed including Kubernetes Dashboard (but no longer, apparently, KubeApps).

Your docs likewise cover migrating the apps and their data, though I haven’t tested those steps as yet. But the result of that would presumably be all the TrueCharts apps running on the new Talos VM, with all their data being present via either Volsync or the NFS mounts they were using before.

Great, now what? What does a TrueNAS user need to learn about Kubernetes to manage this new system? To install and upgrade new apps? Or to change their configuration? And what resources would you recommend such a user consult to learn these things?

I’d like to continue using the TrueCharts catalog–everything from that catalog that I’ve used has been (at least as far as I can tell) well thought-out and featureful. I know my way around a *nix command line, but know very little about Kubernetes. I’m willing to learn, but some pointers would be helpful here.

Discussion on this should probably continue elsewhere, and I expect that your docs aren’t your top priority right now (though I see you’re working on them as well). But thought it was worth raising the question here.

115

It seems to be unclear whether users having app migration failures during their EE upgrade are having difficulties with iX apps or TrueCharts apps, but here are more users having migration issues:

If any further examples arise, I will add them to this post rather than creating a new one.

Edit 4 Nov:

Edit 5 Nov:

Edit 7 Nov:

Edit 8 Nov:

Edit 9 Nov:

Edit: 11 Nov

Edit 12 Nov:

Edit 13 Nov:

Edit 14 Nov:

Edit 15 Nov:

Edit 16 Nov:

Edit 18 Nov:

116

I know at least 2 users reasons, they had their apps on encrypted pools which the update notes say it won’t migrate. Other ones, bugs I presume at least some of them. Some may be app updates really, the latest version has new requirements say and they hadn’t updated in a while (another thing one should always do before updating systems).

Regarding your enterprise example, this is not how it would have been done anyway in that environment so not a fair comparison. One would have had a test system presumably. Which I did for Eel (though have not updated of course) as a dragonfish VM.

117

Whilst @sfatula and I would naturally check the release notes before upgrading - especially since there is an app migration involved as is the case with the EE upgrade - not all TrueNAS users are as tech savvy as us.

This is what the upgrade looks like (from Dragonfish 24.04.2):

  1. Firstly I am by default on the 24.04 upgrade train:

    image
    image1120×627 25.3 KB

    So I have to explicitly switch to the ElectricEel train which I need to explicitly confirm:
    image

  2. Here is the EE Upgrade page.

    image
    image1118×635 24.9 KB

  3. The EE specific release notes page linked to does have quite a lot of detail about how to prepare for the EE upgrade.

    image
    image904×942 43.8 KB

    I haven’t done an upgrade myself, but this list does at first glance look reasonably comprehensive. I would say that any user who actually clicks on this link and actually reads the page will have sufficient warning of the difficulties they might experience - in other words I think that iX has done a pretty good job in this respect.

    If there is one omission from these notes it is the absence of any mention of 3rd party catalogues [EDIT: I was incorrect about this - see below] as distinct from “official catalogs” (note the plural):

    All applications available from official catalogs in 24.04 are available to install in 24.10. Supported catalog applications automatically migrate to Docker deployments on upgrade from from 24.04 (Dragonfish) to Electric Eel.

    Is a relatively inexperienced user really going to be able to know that the TrueCharts catalogue they added a couple of years ago isn’t an “official catalog” - or are they going to read this and assume that it means all the apps they have installed. I would prefer to see - and indeed really expect to see - a much more explicit warning that apps installed from 3rd party catalogues will not be migrated and will not work after the upgrade.

    EDIT: It has quite rightly been pointed out in post 131 below that this screen shot clearly shows a table section on Third Party Applications - so genuine and sincere apologies for my being completely wrong about this - and an extra sentence has also just been added to the introductory paragraph I quoted above to draw more attention to this table section.

  4. However, if the user doesn’t click the link and read the page and instead clicks to upgrade without undertaking the pre-requisite actions, then I fear that they will not receive any other warnings and will find out that some of their apps haven’t migrated only after the upgrade completes when they find those apps are no longer installed under docker - or installed and inexplicably broken.

118

My main complaint is on social media platforms and even here, reddit, etc, IX promotes the .0 version as in install it now. I don’t think they should. They should of course announce it, but, hyping it without some type of disclaimer to me is counterproductive, esp for existing users i.e. upgraders. It’s the casual non tech update user I am concerned about.

1 Like
119

In my eyes (friendly–even though it may not read that way–reminder):
I’ve been labeled a troll so I’m going to try and step as lite as possible but also try to respect your time (not mince words much). My TN server got me through the “pandemic-home schooling” (my oldest was learning to read–first grade) and I spun up a FAMP, with an M4 generated website (to me, then, my TN Mini wasn’t just for apps). That FAMP jail ran for a while and I’m sure I probably even updated TN during that time (and probably didn’t read the release notes either). …Roll back, sure but not the point.

Was that “M4 and decimated javascript lib website” good enough for most power users, probably not but part of my point behind choosing M4 was because I typed simple text files, and I was done (M4 did the rest) because, I had more important things to do then screw around looking at config files. “I am root” (or was and will be again soon) so, at some point during config nightmare situations, I’ll just cut the damn thing’s head off but at the same time that’s a luxury and effort I just don’t want to spend–because even though ‘I am root’ I’d rather spend time with my kids.

NOTE: This is a vague note to all, not Protopia. I was adding to. Not contradicting.

120

Exactly!!

No offense meant and none taken. Though I’m afraid I have no idea what you are talking about or its relevance to this thread and TrueCharts.

121

Tech savvyness is not an impossibility.

I myself, think of making a VM and run everything there. No matter what iX does, as long as VMs are there, the new… whatever, will not brake my setup. ( but I will not take that route )

Am about to run everything with Dockeg. That should take care of everything.

My knowledge of … anything here is nill. Nada.
Wish me luck :slight_smile:

1 Like