This was a very poorly implemented change to virtualization. The upgrade should check for configured or running VMs and warn the user that they should NOT upgrade until the virtualization overhaul and migration process has been addressed in future releases.
The notion that someone “should have read the release notes” is completely unreasonable. Yes, some people might read them, but no one expects an entire functioning portion of the platform to completely disappear during a routine update.
Implementing changes in this manner will cause users to leave the platform and find better solutions for their needs. TrueNAS is a good storage solution, but decisions like this will undermine the good faith and trust that the community has put in the product. There are thousands of people upgrading and then frantically searching for “where did my FreeNAS 25 VMs go?” or “Why is my FreeNAS 25 VM broken?”
Yes iX messed this up. Questionable decision to pull out a production grade virtualisation platform, replace it with something experimental (yet again), and push a new general release. The fallout/damage then made much worse by communications failure combined with complete lack of guidance in the upgrade process itself.
I think it further illustrates a few things:
Product strategy? see further above.
When iX says ”community edition” they really mean it. The 6 month release cadence seems to be a priority, and if that means breaking changes mid-cycle, then so be it. Upgrade at your own discretion. A clear change in attitude compared to some time back and the Core/FreeNAS days.
Conclusions?
Certainly read release notes carefully and specifically look out for breaking changes before upgrading anything in the future…
Keep config as vanilla as possible to ensure you remain on iX’s radar screen wrg their own pre-release testing etc
TrueNAS is not anywhere near ready to be used as virtualisation platform yet, YMMV…
Personally I use other hypervisor platforms for virtualisation so am not affected by this VM stuff. But if I see signs that iX would take a similar attitude to the more core NAS parts of the platform, putting my data at risk between upgrades, I’ll be off the platform faster than green grass through a goose.
If we look at the number of forum posts from people who come here for the first time to ask for help after their version upgrade went wrong, then this is undoubtedly true.
I have a lot of respect for iX and their key staff for a lot of reasons, and they demonstrate a lot of strengths in what they do, however …
They are IMO letting themselves (and non-Enterprise users) down by several core principles that repeatedly do severe damage to their reputation version-after-version:
Technical
The iX team are very technically competent. It is obvious to everyone that adding warnings to the TrueNAS UI about breaking changes would be easy-peasy for them. Indeed, adding functionality to check whether the user is using functionality with breaking changes would also be easy for them to do too.
So you have to wonder why the iX team don’t do this in order to safe-guard users from breaking their systems?
Indeed, the iX team is so technically competent that delivering parallel running of both obsolescent and new technologies e.g. Kubernetes and Docker (which they are on record as admitting was possible) or old-style Virtualisation alongside Incus (also likely possible since they both use KVM) would be achievable if they had the wish to do so.
Parallel running would turn a breaking-change release into a migration release - users would NOT have to combine e.g. the rework of a significant number of apps or VMs into their TrueNAS version upgrade, but instead could do the upgrade and then migrate their apps/VMs one-by-one afterwards. The next major version upgrade could then drop the parallel obsolete technology.
So you have to wonder why the iX team don’t make any attempt to do this in order to allow non-technical users safely to upgrade their systems and to allow technical users with complex environments to migrate more safely?
Beta testing
We need to distinguish between two distinct types of beta testing:
Explicit & Informed - When a user decides to install a nightly, beta release or release candidate, they are explicitly choosing to be a beta-tester and to accept the risks of trying out software that might create issues with their system. This is an entirely legitimate way for iX to benefit from the technically competent and adventurous part of the non-Enterprise TrueNAS community who enjoy beta testing and being part of the evolution of TrueNAS and who are capable of understanding the risks and coping with them when issues do arise.
Implicit & Naive - When a user naively upgrades to a new major version of TrueNAS that is not ready for risk-averse users without understanding the risks that brings, they are effectively becoming implicit beta testers. So when a non-technical user is prompted by TrueNAS to install an upgrade without a single warning or caveat being shown to them, they are being asked to assume risks to their system that the don’t know about and are not being warned about and to become implied beta testers without explicitly choosing to do so.
There is a world of difference between these two types of beta testing - IMO one is a partnership, the other exploitative.
Caveats & Communications
It is both natural and acceptable for the iX team to want to publicise all the good stuff they are doing - and they do create good stuff.
However, when the new functionality is a breaking change, IMO the rah-rah crowing needs to have some caveats attached to give even minimal advice to users about breaking changes and risks.
Adding brief caveats would be so easy to do.
But what we see is an almost absolute lack of willingness give notice of breaking changes and risks - in announcements, in release notes, in T3 vLogs, or in upgrade notices in the TrueNAS UI.
By avoiding giving even minimum notice, iX is effectively encouraging non-technical users to do something that will break their systems and / or to move to “experimental” functionality.
Just how difficult would it be to give warnings about breaking changes and risks in each and every communication? The lack of such warnings despite repeated requests from the community that they be included appears (rightly or wrongly) to be a deliberate choice by iX to increase the number of users testing new functionality without them making an informed choice to do so.
These users then end up with broken systems - but it seems to me to be unlikely that they give useful feedback to iX on the issues they encountered.
There is a reason that the oath in court includes “the whole truth”, and a reason that TV adverts are made to include caveats (even if they are read out at twice normal speed making them difficult to understand).
But such users do increase the upgrade statistics that iX can crow about, so perhaps this is the motivation - getting some nice headline stats. But if this is achieved at the cost of dissatisfaction, bad publicity and a loss of confidence and reputation, is this really a good trade-off?
Blame the user
To make things worse, iX then attempt to justify this by using the “personal responsibility” credo by blaming their users for not having hunted out the detailed web-pages that would have warned them not to do something that would break their systems.
Blaming the user is never a good look - but if there had been widespread caveats warning users and referring them directly to these web pages, then at least blaming the user would have some credibility.
Summary
I think it is a real shame that the iX folks don’t do more to make life easier and less risky for non-Enterprise users, and in particular non-technical users.
They are bright, intelligent, committed people with long histories of innovation and support for open source. They provide great functionality for free to a world-wide non-Enterprise community, some of whom are willing to give back in return by being beta testers or providing community support in these forums.
So, IMO they are letting not only the community down by failing to communicate fully about the caveats of new releases - they are letting themselves down (and perhaps only for the the shallowest reason of wanting to crow about upgrade rates).
At the risk of over-analysing, I won’t go further into speculation as to ulterior motives and such - could also be as simple as a tech-driven organisation building a bunch of tech and focussing more or the underlying tech in itself than the actual problem it is there to solve, for whom, and delivered how.
The release announcement referenced above being just one example but a very good one. While technically correct - and for someone who spends all their time inside the product and follows all its minor and major builds, from dailies to betas and RCs, probably entirely relevant. But as a broad announcement marking a new major production release, completely missing the mark. Why? Because it doesn’t understand who it’s written for, and what information those people need. So it comes across as actively misleading instead. I would opt for believing and certainly hoping that this isn’t the case.
It’s kind of interesting to me to compare the community reaction (measured by posts here–i.e., very unscientifically) to TC’s missteps vs. to iX’ missteps.
TC’s breaking changes:
In late 2022, TC announced that a refactor of their charts was coming, which would require some users to reinstall some charts. In fact it required every user to reinstall every chart, losing data unless they took affirmative steps to back it up. This was not communicated in a public forum until well after the fact. Some time later–after most users had presumably already gone through the manual process–they made a script available to facilitate the process.
Shortly after the release of 23.10, iX announced that they’d be removing a Kubernetes storage feature (that TC had been using, and iX hadn’t) with the release of 24.04. TC adjusted their charts accordingly and provided a script and pretty thorough docs to migrate users’ charts to the new system.[1]
The first was 100% their decision (leaving to the side whether it was a good one) and was communicated poorly, if at all, to their users. The second, while it was their decision, was driven by iX removing features from TrueNAS. It was, IMO, communicated as well as it could have been, and TC had a migration tool ready to go when 24.04 dropped.
And then there’s the coup de grace. Not long after iX announced that they were throwing k3s out the window, TC pulled their apps repo, and there was great weeping and gnashing of teeth (you’d think the TC team had kicked a dog or something). Whatever TC’s reasons for this were,[2] it didn’t stop their users from using their apps, and TC had already said they wouldn’t be updating them. The only thing it stopped users from doing[3] was installing new apps. But the response to this was off the charts. They “screwed their users.” They “stabbed them in the back.” And so forth.
The outcome? TC are so hated that someone parachutes into this thread to blame them for iX’ decision to migrate to Docker (never bothering to explain how that would be the case). We see emotive language like “breaking changes every few months” (which just didn’t happen) and “stabbing in the back.” Heck, one user was so deeply infected with TrueCharts Derangement Syndrome that he felt the need to start new threads at random, even well after the EE announcement, warning people, in the most hyperbolic language possible, against using them (I’m writing in the past tense because I haven’t seen anything from that user in quite some time, thankfully).
Compare that with iX. Of course, there’s the present fiasco that’s the subject of this thread. There’s the fact that they can’t or won’t commit to a competitive apps/plugins ecosystem (whether Apps 2.0 will be The One remains to be seen, but I’m doubtful). Related to that is the removal of k3s and breaking of any third-party apps with release of 24.10 (oh, and removing the possibility of any third-party apps repository). The recommendation for SCALE in new deployments since some time in 2023, even though it didn’t have feature parity with CORE until 25.04.[4]
And sure, iX is criticized–and vigorously so–for these missteps, but I don’t see nearly the level of vitriol for the for-profit company with the paid developers who’s breaking things far more badly, more frequently, and for much longer (I could cite examples going back 10 years or more) than TC ever did. And sure, they have a habit of closing threads when they get too critical of them, but I think I’d see it if it were happening. Curious.
It’s a near certainty that iX knew when they made this announcement that they’d be moving to Docker in 24.10, but they chose to remain silent and let TC spin their wheels doing pointless work on their charts. ↩︎
The spat on reddit didn’t look at all good for them. Wanting to prevent new installations of an unsupported system, which was what they said on their website, makes quite a bit of sense. Certainly plenty of people saw it as TC picking up their marbles and going home. ↩︎
and that not very effectively; clones of the repo popped up pretty quickly ↩︎
It still doesn’t, really. OpenVPN client and server are gone, never to return (no great loss in my book; they were an undocumented nightmare to configure). WebDAV and Minio are gone from the services as well, but at least they have apps to replace them. Jails are–kind of–in 25.04, but “experimental.” ↩︎
I would also like to do the same, but given the number of times the community has raised this, and given the illogically defensive comments previously made, I find it difficult to see how they could be naively unaware of these things - and thus is makes more sense for them to be aware and to have actively decided to continue to avoid giving warnings for whatever reasons.
I agree - iX invited TC to create a 3rd party catalogue, benefitted from TC providing a whole bunch of apps that iX weren’t prepared to invest time in creating and maintaining, and then iX unilaterally decided to dump both Helm Charts and 3rd Party Catalogues without any regard to the time and effort that TC had invested and the benefits to iX that had accrued.
Yes - TC’s response was a bit childish and hurt TrueNAS users rather than iX, but they were definitely provoked.
More precisely (unless there were relevant communications I’m not aware of), iX allowed 3d-party catalogs in TrueNAS. TC took advantage of that feature. I’m not aware of any communications between iX and TC asking or inviting the latter to do this–though we agree iX benefitted greatly from having a large app catalog. Slightly less bad for iX, IMO, than if those communications had taken place.
More precisely, iX made a general invitation for 3rd parties to create catalogues (because having more apps was in iX’s interests and benefit) and TC took up this general invitation. So it was IMO still an invitation even if it didn’t involve any communications or negotiations between iX and TC.
Edit: on the comments directly above, and from this blog post (which didn’t age so well) - it was clearly a collaborative effort and with a commercial element too (iX paid for some of the development).
I either hadn’t seen, or had forgotten, that post–thanks for finding it. Sadly not much of a surprise, though. Anyone remember iX’ partnership with Nextcloud? Asigra?
I also upgraded to 25.04 from 24.x. I also lost my virtual machine but I don’t mind.
I have tried to create a new instance, I added a IP address and clicked on save button and the system stopped working. It can boot I see the console screen on hdmi and keyboard works but any apps or web ui are unaccessible.
Can I remove somehow the wrong Instance with Linux shell or Truenas CLI?
