I meanâŚ
The idea to use Incus for VM management sounded reasonable, considering that Incus was to be used for managing LXC âsandboxesâ. To rush, because of the hasty 6 month release schedule, introduce an admittedly incomplete and unstable implementation of Incus management AND remove the previous, stable and trusted, management systemâall while denying that this is a regressionâis far less reasonable, and does not foster confidence.
iX heard the backlash and will restore the old system⌠and freeze the new system. Which then gives rise to speculations/expectation that Incus will be removed altogether.
Even if sticking with libvirt were the best decision (I have no opinion on the issue), the lack of planing, the hasty implementation and the lack of perseverance to develop the new system while mainting the old one does not inspire much confidence that iX will achieve much with virtualisation. (But the lack of engagement is consistent with the reluctance to allocate ressources to have a robust catalog for the app systemâŚ)
So stick to 24.10.2.2 or 25.04.2 (when it is out) and use âClassicâ/libvirt for anything you expect to migrate to 25.10. Beyond that, refer to your favourite cristal ballâŚ
Once we have working (and validated) code that addresses the issues we can resolve, then crystal balls can be retired. The âexperimentalâ tag was used for that reason.
@etorix is spot on with this summary - the Incus implementation has been a fiasco (and I say that as someone with decades of professional experience as a project / programme manager and technical strategist with personal experience of having been employed to rescue several fiasco projects).
All people and organisations make mistakes sometimes - but when all you get is denial and misdirection and Orwellian Newspeak about âclassic virtualisationâ rather than any admission of mistakes is IMO more worrying than the mistakes themselves.
Not that iX staff give a hoot what the more experienced community members think (well at least not as long as there is an adoring mass of naive users to stroke their egos anyway).
When I read these words, what I hear in my head is "we knew that this was highly likely to be a complete disaster, but we decided to A) go ahead anyway regardless; and B) weâll call it âexperimentalâ in order to be able to excuse the mess later.
Weâve been quite detailed about it⌠both on the forums and in the T3 videos.
Incus was selected for both its LXC capability and VMs. It showed promise and could mean a major leap in capabilities in future.
It was tagged as âexperimentalâ because there were many unknowns about the project and we did not have the ability to commit to smooth updates from production VMs both from Electric Eel and to Goldeye. We committed to allow Electric Eel updates directly to Goldeye.
It can be marked as a âfailureâ that the message did not get out better. We have been warning people via release notes and software status page. The Electric Eel version 24.10.2.2 remained the recommended version for Conservative users.
Despite this, almost 100,000 users are on Fangtooth, and the general quality is good. Many have operational VMs, but we hope quite a few delayed their updates. It is still not recommended for Conservative users.
We expected that by 25.04.x we would have better quality and a smoother VM migration path to Goldeye identified. Around the start of 25.04.1 testing we came to the conclusion that we would not be successful with that goal. We failed to meet that software quality level, but not because of lack of effort or skills. Integrating two living projects with independent design goals is complex. With 20/20 hindsight, we failed to forecast the complexity of the Incus project.
We then looked for the best path forward and decided that re-enabling âclassic Virtualizationâ was the best approach. After confirming that we could re-enable it while maintaining Incus operation, we announced the plan for 25.04.2.
Meanwhile, Goldeye development has been continuing. The focus is on VM and LXC migration while maintaining the best aspects of Classic and Incus virtualization. That development is not completed and so Iâm not in a position to report on all solved and unsolved issues. That discussion will start in August.
This forum thread is about the change in plans for 25.04.2. Is anyone objecting to the change in plan or is the focus on relitigating previous steps?
For those who do want to relitigate, please start or continue another thread.
The messaging was clear to me. I did not upgrade specifically because the VM feature was marked as experimental and that people who use them in production should wait. I have waited (even though my VMs are not âproductionâ critical).
But itâs understandable people want the latest and greatest and rush to upgrade as soon as itâs released.
SighâŚ
The issue is not with messaging. The issue is the schizophrenic messenger. The issue is deciding between âdonât update if youâre impacted by breaking changesâ and âthereâs a new release whith shiny new features, come playing!â.
Itâs fine to introduce unfinished âexperimentalâ features. But not while disabling existing, working features without a replacement or an upgrade path!
Introduce experimental Incus VM alongside âclassicalâ VMs, both systems running in parallel, and give the time to develop and polish the new system. If it is eventually determined that integration is not satisfactory, too bad, remove it. If the new system develops to satisfaction, then announce that âclassicalâ will be frozen in the next release and then removed in the next next release, preferably with some automated assistance for the transition.
More work? Probably. But better service and better quality.
As it stands, the perception is that iX has hastily pushed a half-baked feature, throwing some users under the bus, refuses to acknowledge the regression, accordingly misreads a backlash against said regression as a backlash against Incus itself, and will now remove the shiny half-baked feature without giving it a fair chance to develop to satisfaction.
If it took only two minor dot releases to determine that Incus would not be successful, then development should have spent more time in the nightlies before making it to release. So it may be an issue with the 6-month release cycle.
Anyway, I bet that the next time iX introduces a new feature âto get user testingâ, the general mood will be more cynical: âA new feature? Ha! Letâs WAIT until the next release and see whether itâs still there.â
For what itâs worth: I did upgrade right away. I experimented with LXC, saw that NVMe latency for my database still sucks on ZFS (not surprising tbh), and put the experiment on ice. I may try again with Goldeye and DirectIO, and whatever flavor of LXC is production-ready then.
Apps have been fine. They worked on k3s and migrated over to Docker. No complaints there.
LXC is a little mad, but useful for this particular, more complex Docker app. Only because TrueNAS is an appliance, and so running random Compose things with a bash handler script doesnât fit well into its app paradigm.
VM is sheer lunacy. Iâm sure there are apps that run best in a VM instead of a container - itâs still lunacy. Why in the century of the fruit bat are we still using full virtualization? Rhetorical question.
I must agree, that it seems this way.
Especially if the complaints for Incus VMs I saw would be very easily solved.
Want to use normal Truenas zvols for VMs like in libvirt? Those that can be already snapshotted and dont need to be in any special pool?
Just use Path on the host as described in Incus docs: incus config device add <instance_name> <device_name> disk source=<path_on_host> [path=<path_in_instance>]
Want CD-ROM device or AHCI disk inside VM like you could with libvirt?
Just add them directly via raw.qemu or other settings like QMP as described in Incus docs.
Both Incus and libvirt configure QEMU like this. Only difference if that this way you do it directly yourself for these devices. So you just need someone who knows QEMU. But I think you should have such a developer if you use QEMU already.
Or want e1000 network driver? There is scriptlet that does it on Incus forum.
So when I see these things that people miss and see how easily it could be implemented, I just dont understand what is the problem. Why not just implement it?
Maybe I dont understand some other reason why is Incus not good for Truenas? Maybe the problem is direct configuring of QEMU because its harder than I think and there isnt other solution that would work for Incus?
I would understand such reasoning. But currently I see no explanation or specific reasons. So I just wonder whats going on.
I think we can all understand that whilst ideally the UI might have support for every single Incus option, in practice iX have to prioritise the features that are most commonly used. The issue is not that some features of Incus havenât yet been implemented in the UI, but rather:
That some critical features werenât implemented from the start;
That some features were implemented in a bad way (root zVols having to be in a fixed place);
That they didnât focus only on LXCs and leave Incus virtualisation for a later version;
That they didnât provide Incus virtualisation in parallel with existing libvirt virtualisation
The community feedback has been so negative that they have now reversed course completely - and despite this being patently obvious to everyone they refuse to acknowledge that it is a retreat and admit that they made a mistake and instead call it a new feature called âClassic Virtualisationâ.
No organisation is perfect - and mistakes will happen. But good organisations admit mistakes, apologise and learn from them (and here is a Youtube video giving a genuine example of how this can be a positive thing).
IMO part of the problem is that the iX decision makers are too close to the coalface - and having some genuine advice from experienced people who are not part of the iX clique and who have a different perspective would IMO help iX make better decisions and avoid issues and allow these forums to be full of positive vibes again.
A genuine Advisory Council (rather than a cynical marketing ersatz version) is IMO the way to achieve this - but only if iX are prepared to listen to the advice it gives. iX does NOT need to cede decision making to the Advisory Council which only needs to provide advice. And a time-limited confidentiality agreement can be used to avoid the advice becoming public until iX announce their decision in terms of what the next release will contain.
I am still hopeful that iX will start to genuinely listen to what has and is still being said in various posts here and on Reddit etc. - unfortunately I am only hopeful rather than optimistic.
I think itâs totally understandable and reasonable that Incus is marked as experimentalâitâs a new feature after all. But for most users like me, it mainly signals that TrueNAS still needs time to improve the Web UI, middleware, and to prove long-term stability. Even if 25.04.x isnât yet at an Enterprise General level, itâs still a stable major release (not an RC or beta), and it introduced a significant new feature. To completely remove that feature in the very next release feels like a step in the wrong direction. It comes across as a hasty or questionable decision, especially for users who were excited about finally having better container support.
The community has long wanted system-level containers. The popularity of CLI-only tools like Jailmaker already shows the strong demand for thisânot just for running simple Docker containers. In terms of functionality, Incus offers powerful CLI options for managing system containers, and I think that seriously improves TrueNASâs ability to run more complex applications.
Now about switching the backend to libvirt-lxcâsaying âusers donât need to care about the backendâ just doesnât reflect reality. The limitations of middleware and Web UI integration were already clear with previous versions of VM support. While TrueNAS VMs may be stable, theyâre very restricted in terms of configuration and hard to manage directly. Thatâs why many users choose to run TrueNAS on Proxmox VEâto get more flexible and manageable VM environments.
Also, Incus has a much more active community, with lots of documentation and solutions available. Thatâs a big advantage libvirt-lxc just doesnât have. This CLI flexibility and stronger community support are exactly why I initially thought TrueNAS chose to replace the old libvirt VM with Incus in a stable 25.04 release. I assumed it reflected a real commitment to improving the limited and hard-to-use VM system. But now, it seems that might not have been the case after all.
Some people might say, âTrueNAS is just a NAS, it doesnât need all these extra featuresâjust do storage well.â But if that were really the goal, why switch to Linux at all instead of continuing with Core (FreeBSD) ? Iâd imagine the move wasnât just about better driver support, but also about expanding features and enabling new use cases.
I get why people would need that - but software vendors that release a product that has appliance / service functions on Windows, and donât also release a Linux version, should be ashamed of themselves.
