Fangtooth 25.04.2 Nightly is available with Classic Virtualization

[Foxtrot314]

@Protopia

• That some critical features weren’t implemented from the start;
• That some features were implemented in a bad way (root zVols having to be in a fixed place);
• That they didn’t focus only on LXCs and leave Incus virtualisation for a later version;
• That they didn’t provide Incus virtualisation in parallel with existing libvirt virtualisation

Yeah, I must agree. If Incus was at first LXC-only it would allow iX to introduce it without any pressure. They could do that, make themselves familiar with Incus and once LXC was done, they could decide if they also switch VMs to Incus once they know Incus better.

But they decided to do both in single jump which meant they must do everything at once. And once they realized they cant do it in time it was over.

Honestly, if they did LXC first, they would have less work with VMs because Incus reuses a lot of logic so doing things for LXC means doing a lot of work for VMs as a byproduct.

@ChanningHe

Also, Incus has a much more active community, with lots of documentation and solutions available. That’s a big advantage libvirt-lxc just doesn’t have. This CLI flexibility and stronger community support are exactly why I initially thought TrueNAS chose to replace the old libvirt VM with Incus in a stable 25.04 release.

This was also my thought originally. Incus seems to have open and flexible development so I thought iX would be better able to cooperate and push upstream things they would like to use in Truenas.
That was why I thought Incus would be good long-term solution.

Btw, if I heard right iX plans for 25.10 enterprise ready VMs. So they basically self imposed on themselves time limit and when they discovered they are unable to implement Incus in time the only way was go back to libvirt and hope for the best?

I wonder how they want to achieve HA on libvirt. I know Incus has built-in clustering which is one of the reason why I thought its a good fit for Truenas.

[Captain_Morgan]

@etorix

refuses to acknowledge the regression, accordingly misreads a backlash against said regression as a backlash against Incus itself,

Not really true.

We marked it as Experimental to indicate it was a regression.

We are releasing 25.04.2 as a fix to that regression.

We have not indicated any issues with Incus… its simply that the integration with TrueNAS and Incus (for VMs) is more complex than expected.

[Protopia]

@yorick

Goldeye: A fish. Also the code name for a TrueNAS release.

Yup - I realised that a few days ago. I guess it’s obvious I am a Bond fan (since I was about 10 when my father took me to see Goldfinger at the local cinema).

[Foxtrot314]

@Captain_Morgan

We have not indicated any issues with Incus… its simply that the integration with TrueNAS and Incus (for VMs) is more complex than expected.

Looks like you are willing to answer questions pretty straight forward.
I already asked in podcast but I didnt get exactly straight answer, so I will try asking you.

Were any decisions made about keeping or removing Incus from Truenas?
What I mean is… Was decision made that Incus experiment failed so Incus will be removed completely in future? Or was decision made that Incus integration for LXCs and VMs will continue, just slower than expected. Or was no such decision made yet and this is subject to current internal discussions?

I am asking specifically about Incus as backend service.

Thanks.

[PrivatePuffin]

@Captain_Morgan

We marked it as Experimental to indicate it was a regression.

I agree 100% that marking it as experimental is totally reasonable and should be enough to warn users “THIS CAN BE TROUBLESOME!!!”

@Captain_Morgan

We are releasing 25.04.2 as a fix to that regression.

I personally think that most people dont mind moving to a new system, but more the fact that a major update also included a regression in featureset, experimental or otherwise.

Imho, your choice to open-up classic virtualisation should’ve been kept in latest major. But it’s never too late to fix mistakes and you guys did.

While I’m extremely critical on what iX does, its not okey for you guys to be attacked over fixing your mistakes.

Communication wise: Yes it could have been handled a lot better in many places, but I dont think that is really the core issue here.

Mistakes where made and the fix is expedited in a dot-release with due haste.

@Captain_Morgan

its simply that the integration with TrueNAS and Incus (for VMs) is more complex than expected.

Thats very well worded, Incus is a great product. But in terms of default/opinionated design choices, it’s “different” in a lot of places compared to TrueNAS.

These problems mostly show themselves when you’re exposing it to hard-core users that have specific requirements.

Yes it could’ve been noted earlier and delayed, but it also isn’t really time-efficient to spend QA on trying-to-break a new experimental integration either.

[Jorsher]

Thanks for the update. I’m only using Incus to host a single HAOS VM, so not a huge impact for me.

But, I hope iX revisits Incus for VMs in the future when it’s closer to ‘ready.’

[Protopia]

I could write a lengthy summary of both sides of this argument - and indeed I did and then deleted it without posting - but in the end I decided that there is little point because:

1. iX staff have their own viewpoint and they are not going to change their minds.

2. Active Community members (as opposed to users who just use TrueNAS and don’t participate in the forums and are not generally aware of what is happening) have also made up their minds based on their personal use and/or broader experiences.

But I will make one new point to try to put the iX approach in a wider software-industry context - or rather ask it as an question so as to be open to persuasion…

Are there examples of any other major software vendor (with a comparable user base) that has delivered a major software version which completely replaced a mature and widely used area of functionality with something that was knowingly classified “experimental” because the vendor was doubtful that they could make it work effectively?

I am genuinely interested in knowing if this is a more common way of doing things, because if a number of other vendors follow the same approach then it is more widely acceptable than my prior experience has led me to believe.

[Farout]

@Protopia

Are there examples of any other major software vendor (with a comparable user base) that has delivered a major software version which completely replaced a mature and widely used area of functionality with something that was knowingly classified “experimental” because the vendor was doubtful that they could make it work effectively?

Happened to some online games i used to play. Ususally causing a big part of the playerbase to leave…

[shoulders]

I am just happy ix are putting back the classic VM system, it does what it says on the tin.

An earlier commented suggested when ix moved to incus they should of left virtualization with classic and I agree with that.

[ChanningHe]

@Captain_Morgan

We are releasing 25.04.2 as a fix to that regression.

We have not indicated any issues with Incus… its simply that the integration with TrueNAS and Incus (for VMs) is more complex than expected.

So why not continue with the approach used in 25.04.2 for future releases like 25.10+? Bring back libvirt VMs, and keep Incus container support—even if it stays labeled as experimental.

In my view, having access to Incus’s advanced CLI on TrueNAS has never been a bad thing. I understand TrueNAS may want to avoid introducing uncertain or unsupported CLI configuration during community debugging. But not all support needs to come from the official team—user-to-user support matters too. TrueNAS is still an Enterprise Storage platform, and I believe users should be encouraged to understand what they’re doing, learn Linux, and gain more control if they want it.

In this sense, Proxmox VE is still the best example. Its open and flexible CLI has enabled strong community collaboration and a wide range of use cases. Just look at the NVIDIA vGPU project—long before it got official support from Proxmox or was listed on NVIDIA’s compatibility pages, the community had already figured out how to make it work through CLI, discussed it, and used it for more than a full major release. That’s the power of community-driven innovation.

@Foxtrot314

Btw, if I heard right iX plans for 25.10 enterprise ready VMs. So they basically self imposed on themselves time limit and when they discovered they are unable to implement Incus in time the only way was go back to libvirt and hope for the best?

I think that’s exactly what they did. I checked the current nightly (master) builds, and it just looks like the old VM UI is back. I also noticed some commits related to fixing HA functionality.

But honestly, I’m not sure how they define “enterprise ready VMs.” If it’s just the same old libvirt-based UI with minor fixes.

[AoSoG]

Podcast, where?

[SmallBarky]

T3 or TrueNAS Tech Talk on YouTube

TrueNAS Tech Talk - T3

[troy]

Has anyone been able to create a VM using the nightly?

I downloaded the TrueNAS nightly 25.04.2-MASTER-20250720-234152 to test the return of classic VMs; however, the VM creation fails with the following error.

[EINVAL] vm_create.enable_secure_boot: Extra inputs are not permitted

More info

 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 323, in process_method_call
    result = await method.call(app, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 40, in call
    result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 906, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 715, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 256, in create
    return await self.middleware._call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 715, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/service/crud_service.py", line 287, in nf
    rv = await func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 91, in wrapped
    args = list(args[:args_index]) + accept_params(accepts, args[args_index:])
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/handler/accept.py", line 23, in accept_params
    dump = validate_model(model, args_as_dict, exclude_unset=exclude_unset, expose_secrets=expose_secrets)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/handler/accept.py", line 80, in validate_model
    raise verrors from None
middlewared.service_exception.ValidationErrors: [EINVAL] vm_create.enable_secure_boot: Extra inputs are not permitted

I am not using secure boot.

Bug report filed: NAS-136817
​
​----

Fixed in nightly version: 25.04.2-MASTER-20250722-140850
​

[SmallBarky]

From your linked ticket
“Unfortunately 25.04.2 nightlies have this issue and isn’t going to be fixed. You’ll need to either update to 25.10 nightlies or wait for 25.04.2 proper to be released.”

[davistw]

Does this mean we wont be able to create VM’s using VM Classic?

[pmh]

We will, but only in the released 25.04.2 - not in the current nightly build.

[davistw]

Gotcha. I read that wrong…Whew…

[troy]

I was confused about that. I thought because they said the nightly was in a code freeze, that’s what would be released as 25.04.2

[Captain_Morgan]

@troy

I was confused about that. I thought because they said the nightly was in a code freeze, that’s what would be released as 25.04.2

That’s almost true and was my expectation, but there is an unintended difference in the nightly that is not in the 25.04.2 version under test. We are planning to fix.

25.04.2 is in its last QA cycle and will be released next week.

[Captain_Morgan]

@troy

So, according to the OP, 25.04.2 nightly is in a code freeze and they’re asking us to test it. But upon testing, creating a virtual machine fails.

Apologies for the snafu.

I’ve been informed that tonight’s Nightly resolves the issue. Actual 25.04.2 release is a week away.