Extremely poor performance on 25.04.2

TrueNAS General
1

I just updated to 25.04.2.
I am experiencing a problem that I already had when I tried 25.04.0 (after which I moved back to 24.10 because of virtualization issues).

My whole NAS performance is extremely poor under 25.04.2.
The UI feels less snappy, stuff on CLI takes a long time (just launching htop takes a few seconds) but what’s by far the most noticable is VM performance.
My Windows 11 VM is extremely slow. It takes minutes to boot and even the login process takes about two minutes until everything is booted up but even after that the VM is not usable.
The VM was very fast before the update, as was the whole NAS.

As mentioned this happened as well on 25.04.0 but I then attributed it to the VM problems that came with that version.

It’s also not just my imagination.
Here is my CPU load over the last 8 days:

image
image753×354 38.5 KB

And here areee the last 4 hours:

image
image1042×414 45.3 KB

The small dip to “just” about 2 is when I stopped my VM.
With a user logged into the VM the CPU load hovers at around >=15.

image
image1063×371 55.2 KB

The CPU utilization stays under ~80% though so it seems the CPU is waiting for something.

Here are the outputs of iostat and mpstat:

root@truenas1:~# iostat
Linux 6.12.15-production+truenas (truenas1)     07/31/25        _x86_64_        (12 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.23    0.75   23.06    0.49    0.00   47.46

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
loop0             0.11         1.42         0.00         0.00       2756          0          0
nvme0n1         129.28       852.96      1452.98       614.71    1661134    2829660    1197136
nvme1n1         125.27       813.60      1452.98       614.71    1584478    2829660    1197136
sda               2.56        24.01         6.15         0.00      46768      11984          0
sdb               2.77        34.71         6.07         0.00      67588      11820          0
sdc               2.57        32.87         5.96         0.00      64008      11600          0
sdd              95.95      1544.65       250.52         0.00    3008182     487880          0
sde               2.28        22.36         6.03         0.00      43544      11748          0
zd0              96.07      2427.61       288.62       324.97    4727745     562087     632876


root@truenas1:~# mpstat
Linux 6.12.15-production+truenas (truenas1)     07/31/25        _x86_64_        (12 CPU)

19:05:07     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
19:05:07     all   14.41    0.76   22.52    0.50    0.00    0.47    0.00   14.24    0.00   47.10

I don’t think it’s waiting on disks.

If I cannot resolve this I’ll have to go back to 24.10.2.3 again and stay there until it is resolved.

Any ideas?

2

How did you set up your VM?

3

Updatet myself just now, UI and VM (Windows 11) are at normal operation and fast, no issues as i can see for now. htop runs at no time.

4

As I said the problem seems seems not to come from the VM. Running it just makes it way more apparent. I can still describe my configuration. What exactly do you want to know?

5

Has 25.04.2 been released? I haven’t seen a release announcement.

6

The GUI offered me the update - it’s already on the servers.

7

Oh No … still not working …

After update all my VM’s were still gone !
See screenshot

CanNotReiveResponse
CanNotReiveResponse1682×568 36.9 KB

1 Like
8

Went back to 24.10 for now, will try again soon. I tar’d /var/log though just in case.
Performance is fine again now.

9

That’s a different problem. I suggest a seperate threat for that.

10

I will switch back as well …
not working … :frowning: :frowning: :frowning:

11

Just to sure: I deleted the 25.04.2 boot environment, deactivated all my unsupported customizations and applied the update again.

This time I also watched the first boot and I knew it wouldn’t be faster even before that was completed.

The boot took forever and CPU load was between 8 and 16 before the VM even tried to start.

So that also happens with an unmodified TrueNAS.

2 Likes
12

What customizations did you have running?

13

Check mk agent, enabling ASPM, adding some binaries to /usr/local/bin (e.g. ncdu), disabling console-setup and ipa-epn services (those seem to fail on basically all truenas installations), setting ‘options kvm ignore_msrs’ to disable log spam

14

Thanks for the thorough testing… please report a bug with you hardware details…

If there are any processes that are running harder let us know.

15

I’ll file a bug report :+1:t2:

16

Please ensure you comment on your report that this was an issue for you on 25.04.0 as I understand it above, as if this was introduced earlier it narrows where to investigate.

17

I did: Jira

I hope it’s acceptable that way.
I wanted to create a debug file but the process took extremely long and I wanted to have a working NAS again.
If you do need it I will of course still try to generate one. As mentioned above I already backed up /var/log though and I’d be happy to mount any part of the 25.04 environment read only to extract information.

18

bugclerk has added the link for the debug upload, please upload to allow investigation - Thanks!

19

I started the process twice - one run completed but I had to reload the browser window in between so I got no download prompt. After that I tried again but lost patience due to time pressure.

Is there a way for me to extract the generated debug file by mounting some part of the boot environment and copying the file?

If not I’ll try to get a file generated tomorrow.

Also something I forgot to mention so far: the ix-vendor service failed to start but I couldn’t find the reason via journalctl -u.

20

minimally /var/log/middlewared.log is necessary to understand potential system state.