One busy VM causes other VMs to fall down

Apps and Virtualization
1

My TrueNAS Scale 25.10.4 is running on an HP ProLiant DL360 Gen10. 2 x Intel Xeon, 24 cores each, 48 core total (ignoring hyperthreading). 256 GiB RAM.

I run 10 VMs, total virtual cores 36, total RAM assigned to VMs is 138 GiB. So this ought to work smoothly I would think.

But I have a problem which I’ve narrowed down to one of the VMs being busy seemingly causing other VMs to fall down.

The VM in question is running FreeBSD 15.1 and using poudriere to build many many packages. It used to be assigned more cores and more RAM, but due to this issue I’ve been reducing it. Now it’s assigned only 2 cores and 64 GiB of RAM, yet still this issue occurs.

When poudriere is busy, it causes other VMs to fail to ping and/or become suspended.

Anyone else seen similar?

2

Have you tried installing the QEMU guest agent in all your VMs, so the guests can correctly provide feedback to the host? (In your Windows VMs, this should be the Hyper-V guest tools, and Hyper-V enlightenments should be enabled for these VMs.)

3

I hadn’t. But I think I’ve found the software you mean, here:

https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-qemu-ga/qemu-ga-win-110.0.2-1.el10/qemu-ga-x86_64.msi

I’ve installed it on the Windows VMs… now to learn more about configuring it… :slight_smile:

4

So all my Windows VMs already had the ‘Hyper-V enlightenments’ option enabled. But I didn’t have any guest agent/guest tools installed in the guests themselves. I wonder if that’s an invalid combination that was causing my problems?

So like I said, I installed qemu-ga on some of the Windows VMs, then powered them off, then on. Last night they didn’t become suspended but they did fail to ping many times overnight (as detected by our monitoring). So merely installing that software did not fix it.

But rereading your post, I think maybe you were suggesting to install/enable some Microsoft software related to Hyper V? Were you talking about the ones listed by Get-Service -Displayname “Hyper*”?

5

Basically, ensure these services are enabled and running in the Windows VM:

  • Hyper-V Data Exchange Service
  • Hyper-V Guest Shutdown Service
  • Hyper-V Heartbeat Service
  • Hyper-V Time Synchronization Service

Don’t forget to install the QEMU guest agent in your FreeBSD VM as well. All the guest VMs need to have it running.

6

Yep, that will not fix this…

The “idea” that you need a guest agent in order to NOT crash the system, is pure guessing.

Funny how no other hypervisor needs such things in order to work as expected…

You should install the virtio drivers so you can use virtio disks and shutdown properly when TN shuts down etc.

Are the VM hard drives AHCI right now?
What’ are the memory settings?

Check, ENSURE the state of the other VMs:
Do they crash, shutdown or get suspended???

7

Indeed it does seem like not having them shouldn’t cause such severe problems, but hey if it fixes it, that’s fine with me. Remains to be seen of course.

My FreeBSD & Linux VMs use VirtIO for the disks. My Windows VMs use ACHI. The TrueNAS docs say “We recommend using AHCI for Windows VMs.”

I have installed the VirtIO drivers for networking in the Windows VM though.

My VMs range from 8 to 64 GiB RAM. As I said, the total of all VMs is 138 GiB and the host has 256 GiB. That ought to be a fine margin.

My lesser symptom is that our monitoring pings fail, the worse sympton is that the VM becomes suspended. midclt call vm.resume brings it back.

8

The underlying hypervisor is QEMU. QEMU needs the guest agent to be able to correctly suspend/resume VMs. There’s also an API call to suspend the VM during a snapshot of its zvol which also relies on the guest agent.

9

Have I missed it, or is this nowhere in the TrueNAS docs? I see nothing here for example.

10

I can’t comment on the docs, but running ps while VMs are actively running shows QEMU running.

11

You can always submit Feedback for the online docs.

FEEDBACK TrueNAS Documentation Hub
FEEDBACK TrueNAS Documentation Hub965×596 13.3 KB

12

So I’ve installed CrystalDiskMark on all my Windows VMs.

Running it also reproduces my problem. That is: after CrystalDiskMark is running for a bit, my RDP connection to my VM will drop, and pings to the VM will fail. Then, after a minute or so, the pings restart and the RDP connection resumes. It seems to always happen during the write tests, not the read tests.

For example see the horrible results below. My connection dropped during those first write tests, where the latency hit the roof:

------------------------------------------------------------------------------
CrystalDiskMark 9.0.3 x64 (C) 2007-2026 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
  SEQ    1MiB (Q=  8, T= 1):  1629.023 MB/s [   1553.6 IOPS] <  1255.85 us>
  SEQ    1MiB (Q=  1, T= 1):  1130.381 MB/s [   1078.0 IOPS] <   925.81 us>
  RND    4KiB (Q= 32, T= 1):    36.367 MB/s [   8878.7 IOPS] <   347.31 us>
  RND    4KiB (Q=  1, T= 1):    11.848 MB/s [   2892.6 IOPS] <   342.76 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):     0.418 MB/s [      0.4 IOPS] <3002081.62 us>
  SEQ    1MiB (Q=  1, T= 1):     0.419 MB/s [      0.4 IOPS] <1927943.38 us>
  RND    4KiB (Q= 32, T= 1):    11.255 MB/s [   2747.8 IOPS] <   792.83 us>
  RND    4KiB (Q=  1, T= 1):     8.110 MB/s [   1980.0 IOPS] <   277.76 us>

(My pool is two 2.5" SSDs in a mirror. They are not old, SMART is good, scrubs are good.)

Next I’ll try switching from AHCI to VirtIO.

13

Yes, you should be using virtio mode. But even without them, should not do this.

Given that this happens whit i/o, I suspect the disconnects will continue even in virtio mode.

I have seen where a VM would disconnect RDP during backups due to the heavy data transfers, but that’s network related, where this seems i/o related. (The dataset has sync enabled??).

14

The zvol uses sync=standard.

15

Disable sync and report back.

1 Like
16

Good news is that installing the virtio drivers and/or disabling sync seems to have fixed the issue, though it’s maybe a bit early to say for sure. Bad news is I changed two things at once so don’t know which fixed it. After a few weeks with no problems, I’ll try bringing back sync=standard.

1 Like
17

I do not like guessing, so could you pls do us a favor and find out WHICH it was:

Guess easiest would be re-enabling sync on the dataset. What happens?

Otherwise, could leave sync disabled and then Clone that VM and in cloned VM, uninstall virtio drivers (change VM disk to AHCI). What happens?

18

Yes, like I said already, I’ll try that. But I’ll first wait a few weeks to be sure this issue is fully solved. (This is a prod system with several employees relying on it for work.)

1 Like
19

Holy cow, production vms with no experience in TN?

I would have gone the sure way with the latest free esxi 8.

Or even virtual box under windows.

Once you had learned, tested TN VMs, move them, as it’s trivial to move other VMs to TN.

One last observation. Ensure you have an UPS, automatic shutdown if power fails, and good VM backups! (Sync disabled can create a non booting VM if not shutdown properly).

Your SSD containing the VMs must not be very fast if sync enabled is the real culprit here?

I do run my VMs in Samsung nvme ssds with sync disabled, but do have daily backups, ups and automatic shutdown etc. Happy, no issues or surprises.

20

Who says I have no experience with TN?

We did lots of testing before moving these VMs from Core to Scale but as the problem only occurred under heavy load it was not detected until later.

As I said, it’s two 2.5" SSDs in a mirror. Not the fasting thing ever, but should not be as slow as I’m seeing. And anyway, like you said, it’s an “if”. We’ll see, I’ll update this thread once I retry.

1 Like