That’s the first time that I’ve heard Hyper-V integraiton services should be installed in qemu VMs. I would’ve thought virtio-windows drivers would be the right choice?
I know that hyper-v enlightenments can be enabled on TrueNAS, Proxmox and such but why should Hyper-V integration services be installed?
Because QEMU emulates some of the Hyper-V services on the host side.
What is missing and how does that impact the vm?
See this table of Hyper-V integration services:
| Name | Windows Service Name | Linux Daemon Name | Description | Impact on VM when disabled |
|---|---|---|---|---|
| Hyper-V Heartbeat Service | vmicheartbeat |
hv_utils |
Reports that the virtual machine is running correctly. | Varies |
| Hyper-V Guest Shutdown Service | vmicshutdown |
hv_utils |
Allows the host to trigger virtual machines shutdown. | High |
| Hyper-V Time Synchronization Service | vmictimesync |
hv_utils |
Synchronizes the virtual machine’s clock with the host computer’s clock. | High |
| Hyper-V Data Exchange Service (KVP) | vmickvpexchange |
hv_kvp_daemon |
Provides a way to exchange basic metadata between the virtual machine and the host. | Medium |
| Hyper-V Volume Shadow Copy Requestor | vmicvss |
hv_vss_daemon |
Allows Volume Shadow Copy Service to back up the virtual machine without shutting it down. | Varies |
| Hyper-V Guest Service Interface | vmicguestinterface |
hv_fcopy_daemon |
Provides an interface for the Hyper-V host to copy files to or from the virtual machine. | Low |
| Hyper-V PowerShell Direct Service | vmicvmsession |
not available | Provides a way to manage virtual machine with PowerShell without a network connection. | Low |
The QEMU guest agent in the VirtIO guest tools overlaps these to some extent. However, my experience is it’s better to use native Windows services instead of the QEMU guest agent equivalents for Windows VMs, especially if you use the latest release of the VirtIO tools instead of a stable release.
What specifically are the SSD’s?
Make, Model, SATA, SAS, NVMe?
Are they enterprise or consumer. Consumer SSD’s have a habit of having a bunch of fast cache and then slow NAND - so once you run out of cache due to (for example) a lot of writes you are left with slow NAND which will choke causing the scenario you are seeing until the drive sorts itself out
Just a thought
I think I’m still missing something. Those all seem to add features for hyper-v, not improve performance. How would I use those features when running on kvm?
Great question. 2,5" doesn’t narrow it down much.
The purpose of these is not to improve performance, although that may be a side effect. The purpose is to improve the integration between the guest and the host. Look at my post #4 on this thread.
Both of which are consumer drives. The Micron is described as entry level, whilst the 870 is probably a bit better (it isn’t a QVO - which are utter garbage).
They both use TLC and will likley suffer from performance throttling under high IO Load such as you have indicated that you are seeing (and I assume confirmed by looking at the reports)
My opinion is that you have the wrong drives, probably. They would be just fine with typical loads but I think will falter under sustained high load, in particular with writes.
Of course, proper enterprise drives are extremely expensive. So a few more questions:
I have some other ideas that might help. Depending on total RAM, available RAM etc you could use tmpfs for build folders - that would significantly reduce the SSD IO Load (but would depend on size requirements and RAM availability). My other idea involves Enterprise SSD’s and possibly significant expense.
I have 256 GiB RAM total, indeed I probably could use some kind of RAM disk for poudriere…
Then the first option is tmpfs
The second option is a couple of second hand enterprise SSD’s of a suitable size for just the poudriere VM and its data, possibly combined with tmpfs. The SSD’s I have found between £300 & £400 approx each on ebay (onbviously some are a lot more)
Would a small SLOG help here? Have the same issue, disabling sync makes things much faster. But want to avoid the risk of data loss.
A SLOG is not a write cache. It will only help if you are using sync writes, and that won’t be as quick as async writes.
If you insist on using sync writes (which is moderately sensible for VM virtual disks and databases) then. providing the SLOG is suitable (hardware), then it will be better than sync without a SLOG.
Suitable hardware: Ultra Low Latency, High Speed, Power Protection, High endurance all in comparison to the pool its attached to. It used to be that the Intel Optane (but not the diddy little M.2) drives were ideal. Also, some of the PCIe drives that used DRAM that I can’t remember what they are called). Useful Optanes were - 900/905, P5800X or the P1600X. If you have a pool of HDD, then an enterprise SSD would work. If however you have a pool of SSD’s then you will want something better.
In latency terms Sync (worst) > Sync+SLOG > Async (best)
Thanks for this idea. Come to think of it, I also have a small old SSD, I could perhaps move the poudriere VM to that.
Still, the original issue is still suspicious. The performance was so horrible that I still suspect some qemu/kvm/truenas bug, but it’s not so easy to reduce to a small reproducible example.
Oh wait, I already have USE_TMPFS=yes, it seems that’s the default. Maybe switching to all would be even better…
I know what they do on Hyper-V but I doubt that a KVM based virtualization environment can use them in any way. What exactly can I do more with these services installed on a Windows VM on TrueNAS or Proxmox than only with qemu guest tools?
If the Hypervisor does a good job, it should be transparent and the VM should not need helpers or workarounds. They are nice, but stating ‘needed’?
Must have them in order for the VM to shutdown “properly”?
What ever happened to ACPI signal?
(Power On/Off button in physical computer).
For the Console? What if you have RDP?
(still, other Hypervisors offer native console, without needing to add displays).
But now that we are here, let’s get to conclusions:
… Specific performance gains include:Hyper-V Integration Services:
Essential for all guest OSs, these services reduce CPU overhead by up to 30% during I/O operations and enable VMBus communication. Without them, virtual machines rely on emulated legacy hardware, causing significant performance bottlenecks.
So, for most VMs (no need for dedicated SR-IOV, etc.):
Beyond partitioning a GPU and specific CPU, memory efficiencies, they are not “essential” for the VM to run.
This thread was about my crippling VM performance when the system is under load, not about general “essential” recommendations. A 30% win (if your quoted LLM reply is anything better than a hallucination) would indeed be essential for the situation being discussed.
