Incus VM Crashing

[NickF1227]

From my perspective here, I see that iostat will report that writes are being issued based on the block size of the “disk” itself. I believe this to be correct and expected behavior. In this case its 512 bytes, in yours it is 4k. See wareq-sz in my output from both the guest and host perspective. I don’t thing this is the issue but I’ll keep digging.

As a note, I have written alot of data on loop to work around the fact this disk is so small, and I have not been able to reproduce in Ubuntu 24.04.

while true; do   dd if=/dev/zero of=/dev/nvme0n1 bs=1M status=progress; done

TrueNAS IOstat

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
zd160            0.00      0.00     0.00   0.00    0.00     0.00 1628.00 833536.00     0.00   0.00    0.51   512.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.83  58.84

iostat client side

nickf@orion:~$ iostat -x nvme0n1 10
Linux 6.11.0-21-generic (orion) 	04/07/2025 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.89    0.00    0.63    0.00    0.25   96.22

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1          0.00      0.07     0.00   0.00    0.36    17.11    2.34   1155.71   286.59  99.19  109.07   493.94    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.26   0.11



avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.69    0.00   35.00    0.03    0.10   63.18

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1          0.00      0.00     0.00   0.00    0.00     0.00  999.20 511641.60 126932.00  99.22   88.71   512.05    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   88.64  49.30

Client side:

root@orion:/home/nickf# lsblk -o NAME,PHY-SeC,LOG-SEC,SIZE /dev/nvme0n1
NAME    PHY-SEC LOG-SEC SIZE
nvme0n1     512     512  10G
root@orion:/home/nickf#
root@orion:/home/nickf# while true; do
  dd if=/dev/zero of=/dev/nvme0n1 bs=1M status=progress
done
10633609216 bytes (11 GB, 9.9 GiB) copied, 13 s, 818 MB/s
dd: error writing '/dev/nvme0n1': No space left on device
10241+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 14.0427 s, 765 MB/s
10200547328 bytes (10 GB, 9.5 GiB) copied, 12 s, 850 MB/s
dd: error writing '/dev/nvme0n1': No space left on device
10241+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 13.6552 s, 786 MB/s
10583277568 bytes (11 GB, 9.9 GiB) copied, 12 s, 882 MB/s
dd: error writing '/dev/nvme0n1': No space left on device
10241+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 13.1167 s, 819 MB/s
10042212352 bytes (10 GB, 9.4 GiB) copied, 11 s, 913 MB/s
dd: error writing '/dev/nvme0n1': No space left on device
10241+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 12.8883 s, 833 MB/s
10065281024 bytes (10 GB, 9.4 GiB) copied, 11 s, 915 MB/s
dd: error writing '/dev/nvme0n1': No space left on device
10241+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 12.7005 s, 845 MB/s
10613686272 bytes (11 GB, 9.9 GiB) copied, 12 s, 884 MB/s
dd: error writing '/dev/nvme0n1': No space left on device
10241+0 records in
10240+0 records out

[NickF1227]

@gedavids still working on this, in the mean time, can you try this please?
incus config device set Oracle9 disk0 io.cache none

[gedavids]

I don’t know why you deleted that, I think you cracked it. Setting the io.cache to none seems to have vastly improved the performance. The VM isn’t getting bogged down when I’m doing a big write now. I can still give it other commands in a SSH session and it’ll do them in a few seconds instead of few minutes.

[NickF1227]

Was an accident.

[gedavids]

That also seems to have fixed the 4k write issue on the host OS. I’m seeing 1M writes on TrueNAS same as the VM. It also fixed the issue of the overwriting seeming to require a huge amount of reads from the host OS. It was reading a ton before, far more than could be explained as “metadata”.

[kris]

Whats weird is that “io.cache” defaults to none, and we don’t change the default here…

[gedavids]

I set my io.bus manually to nvme and retested as a worst case.

It got a little mad, but, overall a vast improvement. The queue depth on the VM was like 100, because MSIX queues I suppose, but the queue depth for the zd device on the host OS is staying around 10-12. It was up around 32 before this change (regardless of chosen io.bus).

@kris

Whats weird is that “io.cache” defaults to none, and we don’t change the default here…

Yeah, this whole thing has been very weird. Like changing the io.bus seemed to resolve most of my issues, but if I set it back to nvme it wasn’t as bad as it had been before I changed anything in the CLI. I can’t explain that at all.

[NickF1227]

I suspect, but can’t prove, at least part of your specific issue was the backing pool. We did see diskbusy at 100% or thereabouts for the 3 disks in the pool. In the original configuration (io.bus nvme) it may have been possible for your VM queues to have been greater than you realized at that time.

Do you have SATA drives? They would only support 32 queues IIRC.
Can you run lsblk -b -o NAME,SIZE,PHY-SEC,LOG-SEC,RQ-SIZE
Since nvme supports 1024 queues and scsi drives support 256 you may have had an over-run, where the logical device in the VM could queue more than the disks in the backing pool. Setting the drive to SCSI may have helped mitigate.

I’m not sure why its working better now in the original configuration.

[gedavids]

@NickF1227

Do you have SATA drives? They would only support 32 queues IIRC.
Can you run lsblk -b -o NAME,SIZE,PHY-SEC,LOG-SEC,RQ-SIZE
Since nvme supports 1024 queues and scsi drives support 256 you may have had an over-run, where the logical device in the VM could queue more than the disks in the backing pool. Setting the drive to SCSI may have helped mitigate.

george@truenas42:~$ sudo lsblk -b -o NAME,SIZE,PHY-SEC,LOG-SEC,RQ-SIZE
[sudo] password for george:
NAME                 SIZE PHY-SEC LOG-SEC RQ-SIZE
sda         4000787030016    4096     512     256
├─sda1         2147483648    4096     512     256
└─sda2      3998639460352    4096     512     256
sdb         4000787030016    4096     512     256
├─sdb1         2147483648    4096     512     256
└─sdb2      3998639460352    4096     512     256
sdc         4000787030016    4096     512     256
├─sdc1         2147483648    4096     512     256
└─sdc2      3998639460352    4096     512     256
sdd         4000787030016    4096     512     256
├─sdd1         2147483648    4096     512     256
└─sdd2      3998639460352    4096     512     256
sde         4000787030016    4096     512     256
├─sde1         2147483648    4096     512     256
└─sde2      3998639460352    4096     512     256
sdf         4000787030016    4096     512     256
├─sdf1         2147483648    4096     512     256
└─sdf2      3998639460352    4096     512     256
sdg         4000787030016    4096     512     256
├─sdg1         2147483648    4096     512     256
└─sdg2      3998639460352    4096     512     256
sdh         4000787030016    4096     512     256
├─sdh1         2147483648    4096     512     256
└─sdh2      3998639460352    4096     512     256
sdi         4000787030016    4096     512     256
├─sdi1         2147483648    4096     512     256
└─sdi2      3998639460352    4096     512     256
sdj         4000787030016    4096     512     256
├─sdj1         2147483648    4096     512     256
└─sdj2      3998639460352    4096     512     256
sdk         4000787030016    4096     512     256
└─sdk1      4000785104896    4096     512     256
sdl         2000398934016    4096     512     256
└─sdl1      1998251360256    4096     512     256
sdm          500107862016     512     512      64
├─sdm1             524288     512     512      64
├─sdm2       482927427584     512     512      64
└─sdm3        17179869184     512     512      64
sdn          500107862016     512     512      64
├─sdn1             524288     512     512      64
├─sdn2       482927427584     512     512      64
└─sdn3        17179869184     512     512      64
sdo         2000398934016    4096     512     256
└─sdo1      1998251360256    4096     512     256
zd0            1535967232   16384     512
zd16            662700032   16384     512
zd32           1116389376   16384     512
zd48            954908672   16384     512
zd64          10737418240   16384     512
zd80          42949689344   16384     512
zd96         214748381184   16384     512
zd112         32212271104   16384     512
zd128        536870912000   16384     512
zd144         32212254720   16384     512
zd160       1099511644160   16384     512
zd176         32212271104   16384     512
zd224         10737418240   16384     512
nvme1n1       14403239936     512     512     127
└─nvme1n1p1   14258536448     512     512     127
nvme2n1       58977157120     512     512    1023
└─nvme2n1p1   58977071104     512     512    1023
nvme0n1       58977157120     512     512    1023
└─nvme0n1p1   58977071104     512     512    1023

[NickF1227]

Hmm Interesting. Pehaps not that simple, then.

From a few posts up it does look like only 4 queues in each disk. Although its a screenshot, this may not have been the cae throughout the run. It is these disks in that pool right?

The partition table here looks…weird? Did you build this pool manually? It’s also an interesting mix of SAS and SATA (and queue size), and sdl is half the size as the rest and sdm is even smaller…?

sdj         4000787030016    4096     512     256
├─sdj1         2147483648    4096     512     256
└─sdj2      3998639460352    4096     512     256
sdk         4000787030016    4096     512     256
└─sdk1      4000785104896    4096     512     256
sdl         2000398934016    4096     512     256
└─sdl1      1998251360256    4096     512     256
sdm          500107862016     512     512      64
├─sdm1             524288     512     512      64
├─sdm2       482927427584     512     512      64
└─sdm3        17179869184     512     512      64

Can you post
zpool list -v and lsblk -b -o NAME,SIZE,PHY-SEC,LOG-SEC,RQ-SIZE,PARTUUID

[gedavids]

@NickF1227

From a few posts up it does look like only 4 queues in each disk. It is these disks in that pool right?

That particular screenshot is a second pool that is a 1vdev RAIDZ1. I backup my primary pool to this one. I created a blank zvol there as a test initially because it’s basically an idle pool 99% of the time, so writing data to it seemed like a good test.

@NickF1227

The partition table here looks…weird? Did you build this pool manually? It’s also an interesting mix of SAS and SATA, and sdl is half the size as the rest.

No, but things here are old. The drive letters move around a lot on reboots, no idea why. This started life as a Core system (11 something?) years ago and I’ve just been upgrading and replacing things.

sda through sdj are in pools that originated when each disk got a 2GB swap partition.

sdk is a spare drive that’s been added a removed a ton of times while I was sorting out one issue or another.

sdl and sdo are the third pool of SMR drives that originally housed the second zvol for my Security Onion VM, but now that pool just takes replication from that zvol on the primary pool. I recreated this pool when I moved the zvol to the primary pool, that’s why the partition is the new slightly undersized standard and has no swap partition.

sdm and sdn are the mirrored pair that make my boot pool. I’m not sure what they would look like today, but way back when Core made a 16GB partition as swap space.

The boot pool are SATA SSDs attached directly to the motherboard, the rest of the drives are SATA HDDs connected through a SAS HBA.

[gedavids]

@NickF1227

Can you post
zpool list -v and lsblk -b -o NAME,SIZE,PHY-SEC,LOG-SEC,RQ-SIZE,PARTUUID

george@truenas42:~$ sudo zpool list -v
NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Array                                     10.9T  3.19T  7.69T        -         -    17%    29%  1.00x    ONLINE  /mnt
  mirror-0                                3.62T  1.09T  2.54T        -         -    16%  29.9%      -    ONLINE
    sdc2                                  3.64T      -      -        -         -      -      -      -    ONLINE
    sde2                                  3.64T      -      -        -         -      -      -      -    ONLINE
  mirror-1                                3.62T  1.07T  2.56T        -         -    19%  29.4%      -    ONLINE
    sdd2                                  3.64T      -      -        -         -      -      -      -    ONLINE
    sdb2                                  3.64T      -      -        -         -      -      -      -    ONLINE
  mirror-2                                3.62T  1.04T  2.59T        -         -    18%  28.6%      -    ONLINE
    sda2                                  3.64T      -      -        -         -      -      -      -    ONLINE
    sdf2                                  3.64T      -      -        -         -      -      -      -    ONLINE
logs                                          -      -      -        -         -      -      -      -         -
  mirror-4                                54.5G  1.39M  54.5G        -         -     2%  0.00%      -    ONLINE
    nvme0n1p1                             54.9G      -      -        -         -      -      -      -    ONLINE
    nvme2n1p1                             54.9G      -      -        -         -      -      -      -    ONLINE
spare                                         -      -      -        -         -      -      -      -         -
  712787b5-00fd-4dbf-99c5-110673a5d829    3.64T      -      -        -         -      -      -      -     AVAIL
Pool2                                     14.5T  10.7T  3.87T        -         -     4%    73%  1.00x    ONLINE  /mnt
  raidz1-0                                14.5T  10.7T  3.87T        -         -     4%  73.4%      -    ONLINE
    sdj2                                  3.64T      -      -        -         -      -      -      -    ONLINE
    sdi2                                  3.64T      -      -        -         -      -      -      -    ONLINE
    sdg2                                  3.64T      -      -        -         -      -      -      -    ONLINE
    sdh2                                  3.64T      -      -        -         -      -      -      -    ONLINE
boot-pool                                  206G  27.0G   179G        -      258G    18%    13%  1.00x    ONLINE  -
  mirror-0                                 206G  27.0G   179G        -      258G    18%  13.1%      -    ONLINE
    sdn2                                   450G      -      -        -       14G      -      -      -    ONLINE
    sdm2                                   450G      -      -        -       14G      -      -      -    ONLINE
nsm                                       1.81T   482G  1.34T        -         -     0%    25%  1.00x    ONLINE  /mnt
  mirror-0                                1.81T   482G  1.34T        -         -     0%  25.9%      -    ONLINE
    76d73676-83f5-4238-9571-fe1381bc5c36  1.82T      -      -        -         -      -      -      -    ONLINE
    8128219b-52c0-4396-973e-b305a166a155  1.82T      -      -        -         -      -      -      -    ONLINE
logs                                          -      -      -        -         -      -      -      -         -
  8c80aeb6-7101-4f55-a8af-e568198c3a7a    13.3G      0    13G        -         -     0%  0.00%      -    ONLINE

george@truenas42:~$ sudo lsblk -b -o NAME,SIZE,PHY-SEC,LOG-SEC,RQ-SIZE,PARTUUID
NAME                 SIZE PHY-SEC LOG-SEC RQ-SIZE PARTUUID
sda         4000787030016    4096     512     256
├─sda1         2147483648    4096     512     256 5daaaa26-0098-11ee-81e7-f02f74cc49b8
└─sda2      3998639460352    4096     512     256 5db8b5ca-0098-11ee-81e7-f02f74cc49b8
sdb         4000787030016    4096     512     256
├─sdb1         2147483648    4096     512     256 3f7fdb72-ef9d-11ed-b1cd-14dae9c52267
└─sdb2      3998639460352    4096     512     256 3f939750-ef9d-11ed-b1cd-14dae9c52267
sdc         4000787030016    4096     512     256
├─sdc1         2147483648    4096     512     256 7f7fa463-008e-11ee-81e7-f02f74cc49b8
└─sdc2      3998639460352    4096     512     256 7f8d4ac8-008e-11ee-81e7-f02f74cc49b8
sdd         4000787030016    4096     512     256
├─sdd1         2147483648    4096     512     256 28e6d7bb-ef9d-11ed-b1cd-14dae9c52267
└─sdd2      3998639460352    4096     512     256 28f9ef15-ef9d-11ed-b1cd-14dae9c52267
sde         4000787030016    4096     512     256
├─sde1         2147483648    4096     512     256 9956b36f-de51-11ee-9e1b-f02f74cc49b8
└─sde2      3998639460352    4096     512     256 9969857e-de51-11ee-9e1b-f02f74cc49b8
sdf         4000787030016    4096     512     256
├─sdf1         2147483648    4096     512     256 abc48510-009a-11ee-81e7-f02f74cc49b8
└─sdf2      3998639460352    4096     512     256 abd5b893-009a-11ee-81e7-f02f74cc49b8
sdg         4000787030016    4096     512     256
├─sdg1         2147483648    4096     512     256 c380cb32-fa63-11ee-bc93-00089bf786d9
└─sdg2      3998639460352    4096     512     256 c3ca484f-fa63-11ee-bc93-00089bf786d9
sdh         4000787030016    4096     512     256
├─sdh1         2147483648    4096     512     256 c3a0a6d6-fa63-11ee-bc93-00089bf786d9
└─sdh2      3998639460352    4096     512     256 c3beafcf-fa63-11ee-bc93-00089bf786d9
sdi         4000787030016    4096     512     256
├─sdi1         2147483648    4096     512     256 c38d2d85-fa63-11ee-bc93-00089bf786d9
└─sdi2      3998639460352    4096     512     256 c3b5ee04-fa63-11ee-bc93-00089bf786d9
sdj         4000787030016    4096     512     256
├─sdj1         2147483648    4096     512     256 c36dddf5-fa63-11ee-bc93-00089bf786d9
└─sdj2      3998639460352    4096     512     256 c3aa7fd3-fa63-11ee-bc93-00089bf786d9
sdk         4000787030016    4096     512     256
└─sdk1      4000785104896    4096     512     256 712787b5-00fd-4dbf-99c5-110673a5d829
sdl         2000398934016    4096     512     256
└─sdl1      1998251360256    4096     512     256 8128219b-52c0-4396-973e-b305a166a155
sdm          500107862016     512     512      64
├─sdm1             524288     512     512      64 4a257e39-4945-11ef-9263-f02f74cc49b8
├─sdm2       482927427584     512     512      64 4a2b7265-4945-11ef-9263-f02f74cc49b8
└─sdm3        17179869184     512     512      64 4a297652-4945-11ef-9263-f02f74cc49b8
sdn          500107862016     512     512      64
├─sdn1             524288     512     512      64 f31bf665-4942-11ef-a6c1-f02f74cc49b8
├─sdn2       482927427584     512     512      64 f3233a39-4942-11ef-a6c1-f02f74cc49b8
└─sdn3        17179869184     512     512      64 f320dc57-4942-11ef-a6c1-f02f74cc49b8
sdo         2000398934016    4096     512     256
└─sdo1      1998251360256    4096     512     256 76d73676-83f5-4238-9571-fe1381bc5c36
zd0            1535967232   16384     512
zd16            662700032   16384     512
zd32           1116389376   16384     512
zd48            954908672   16384     512
zd64          32212254720   16384     512
zd80          42949689344   16384     512
zd96         214748381184   16384     512
zd112         32212271104   16384     512
zd128        536870912000   16384     512
zd160       1099511644160   16384     512
zd176         32212271104   16384     512
zd224         10737418240   16384     512
nvme1n1       14403239936     512     512     127
└─nvme1n1p1   14258536448     512     512     127 8c80aeb6-7101-4f55-a8af-e568198c3a7a
nvme2n1       58977157120     512     512    1023
└─nvme2n1p1   58977071104     512     512    1023 271dbfdd-f4f6-11ed-9675-0015179fc478
nvme0n1       58977157120     512     512    1023
└─nvme0n1p1   58977071104     512     512    1023 271d878c-f4f6-11ed-9675-0015179fc478

[Captain_Morgan]

Can I summarize that:

io.bus = nvme seems to cause performance issues
1MB writes appear as many 4KB writes

(If that is what is happening then all the performance issues and reads would be expected)

[kris]

No, io.bus=nvme may not have been a factor here.

[gedavids]

I did more testing with different cache settings and setting io.cache=writeback seemed to behave the most like what I was seeing initially. It’s definitely not as bad as it was before. I’m going to create a new VM and set nothing via the CLI and see if the issues I was having come back.

[NickF1227]

@Captain_Morgan

io.bus = nvme seems to cause performance issues

Just so my position is clear here - in this case for @gedavids I don’t think that is true.

I have seen stability problems with io.bus=nvme in Windows systems but this does not seem to be isolated to Incus and is instead more likely a limitation of the virtual NVME driver in Windows itself.

This is also going to be selectable in the UI soon.

[gedavids]

Well I think there’s some sort of issue with the Incus/zvol IO stack. I can’t put my finger on exactly where, but I had to delete my test VM that I had modified in the CLI to recreate the issue and fill the 3vdev pool of mirrors to around 80% to get this happening consistently.

Note here I’m not writing to the NVMe root drive here. When I added my test zvol it was attached as a virtio-scsi type without me doing anything in the CLI. The IO seems to be backing up and causing a log jam over time and the truenas system rises to over 100.

The drives are busy, but not maxed either. There’s no compression on this zvol, and this is a pretty fast Ryzen 5600 with 128GB of 3200 MTs ECC RAM. This should have plenty of processing power and bandwidth, so why is it slow? Also, why is one zvol (the test volume) so greatly affecting other zvols (in this case the NVME boot volume of the test VM and my other VMs). It just isn’t handling IO congestion in a very graceful fashion at the moment. io.cache=none seems to help the most with this, but why as that it is supposed to be the setting already?

Like there’s just a few things that seem to need polish (being able to set the io.cache in the UI would be nice) and some inefficiencies that need to be investigated. I’d help more if I could, but it’s just over my head to figure out these issues.

[NickF1227]

@gedavids I’ve tested this on a few systems now. This testing includes one which has a similar config to yours, a 5600g am5 system. On another system I even made a very slow and bad pool, a single stripe, to host a zvol. I still couldn’t reproduce the problem you are having in my Ubuntu VMs.

That said I did speak to some of the developers today on this topic. There was a change that’s coming for .0, and I believe it may resolve your issue.

In the mean time to see if the issue is what I believe it to be, can you actually reproduce this issue if you write to a file in the default root disk? I think for each instance here you’ve tested on a second disk attached to the VM.

[gedavids]

@NickF1227

In the mean time to see if the issue is what I believe it to be, can you actually reproduce this issue if you write to a file in the default root disk? I think for each instance here you’ve tested on a second disk attached to the VM.

I’ll rig that up tomorrow for ya. Time for sleep in my time zone.

[gedavids]

Wild, writing to the root partition behaves entirely differently.