Proxmox VE against SCALE 25.10 over NVMe/TCP in production: a target-side MDTS finding, and notes on the storage plugin

We run three Proxmox VE 9.2.4 nodes against a TrueNAS SCALE 25.10 array over NVMe/TCP in production, using the storage plugin from GitHub - truenas/truenas-proxmox-plugin: For providing seamless block storage to PVE · GitHub, with iSCSI on the side for testing. Three things came out of that. The first one is purely target-side and applies to any Linux initiator, not just Proxmox.

nvmet-tcp advertises no MDTS

The NVMe/TCP target in SCALE reports MDTS = 0, meaning “no limit”. A Linux initiator takes that literally and, on a namespace with a large NOWS, can build commands far beyond what the target will actually accept: nvmet_tcp_map_data() rejects anything above NVMET_TCP_MAXH2CDATA (4 MiB) with an SGL error, so the advertised limit and the enforced one are two different numbers. A patch to advertise a bounded MDTS is under review on linux-nvme; the discussion there also established that the target does not currently reject an over-MDTS command with Invalid Field as the spec requires, which needs a separate fix in the nvmet core. If you export NVMe/TCP namespaces from SCALE to Linux hosts this is worth knowing whatever hypervisor you run. On the Proxmox side we cap it meanwhile with the plugin option tn_nvme_max_io_kb 1024.

Snapshots taken on the TrueNAS side are invisible to Proxmox, and block rollback

If a periodic snapshot task (or a manual zfs snapshot) snapshots a zvol that backs a VM, Proxmox never learns about it. Two consequences: the snapshot does not appear in the Snapshots tab, and if it is newer than your last Proxmox snapshot, qm rollback refuses with “not most recent snapshot” and gives you no way to see or remove the one that is in the way. We wrote an importer that registers those snapshots in the guest configuration, for VMs and containers, using the ZFS creation time as the snapshot timestamp, so they show up in the GUI and can be rolled back to and deleted from Proxmox. It is PR #110 upstream.

The plugin: a fork with the fixes, and prebuilt packages

Upstream is moving fast in beta and several things bit us, so we keep a fork with the fixes applied and packaged builds, and send everything back as pull requests. Open upstream right now besides the importer: a portal that was down at connect time is never retried (#94), iSCSI CHAP against 25.10 (#95), and unbounded API retries that stall pvestatd while the array is unreachable (#97). We also reported that the install.sh linked from the README is from an older lineage and fails with “broker: scfg missing api_host/api_key” (#107).

Packages, sha256 and per-release notes: Releases · alfonsokuen/truenas-proxmox-plugin · GitHub

Happy to answer questions here, and to hear from anyone running the same combination.

Why are you not submitting this as a Bug or in the thread announcing this?

Fair question, and the honest answer is that I got the order wrong: this should have gone to the bug tracker first. I will file the MDTS one there, since it is a target-side defect and not a discussion point, and I will link it back here so anyone who lands on this thread can follow it.

On the announcement thread: I stayed out of it because it is an announcement and my post is three separate things, one of which (the qemu-img cache behaviour) is not a TrueNAS issue at all. But you are right that the plugin part belongs where the people working on it are reading, so I will take that side of it there rather than keep it in its own corner.

For what it is worth, the plugin-side findings are already upstream as pull requests rather than just as complaints: portal reconciliation, iSCSI CHAP against 25.10, bounded API retries, and importing snapshots taken on the array into the guest config. The last one is the one I would most like to see land, because it removes a sharp edge that has nothing to do with which build you run: a periodic snapshot task on the array silently blocks rollback in Proxmox.

Thanks for the nudge.

Filed it (Jira ticket NAS-144014, search the TrueNAS project for the same title as this thread) — and closing the loop, since it’s already resolved: their triage team confirmed the analysis is correct, that it’s upstream kernel code TrueNAS ships unmodified, that there’s no actual I/O failure today (the 4 MiB hard limit rejects an oversized command before it can be mishandled), and closed it as a third-party issue pointing back at the same linux-nvme patch. Nothing left to do here on the TrueNAS side; the fix lands whenever the kernel patch does, through their normal kernel rebases.

Thanks again for the nudge - should have gone there first.