Proxmox Storage Plugin API Enhancements for dataset / zvol management

BooMShanKerX · May 3, 2025, 10:17pm

We are trying to build a native storage plugin for Proxmox that uses the API to manage ZFS over ISCSI. Unfortunately several zfs commands are not implemented in the API. This forces us to use ssh+root to execute zfs commands directly on the TrueNAS host.

Eg. Creating linked clones requires the renaming of a zvol from
vm-<VMID>-disk-0 → base-<VMID>-disk-0.

We noticed that the ability to rename a zvol has also been removed from the WebUI, leaving the only option shell commands.

Considering there are reasonable reasons to use these zfs commands would it be reasonable to implement them in the API for advanced users but leave them out of the WebUI to prevent whatever problem you are trying to avoid?

SinisterPisces · May 3, 2025, 10:22pm

Thanks for posting this. I’m out of votes, unfortunately.

I’d really like to start teaching myself how to use ZFS over iSCSI in Proxmox. I think it would really simplify my storage workflow for new VMs and LXCs.

Good luck on your webhooks-based plugin.

swc-phil · May 3, 2025, 10:48pm

I wonder whether ZFS over ISCSI naming will cause a shitstorm this time. I saw it once at the old forum…

BooMShanKerX · May 8, 2025, 4:36pm

I haven’t seen that but I think I’m going to have to check it out. Sounds spicey.

awalkerix · May 8, 2025, 8:04pm

It’s not a great naming convention since it sounds like an RFC-defined feature (e.g. an extension of the iSCSI protocol) rather managing ZFS through SSH or an API outside of the iSCSI protocol.

SinisterPisces · May 8, 2025, 8:24pm

The PVE devs would probably be open to a feature request for a name change if it’s confusing.

awalkerix · May 8, 2025, 8:27pm

It’s fairly obviously confusing. I know what iSCSI is, I know what ZFS is, but I did not know what “ZFS over iSCSI” is. If they had named it something like a “proxmox remote storage plugin” that would make sense. The naming smacks of marketing branding

BooMShanKerX · May 8, 2025, 8:31pm

Fair comment. The storage plugin technically manages both the zfs zvols AND the associated iscsi extents / LUNs. I’m just following the naming that the original plugin devs are using for consistency. The TrueNAS API allows for managing the iSCSI targets, but currently doesn’t provide raw zfs interaction. I certainly understand the risks. Having to setup root+ssh connectivity is inconvenient but manageable.

I’m not sure how big a market there is for Proxmox + TrueNAS but for those of us that use it, it works incredibly well.

My end goal is a pure TrueNAS storage plugin for Proxmox VE 8 completely driven by TrueNAS API

BooMShanKerX · June 15, 2025, 4:50pm

I’d like to bump this for a response from TrueNAS. I think all I need is zfs rename added to the pool.dataset API. With that I believe I could write a fully API driven plugin for Proxmox.

Stux · June 15, 2025, 11:13pm

Have you looked at

This tool is used to drive the Incus/TrueNAS storage driver

github.com/lxc/incus

Add TrueNAS Storage Driver

main ← truenas:truenas-storage-driver

opened 09:57AM - 29 May 25 UTC

mrstux

+4142 -53

The TrueNAS Storage driver allows an Incus node to utilise a remote TrueNAS stor…age server to host one or more incus storage pools. When the node is in a cluster, then all members of the cluster can use the storage pool simultaneously. This works well , for example, when live migrating a VM from one Incus node to another. The driver is block-based, ie all Incus volumes are created as zvols on the remote server, and the zvol block devices are accessed on the local incus node via iscsi The driver is modelled after the existing ZFS driver, and supports most existing ZFS functionality, but on a remote TrueNAS server. For example, a local VM can be snapshotted and then cloned. The snapshot and cloning will be performed on the remote server after synchronizing the local filesystem, and the clone will then be activated via iSCSI as necessary. The driver uses the `truenas_incus_ctl` tool, which provides a simple CLI interface to the TrueNAS API to effect changes on the remote server. Additionally, the tool can also activate/deactivate remote zvols using open-iscsi. _If the tool is not found, then the driver is disabled._ The latest version of the truenas tool can be obtained from : https://github.com/truenas/truenas_incus_ctl, and must be installed in the PATH supplied to incusd (v0.7.2+ is required) `open-iscsi` can be installed with `apt install open-iscsi` To login to the remote TrueNAS host first run the command: `sudo truenas_incus_ctl config login` Example: ``` $ sudo truenas_incus_ctl config login [sudo] password for user: Enter a name for this connection: example Enter the TrueNAS hostname or IP address: server.example.com Setting up connection to TrueNAS host: server.example.com Choose authentication method (1 for API Key, 2 for Username/Password): 2 Testing connection to wss://server.example.com/api/current... Enter your TrueNAS username: truenas_admin Enter your TrueNAS password: Generating API key... API key successfully generated Successfully connected to wss://server.example.com/api/current Configuration for 'example' (connecting to server.example.com) saved to /root/.truenas_incus_ctl/config.json ``` After logging in, ensure iSCSI is configured and working with the following command: `sudo truenas_incus_ctl share iscsi setup --test` As an example of the functionality of the TrueNAS tool, this command can be used to list all datasets on the remote server ``` sudo truenas_incus_ctl dataset ls ``` Once logged in and iSCSI has been configured, the following command can be used to create a remote storage pool `incus storage create <poolname> truenas source=[host:]<pool>/<dataset>/[remote-poolname]` `source` is used to describe the location where the storage pool will be created on the remote TrueNAS host `host` is optional, and `remote-poolname` will be set to `poolname` if not supplied. `truenas.config` can be used to specify a specific login config `truenas.host` can be used to specify a specific host, which the tool will lookup in the config.json **Running Tests** There are a number of `INCUS_TRUENAS_` environment variables that can be used to configure the remote TrueNAS that will be used by the driver during the test. The only environment variable that is required is `INCUS_TRUENAS_DATASET` I use the following commands in a script called `start-tests.sh` to run the suite using the default host configured in `truenas_incus_ctl` config file `sudo -E env PATH=$PATH INCUS_BACKEND=truenas INCUS_TRUENAS_DATASET=tank/se/incus-tests ./main-tn.sh $1` Alternatively, I specify a `config` for a different host. Hosts and API keys can also be specified manually to avoid the use of a config file. Alternate config files can be specified. `sudo -E env PATH=$PATH INCUS_VERBOSE=0 INCUS_TMPFS=0 INCUS_BACKEND=truenas INCUS_TRUENAS_DATASET=puddle/se/incus-tests INCUS_TRUENAS_CONFIG=goldeye2 ./main-tn.sh $1` I run the standalone tests using `./start-tests standalone`, as the `cluster` tests require net ns support from `truenas_incus_ctl` which is not implemented. `main-tn.sh` is the same as `main.sh`, except some tests which are problematic on my testing hosts are disabled. `main-tn.sh` should probably be deleted once the `./main.sh standalone` tests are verified When running the test `truenas_incus_ctl` will be launched in daemon mode. It can be useful to prelaunch the daemon to view its console as the test suite runs. for example: `sudo /home/<user>/go/bin/truenas_incus_ctl daemon /home/<user>/tncdaemon.sock` **Incus CI Integration** I would suggest that a VM with a "golden" image of GoldEye with the middleware defer patch is used as a target. The `truenas_incus_ctl` tool should be built from master and the standalone tests used. A preconfigured config.json for the tool could be used to provide the host/api-key to use for the tests. A container dataset should be created and removed after the test suite is run. **Known Issues** - *Creating and Deleting remote Volumes is relatively slow* due to current TrueNAS bottlenecks. Work is ongoing on resolving and hopefully will be incorporated in a future builds of Goldeye. https://github.com/truenas/middleware/pull/16614 - Requires TrueNAS Goldeye for `snapshot rename` capability (currently in Nightlies), although there is a patch that can be easily installed in Fangtooth (25.04) https://github.com/truenas/truenas_incus_ctl#zfssnapshotrename-backport - Cluster tests are not currently passing (requires net ns work-arounds), but cluster usage is supported. Use `standalone` tests - TrueNAS hosts don't currently support zvol-shrinking via API. The driver attempts to zvol-shrink via the tool, and then ignores the inevitable "shrink failure", but warns with the correct `zfs set` command required. If the API is ever changed, the tool will be updated to add support. - Optimized migrations between storage pools are not currently implemented.

It supports a simplified blocking mode of operation with automatic connection caching/aggregation

The CLI is modeled after the
zfs commands

Ie

tnc dataset create -V 10G pool/some-zvol

tnc dataset rename pool/some-zvol pool/better-name

BooMShanKerX · June 15, 2025, 11:58pm

Apparently they are using an undocumented API call to zfs.dataset.rename. I’ll have to experiment and see if I can get anything to work.

Captain_Morgan · June 29, 2025, 10:57pm

Please let us know the status and who needs any specific assistance.

You can private message me if you prefer.

BooMShanKerX · June 29, 2025, 11:36pm

If I can confirm API compatibility for a native Proxmox->TrueNAS storage plugin I will attempt to build it.

The existing plugin relies on some ssh+root zfs commands. If we can get support for all of the zfs commands we need in the API we can build a pure storage plugin instead of patching the existing ZFS over iSCSI provider.

Captain_Morgan · June 29, 2025, 11:41pm

If you have roadblocks… let us know.

SinisterPisces · June 30, 2025, 3:10am

Would it be useful to file a feature request/bug report listing off the undocumented API calls/ZFS commands that are needed by @BooMShanKerX 's plugin?

EDIT:
In case it’s not clear to anyone finding this thread later, @BooMShanKerX 's repo for the PVE storage plugin is here: GitHub - boomshankerx/proxmox-truenas: TrueNAS Websocket API Extension for Proxmox VE ZFS over iSCSI Storage

Captain_Morgan · June 30, 2025, 3:13am

I don’t think so… they are commited in 25.10

see api.truenas.com

BooMShanKerX · June 30, 2025, 3:27am

The needed API methods appear to have been added in version 25.04.2. Next week when I take my plugin out of beta I will begin re-evaluating the proxmox storage plugin and the TrueNAS API and verify.

I’m feeling optimistic

SinisterPisces · June 30, 2025, 4:55am

That’s awesome to hear.

I look forward to hearing how it goes.