Proxmox Add ZFS over iSCSI parameters with iSCSI served from truenas scale

I’m soooo close to geting ZFS over iSCSI to work on proxomox.

To do ZFS over iSCSI, proxmox needs ssh root access to scale…

So I added my ed25519 public key in root credentials

Then I created the ed25519 private key in proxmox but I had to name it “rsa” since it only looks for the rsa key.

So you put the private key in /etc/pve/priv/zfs/192.168.1.115_id_rsa because that’s the only key it will use. so it’s mislabelled (since it’s an ed25519 key), but that doesn’t matter. It works.

image

I used “Modern storage” when creating the volume on the truenas side.

The HUGE clue was picking the new zfs storage and then picking the vm disks and that told me that it was failing on the ssh command and it was only trying the rsa key.

I’m close but no cigar yet. Get error message:
image

Here is my storage.cfg file:

zfs: iscsi-test-2
        blocksize 4k
        iscsiprovider LIO
        pool main
        portal 192.168.1.115
        target iqn.2005-10.org.freenas.ctl:iscsi
        content images
        lio_tpg tpg2
        nowritecache 1
        sparse 0

Note

  1. You must use provider=LIO
  2. the tpg has to have tpg before the number, you can just say 2, you have to say: tpg2

I get the error message because targetcli simply doesn’t exist on Truenas scale.

Either this is an oversight, or it is done for security reasons so Scale is managing things.

So I’m I’ve hit a brick wall that I’m supposed to hit and should just use iSCSI without the ZFS abilities like this which is a piece of cake because as soon as you type the hostname, you get a list of all the targets!

ZFS over iSCSI just sounds like marketing-speak for “have proxmox SSH into your NAS and make unsupported configuration changes (possibly breaking ones) via shell that could easily be done by just creating an iSCSI LUN via the TrueNAS UI”.

@HoneyBadger I vaguely recall that we had users try to do this stuff and ended up with zvols that couldn’t be deleted via UI / API. Does that sound right?

2 Likes

Proxmox’s “ZFS over iSCSI” plugin doesn’t support the iSCSI targets used in TrueNAS (scst in SCALE, ctl in CORE) - and to muddy the waters, by enabling the SSH access directly from Proxmox it might have accidentally made some unwanted changes on your TrueNAS machine as @awalkerix alludes to. Might be worth looking at the output of zfs history from a shell on your TrueNAS machine and see if it’s been creating new zvols or datasets.

For sharing to Proxmox, you can either configure LVM on a shared iSCSI extent, manually provision the LUNs yourself, or use a community-created Proxmox plugin:

2 Likes

yup, that’s kind of what I figured from the inability to run targetcli :slight_smile:

I think you meant zpool history since zfs history doesn’t exist.

You were correct about that!

2024-10-09.00:02:22 py-libzfs: zfs create -o org.truenas:managedby=192.168.1.22 -o volblocksize16384 -o volsize2147483648 main/vm/iscsi
2024-10-09.01:37:17 zfs create -b 4k -V 1048576k main/vm-101-disk-0
2024-10-09.01:46:08 zfs create -b 4k -V 1048576k main/vm-101-disk-1
2024-10-09.01:48:39 zfs create -b 4k -V 1048576k main/vm-101-disk-2
2024-10-09.01:54:32 zfs create -b 4k -V 1048576k main/vm-101-disk-3
2024-10-09.02:08:35 zfs create -b 4k -V 1048576k main/vm-101-disk-4