I have a couple NVMe in my TrueNas Scale Server. Doing an Fio test I get about 6000MB/s to one drive. Connecting up Vmware via iSCSI software adapter only gives me about 1000MB/s. I was reading iSCSI over RDMA (iSER) Adapter would be better. I enabled it on the ESX server, added nics to the port binding, added the TrueNAS server ip to the discovery. But the storage does not show up. On the Truenas side I do not see it connecting. Do I need to enable anything different on the iSCSI on Truenas for it to work with RDMA?
You really need to describe your entire setup, hardware and how you are testing. I can’t tell if you are talking about separate servers or virtual.
TrueNAS does not support RDMA at this time. But as said above, to give you better advise, we’d need to know a bit more.
I have a Gigabyte NVMe Server running TrueNas Scale. An Intel E810 NIC (Im not sure if i need to install RDMA drivers for this) 3x NVME Micron 9400 drives. I am only currently using 1 drive to test with.
Vmware 8 I am using a Cisco UCS Server with VIC VIC 15231. RDMA drivers installed and enabled it in the CIMC. It currently shows up under Network RDMA Adapters. Under Controllers I added the iSCSI over RDMA (iSER) and enabled NVMe RDMA support on the network interfaces on the Vswitch.
Current setup using the ISCSI software adapter works and connects to the TrueNAS server. When I add the ip to the iSCSI over RDMA (iSER) Adapter and remove it from the ISCSI Software Adapter the Datastores disappear. I rescan and nothing. I look on the TrueNAS Shared settings and I do not see it connected.
The reason I was looking to switch from VMware’s software iSCSI to iSCSI over RDMA is because when speed testing using iSCSI the fastest I would get is about 1500-2000MB/s. When I mount it as a SMB share I get close to 3500MB/s. If there is a way to increase that performance that would be nice.
I’m not sure which would be better to use. In the long run I want to stripe a couple drives together for higher iops.
I also saw a different thread with performance on iSCSI. Tiebird posted seeing better numbers on vanilla Ubuntu on same hardware (21st post)
I saw a similar post earlier but it looks like they recommended switching to iSCSI over RDMA. It says it’s supported in TrueNAS but my server doesn’t connect to TrueNAS using iSCSI over RDMA. The directions I followed shows to to create an Iscsi share config the esx host to enable RDMA but nothing in enabling it on TrueNas
There is no support for this in TrueNAS.
Here’s a few Jira examples to support that claim:
https://ixsystems.atlassian.net/jira/software/c/projects/NAS/issues/NAS-106190
https://ixsystems.atlassian.net/jira/software/c/projects/NAS/issues/NAS-122936
Where did you read it was supported?
Perhaps Mellanox drivers happen to include code for it but the Intel ones do not? Either way, it looks like an unsupported feature.
What does the dmesg command in the linked guide show after configuring the iSCSI target?
How did this go? Get it working?