Hi folks,
I’m using a single zvol via iSCSI as the VHD store for my 4-node USFF Proxmox cluster. Physically, each node has a 2.5Gb USB NIC connected to a dedicated 2.5Gb+10Gb unmanaged switch, then via a single 10Gb connection to the TrueNAS machine. All 4 nodes are hitting the same IP address on TrueNAS (192.168.4.1). Public traffic flows on different interfaces both on the PVE nodes and the TrueNAS system; iSCSI is dedicated.
A while ago, I was having trouble setting up FreeIPA integration so I looked in /var/log/daemon.log, and was surprised to see excessive iSCSI connections - each node is initiating a connection every few seconds. This is still going today:
Apr 14 12:23:28 poseidon iscsi-scstd[3692]: Connect from 192.168.4.2:44482 to 192.168.4.1:3260
Apr 14 12:23:31 poseidon iscsi-scstd[3692]: Connect from 192.168.4.5:54188 to 192.168.4.1:3260
Apr 14 12:23:31 poseidon iscsi-scstd[3692]: Connect from 192.168.4.4:58794 to 192.168.4.1:3260
Apr 14 12:23:34 poseidon iscsi-scstd[3692]: Connect from 192.168.4.3:56892 to 192.168.4.1:3260
Apr 14 12:23:38 poseidon iscsi-scstd[3692]: Connect from 192.168.4.2:34292 to 192.168.4.1:3260
Apr 14 12:23:41 poseidon iscsi-scstd[3692]: Connect from 192.168.4.5:50360 to 192.168.4.1:3260
Apr 14 12:23:41 poseidon iscsi-scstd[3692]: Connect from 192.168.4.4:60872 to 192.168.4.1:3260
Apr 14 12:23:44 poseidon iscsi-scstd[3692]: Connect from 192.168.4.3:39570 to 192.168.4.1:3260
Apr 14 12:23:47 poseidon iscsi-scstd[3692]: Connect from 192.168.4.2:45988 to 192.168.4.1:3260
Apr 14 12:23:51 poseidon iscsi-scstd[3692]: Connect from 192.168.4.5:54150 to 192.168.4.1:3260
Apr 14 12:23:52 poseidon iscsi-scstd[3692]: Connect from 192.168.4.4:47090 to 192.168.4.1:3260
Apr 14 12:23:53 poseidon iscsi-scstd[3692]: Connect from 192.168.4.3:57936 to 192.168.4.1:3260
Apr 14 12:23:57 poseidon iscsi-scstd[3692]: Connect from 192.168.4.2:55588 to 192.168.4.1:3260
Apr 14 12:24:01 poseidon iscsi-scstd[3692]: Connect from 192.168.4.5:51332 to 192.168.4.1:3260
Apr 14 12:24:02 poseidon iscsi-scstd[3692]: Connect from 192.168.4.4:58422 to 192.168.4.1:3260
Apr 14 12:24:04 poseidon iscsi-scstd[3692]: Connect from 192.168.4.3:40854 to 192.168.4.1:3260
Apr 14 12:24:07 poseidon iscsi-scstd[3692]: Connect from 192.168.4.2:45538 to 192.168.4.1:3260
Apr 14 12:24:11 poseidon iscsi-scstd[3692]: Connect from 192.168.4.5:43904 to 192.168.4.1:3260
Apr 14 12:24:12 poseidon iscsi-scstd[3692]: Connect from 192.168.4.4:58610 to 192.168.4.1:3260
Apr 14 12:24:14 poseidon iscsi-scstd[3692]: Connect from 192.168.4.3:36808 to 192.168.4.1:3260
Apr 14 12:24:18 poseidon iscsi-scstd[3692]: Connect from 192.168.4.2:53502 to 192.168.4.1:3260
Apr 14 12:24:21 poseidon iscsi-scstd[3692]: Connect from 192.168.4.5:42656 to 192.168.4.1:3260
Apr 14 12:24:22 poseidon iscsi-scstd[3692]: Connect from 192.168.4.4:37502 to 192.168.4.1:3260
Apr 14 12:24:24 poseidon iscsi-scstd[3692]: Connect from 192.168.4.3:41080 to 192.168.4.1:3260
Apr 14 12:24:28 poseidon iscsi-scstd[3692]: Connect from 192.168.4.2:45234 to 192.168.4.1:3260
There’s nothing in the logs for iscsid on the hypervisors to indicate abnormal behaviour.
I thought iSCSI connections were supposed to be persistent, and I’m not finding much online to say whether or not this is correct behaviour. Seems like a performance limitation if it keeps closing and re-opening the iSCSI connection that all my VMs/CTs are sharing. At the very least, it’s writing an awful lot of logs to the system SSD every day. Anyone have any insight?
Proxmox VE, also being Debian, is using the Open-iSCSI initiator, v2.1.8. I’m running v8.1.10 on all nodes.
My TrueNAS Scale machine is running 24.04-RC1. It has an Intel Celeron N5100 processor with 16GB of DDR4 (12GB maximum for ZFS ARC) and a dual-port Intel X520 10Gb NIC.
Thanks!