With the upcoming 25.10 TrueNAS release, I noticed that NVMe over TCP is now available in RC. It’s an interesting technology, so I decided to test it in my small homelab.
After some research, reading the documentation, torturing chatgpt and experimenting, I found that the basic setup is actually straightforward. However, securing it properly is more complex. I organized my notes and sample configurations here and also published them on GitHub, which may be more convenient to use than copying from the forum.
I hope this post eventually would be useful to someone. Feedback or corrections from more experienced users are very welcome.
Please also check official documentation https://www.truenas.com/docs/scale/25.10/scaletutorials/shares/nvme-of/ . It contains more details and UI screenshots for setup on the NAS side.
Disclaimer
- This guide is provided without any guarantees, especially regarding security options.
- The author is new to NVMe over Fabrics, VLANs, security and network namespaces and is exploring this field.
Software and Package Installation
The configuration was tested with the following software versions:
- TrueNAS: 25.10 RC1 (first release with NVMe over TCP support)
- Linux kernel: 6.14.0-33-generic (Ubuntu 24.04.1)
- nvme-cli: 2.8 (libnvme 1.8)
Older versions of nvme-cli and libnvme may also work, but compatibility is not guaranteed.
Install the required packages on the client host:
sudo apt-get update
sudo apt-get install -y nvme-cli iproute2 net-tools
Basic Setup
This unsecured configuration is quick and simple to deploy. It is suitable for testing performance or use within a fully trusted, isolated network. For any production or less controlled environment, consider security measures and check the “Secured Setup” section.
- Ensure TrueNAS has a static IP. If unsure, check System → Network.
- Navigate to System → Services and enable the NVMe-oF service.
- Go to Sharing → NVMe-oF Subsystems → Add in the TrueNAS UI.
- In the wizard:
- Provide a name.
- Under Namespace, add the previously created Zvol, then save.
- Click Next, Add, and select Create New.
- Select your static IP.
- Finalize the Add Subsystem wizard by clicking Save.
- Copy the NQN of the created namespace.
- Create an
.envconfig file on the client host at/etc/nvme/nvme-basic.envand fill in the NAS IP and NQN from the previous step:
NVME_NAS_IP=
NVME_NAS_PORT=4420
NVME_NAS_NQN=
(You can also clone the repository and run sudo cp nvme-basic.env /etc/nvme/, then edit the file afterward.)
- Create a systemd service:
[Unit]
Description=Connect NVMe over TCP
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/etc/nvme/nvme-basic.env
ExecStartPre=/usr/bin/modprobe nvme-tcp
ExecStart=/usr/sbin/nvme connect -t tcp -a ${NVME_NAS_IP} -s ${NVME_NAS_PORT} -n ${NVME_NAS_NQN}
ExecStop=/usr/sbin/nvme disconnect -n ${NVME_NAS_NQN}
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
(Alternatively, copy the file using sudo cp nvme-basic.service /etc/systemd/system/.)
- Start the service and verify NVMe connectivity:
systemctl start nvme-basic
nvme list
nvme list-subsys /dev/{device_name}
- Enable the service for persistence after reboot:
systemctl enable nvme-basic
That’s all! You now have an NVMe device that behaves like a local one.
Secured Setup
Key points:
- TrueNAS provides authorization keys for NoT, which are used here to restrict access to the specific client host.
- All communication between the NAS and client is isolated within a dedicated VLAN.
- To forbid non-root processes access to network traffic, the VLAN interface on the client is placed inside a dedicated network namespace.
Client Host Configuration and Service File
Generate the client NQN and copy the provided service file, auxiliary bash script, and .env file containing variables and secrets:
nvme-connect.service
[Unit]
Description=Connect NVMe over TCP in isolated namespace
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
EnvironmentFile=/etc/nvme/nvme-secret.env
ExecStartPre=/usr/local/sbin/nvme-netns-setup.sh
ExecStart=/usr/bin/ip netns exec ${NVME_NS} /usr/sbin/nvme connect \
-t tcp -a ${NVME_NAS_IP} -s ${NVME_NAS_PORT} -n ${NVME_NAS_NQN} \
--dhchap-secret=${NVME_CLIENT_KEY} \
--dhchap-ctrl-secret=${NVME_NAS_KEY}
ExecStop=/usr/bin/ip netns exec ${NVME_NS} /usr/sbin/nvme disconnect -n ${NVME_NAS_NQN}
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
nvme-netns-setup.sh
#!/bin/bash
set -euo pipefail
/usr/bin/modprobe nvme-tcp
CONF=/etc/nvme/nvme-secret.env
[ -f "$CONF" ] && source "$CONF"
# Sanity checks
: "${NVME_NS:?Missing NVME_NS}"
: "${NVME_IF:?Missing NVME_IF}"
: "${NVME_CLIENT_IP:?Missing NVME_CLIENT_IP}"
# Create namespace if not exists
if ! ip netns list | grep -q "^${NVME_NS}\b"; then
ip netns add "$NVME_NS"
fi
# Move VLAN interface to the namespace if not already there
if ip link show "$NVME_IF" 2>/dev/null | grep -q "$NVME_IF"; then
ip link set "$NVME_IF" netns "$NVME_NS"
fi
# Configure inside namespace
ip netns exec "$NVME_NS" bash <<EOF
set -e
ip link set lo up
ip link set "$NVME_IF" up
ip addr show "$NVME_IF" | grep -q "$NVME_CLIENT_IP" || ip addr add "$NVME_CLIENT_IP"/24 dev "$NVME_IF"
ip route show default | grep -q "$NVME_GW" || ip route add default via "$NVME_GW"
EOF
nvme-secret.env
NVME_NS=nvme-network-namespace
NVME_IF=eth0.50
NVME_VLAN_ID=50
NVME_GW=192.168.50.1
NVME_CLIENT_IP=192.168.50.20
NVME_NAS_IP=192.168.50.10
NVME_NAS_PORT=4420
NVME_NAS_NQN=<GENERATED VALUE FROM TRUENAS>
NVME_CLIENT_KEY=<KEY FROM TRUENAS FOR CLIENT AS PRESENTED>
NVME_NAS_KEY=<KEY FROM TRUENAS IDENTIFYING IT AS PRESENTED>
nvme gen-hostnqn > /etc/nvme/hostnqn
cp ./nvme-netns-setup.sh /usr/local/sbin/nvme-netns-setup.sh
cp ./nvme-secret.env /etc/nvme/nvme-secret.env
cp ./nvme-connect.service /etc/systemd/system/nvme-connect.service
chmod 700 /usr/local/sbin/nvme-netns-setup.sh
chmod 600 /etc/nvme/nvme-secret.env
Adjust the network configuration (IP addresses, VLAN ID, etc.) in /etc/nvme/nvme-secret.env according to your network setup. The NAS NQN and security keys should be filled in after completing the TrueNAS configuration.
Client Host Network Setup
Stable Network Interface Name
Modern Linux systems use interface names based on firmware or hardware properties, which can differ from traditional ethX naming. To maintain consistency for configuration, explicitly assign a persistent name to the physical interface (e.g., eth0).
Create a udev rule file with the MAC address of the interface:
sudo vim /etc/udev/rules.d/70-persistent-net.rules
Add line like:
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="aa:bb:cc:dd:ee:00", NAME="eth0"
Add a rule specifying the desired name and MAC address. Then reload udev rules and reboot:
sudo udevadm control --reload-rules
sudo reboot
VLAN Interface Creation
First we need to create a VLAN interface.
In my case NetworkManager managed networking, so using it to create a new interface.
nmcli connection add type vlan con-name "vlan${NVME_VLAN_ID}" dev eth0 id ${NVME_VLAN_ID} ip4 ${NVME_CLIENT_IP}/24
Otherwise please consult your distribution documentation.
TrueNAS NVMe-oF Setup
- Navigate to System → Network to configure VLAN and assign a static IP.
- Create a Zvol with the desired size and properties.
- Navigate to Sharing → NVMe-oF Subsystems → Add in the TrueNAS UI.
- In the wizard:
- Provide a name for the NVMe share.
- Under Namespace, add the previously created Zvol.
- In Access Settings, untick “Allow any host to connect”.
- To restrict access to the client host:
- Click Allowed Hosts → Add → Create New.
- Enter the host NQN (from
/etc/nvme/hostnqn). - Tick Require Host Authentication.
- Generate both keys and save them to the client
/etc/nvme/nvme-secret.env.
- To limit access to the dedicated VLAN, add the VLAN port.
- Save the configuration.
- If NVMe-oF Subsystems is not running, enable it when prompted.
- Notice generated NQN for “share” in the TrueNAS view and store it as NVME_NAS_NQ in
/etc/nvme/nvme-secret.env.
[Optional] Check setup halfway
See network interface in dedicated namespace
sudo ip netns exec $NVME_NS ip a
Discover announced nvme namespaces
sudo ip netns exec nvme-ns nvme discover -t tcp -a $NVME_NAS_IP -s $NVME_NAS_PORT
Enable NVMe Connection Service
Start the service manually to test the connection and verify the NVMe device:
sudo systemctl start nvme-connect
sudo systemctl status nvme-connect
nvme list
Enable the service to start automatically at boot:
sudo systemctl enable nvme-connect
Filesystem Creation and Mounting
A newly connected NVMe device can be used like a local device in various ways. Here, it is shown being used for a single ext4 filesystem and mounted via an fstab entry.
- Identify the newly connected NVMe device using
nvme list. - Create an ext4 filesystem:
sudo mkfs.ext4 -E lazy_itable_init=0 /dev/{nvme dev name}
- Disable journaling, relying on ZFS on the NAS:
sudo tune2fs -O ^has_journal /dev/{nvme dev name}
- Create a mount point for the desired user:
mkdir -p ~/nvme_mount
- Retrieve the filesystem UUID:
blkid | grep {nvme dev name}
- Add an entry to
/etc/fstabusing the UUID bysudo vim /etc/fstab:
UUID={new fs uuid} /home/{user}/nvme_mount ext4 defaults,nofail,_netdev,user,auto,noatime,nodiratime,barrier=0 0 2
- Check mount as the user:
mount ~/nvme_mount
Finally try to reboot, nvme and fs should be mounted automatically.
