Test with an other 10gbe Amazon Nic (amazon links below for parts used) :
Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] (rev 03)
It works with Truenas VM in Proxmox.
Problems solved :
1/ The wire
The wire that connects the two parts of the NIC (ethernet connector part / M.2 part) : with the first card it never connected well, with the second the connector is better and connects well, i immediatly see the green light on the NIC when system starts.
2/ Truenas vs Proxmox
In Truenas system you can’t do much tuning, so i installed Proxmox and some tools to do linux tests :
nala cockpit+navigator btop ncdu intel-gpu-tools tmux iperf3 iftop hwloc lm-sensors.
3/ The oveheating : +102.0°C on the M.2 NIC with the default heatsink !!!
After seeing +102.0°C on the M.2 10gbe NIC, i immediatly shut down the Beelink Me Mini and added active heatsync.
Temperatures with the default heatsync :
sudo nala install lm-sensors
while true; do clear; sensors; sleep 1; done
doge@pxm:~$ sensors
iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: N/A
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +57.0°C (high = +105.0°C, crit = +105.0°C)
Core 0: +55.0°C (high = +105.0°C, crit = +105.0°C)
Core 1: +55.0°C (high = +105.0°C, crit = +105.0°C)
Core 2: +55.0°C (high = +105.0°C, crit = +105.0°C)
Core 3: +55.0°C (high = +105.0°C, crit = +105.0°C)
enp5s0-pci-0500
Adapter: PCI adapter
PHY Temperature: +102.0°C
MAC Temperature: +102.0°C
After adding active heatsync : +53.0°C (after 1 hour test with BlackMagic Disk Speed Test - writing on the NFS share continuously)
doge@pxm:~$ sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +75.0°C (high = +105.0°C, crit = +105.0°C)
Core 0: +75.0°C (high = +105.0°C, crit = +105.0°C)
Core 1: +75.0°C (high = +105.0°C, crit = +105.0°C)
Core 2: +75.0°C (high = +105.0°C, crit = +105.0°C)
Core 3: +75.0°C (high = +105.0°C, crit = +105.0°C)
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C
iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: N/A
enp5s0-pci-0500
Adapter: PCI adapter
PHY Temperature: +53.0°C
MAC Temperature: +53.0°C
Temp few minutes after test stop :
doge@pxm:~$ sensors
coretemp-isa-0000
Adapter: ISA adapter
Package id 0: +60.0°C (high = +105.0°C, crit = +105.0°C)
Core 0: +60.0°C (high = +105.0°C, crit = +105.0°C)
Core 1: +61.0°C (high = +105.0°C, crit = +105.0°C)
Core 2: +61.0°C (high = +105.0°C, crit = +105.0°C)
Core 3: +61.0°C (high = +105.0°C, crit = +105.0°C)
acpitz-acpi-0
Adapter: ACPI interface
temp1: +27.8°C
iwlwifi_1-virtual-0
Adapter: Virtual device
temp1: N/A
enp5s0-pci-0500
Adapter: PCI adapter
PHY Temperature: +53.0°C
MAC Temperature: +53.0°C
4/ Performance
iperf3 gave 8gbe up and down
BlackMagic write to NFS share with dataset SYNC=OFF and 4 M.2 (Crucial T500 4tb - 2 mirrors of 2 NVME) : 318 MB/s write 624 MB/s read (not full 10gbe, bottleneck somewhere ?)
5/ Tuning
Also i modified IRQ and RX/TX in proxmox :
# IRQ -----------------------------------------------
sudo nano /etc/systemd/system/my-irq-affinity.service
[Unit]
Description=Set IRQ affinity for enp5s0
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/my-irq.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
sudo nano /usr/local/bin/my-irq.sh
#!/bin/bash
NIC="enp5s0"
CPU_COUNT=$(nproc)
echo "Detected $CPU_COUNT CPU cores."
# Get IRQs associated with the NIC
IRQS=$(grep $NIC /proc/interrupts | awk '{print $1}' | tr -d ':')
echo "Setting IRQ affinity for $NIC..."
i=0
for irq in $IRQS; do
cpu=$((i % CPU_COUNT))
mask=$((1 << cpu))
printf -v hex_mask "%x" $mask
echo $hex_mask | sudo tee /proc/irq/$irq/smp_affinity > /dev/null
echo "IRQ $irq assigned to CPU$cpu (mask 0x$hex_mask)"
((i++))
done
echo "IRQ affinity updated successfully."
sudo chmod +x /usr/local/bin/my-irq.sh
sudo systemctl daemon-reexec
sudo systemctl enable my-irq-affinity.service
# RX / TX : FIX -----------------------------------------------
sudo ethtool -G enp5s0 rx 8184 tx 8184
service :
sudo nano /etc/systemd/system/my-ethtool-buffer.service
[Unit]
Description=My : Set ethtool buffer sizes for enp5s0
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/sbin/ethtool -G enp5s0 rx 8184 tx 8184
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reexec
sudo systemctl enable my-ethtool-buffer.service
# test
sudo systemctl start my-ethtool-buffer.service
# reboot + check
sudo ethtool -g enp5s0
# test
pxm : iperf3 -s -B 192.168.50.141
mac :
iperf3 -c 192.168.50.141 -P 4 -t 10
iperf3 -c 192.168.50.141 -P 4 -t 10 -R
6/ My Bios Settings
https://drive.google.com/drive/folders/1qKy8X5LHsY8KTqGCWHt1RucNrNHCf933?usp=sharing
7/ Misc
lspci :
doge@pxm:~$ lspci
00:00.0 Host bridge: Intel Corporation Alder Lake-N Processor Host Bridge/DRAM Registers
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [Intel Graphics]
00:04.0 Signal processing controller: Intel Corporation Alder Lake Innovation Platform Framework Processor Participant
00:08.0 System peripheral: Intel Corporation GNA Scoring Accelerator
00:0d.0 USB controller: Intel Corporation Alder Lake-N Thunderbolt 4 USB Controller
00:14.0 USB controller: Intel Corporation Alder Lake-N PCH USB 3.2 xHCI Host Controller
00:14.2 RAM memory: Intel Corporation Alder Lake-N PCH Shared SRAM
00:14.3 Network controller: Intel Corporation CNVi: Wi-Fi
00:15.0 Serial bus controller: Intel Corporation Device 54e8
00:15.1 Serial bus controller: Intel Corporation Device 54e9
00:16.0 Communication controller: Intel Corporation Alder Lake-N PCH HECI Controller
00:1a.0 SD Host controller: Intel Corporation Device 54c4
00:1c.0 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port #1
00:1c.1 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port #2
00:1c.3 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port #4
00:1c.6 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port #7
00:1d.0 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port #9
00:1d.2 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port #11
00:1d.3 PCI bridge: Intel Corporation Alder Lake-N PCI Express Root Port #12
00:1e.0 Communication controller: Intel Corporation Alder Lake-N Serial IO UART Host Controller
00:1e.3 Serial bus controller: Intel Corporation Device 54ab
00:1f.0 ISA bridge: Intel Corporation Alder Lake-N PCH eSPI Controller
00:1f.3 Audio device: Intel Corporation Alder Lake-N PCH High Definition Audio Controller
00:1f.4 SMBus: Intel Corporation Alder Lake-N SMBus
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-N SPI (flash) Controller
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
03:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01)
04:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01)
05:00.0 Ethernet controller: Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] (rev 03)
06:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01)
07:00.0 Non-Volatile memory controller: Micron/Crucial Technology T500 NVMe PCIe SSD (rev 01)
Parts used :
The NIC :
Active heatsink :
USB power for heatsink :
M.2 extension :
Proxmox : network + VM config
iperf3 + BlackMagic writing from the Mac Mini M4 with 10gbe port to the Beelink Me Mini with 10gbe M.2 adaper, via NFS share with SYNC=OFF
M2 10gbe NIC with active cooling (i used tape to secure the fragile cable) :
BalckMagic test 1 hour writing to NFS share via 10gbe NIC with active heatsync :
iPerf3 :
BlackMagic :
Stogare config : “fast” dataset with SYNC=OFF + 2 mirrors of 2 MNVME Crucial T500 :
Truenas Dashboard :
