Thunderbolt to 10Gbe Adapter

TrueNAS General
, ,
1

I’m trying to add a Thunderbolt to 10Gbe adapter to my ZimaCube Pro (it has an integrated 10 Gbe NIC) so that I can use SMB Multichannel to connect to a fast NVMe pool. However, the adapter isn’t seen by TrueNAS Scale 24.04.2.

The adapter is an IOCREST IO-HTB113-GLAN that uses an M.2 connection internally to a Marvell/Aquantia AQC113. It doesn’t show up at all in the console network configuration menu.

It works fine with macOS, with Proxmox, and with a TrueNAS Scale VM on Proxmox with both NICs virtualized. The performance of the VM is pretty close to bare metal but I’d prefer to run Scale on bare metal.

Any suggestions from the networking gurus as to why it doesn’t show up?

Coincidentally, the ZimaCube uses an M.2 10Gbe NIC for the integrated NIC as well.

2

Does Scale even ship with Thunderbolt support? I’m pretty sure it didn’t at first, and I’m not aware of that having charged.

3

You might have problems or issues with just plain 10Gbs NIC as preferred and recommended is Intel or Chelsio, per guides, etc. I have had problems with RealTek and Broadcom 1Gbps NICs and network resets while trying out Scale.

4

Thunderbolt Networking worked out of the box so Thunderbolt support is definitely included in some fashion. But TB is complex and I hate making unsanctioned changes to an “software appliance”.

As I mentioned, Proxmox picked it up immediately but it’s running a different kernel and my understanding that some Thunderbolt functionality had to be back-ported to the current Proxmox kernel. But that’s venturing into areas where my practical understanding is limited

I have a 3.0 x2 slot and a 3.0 x4 slot that are open but but the box has a reputation for cooling issues and so I’ve been looking for an SFP+ card but pickings are slim especially for Intel and Chelsio

6

I just felt the heatsink on my Intel X520 and I don’t think it is a fit for your use ‘case’. I think it needs a nice amount of airflow going over it.

7

Solarflare NICs have the reputation of being low power, and are also well supported by TrueNAS.
But, fundamentally, your Cube is is a weird beast, and does not like a good candidate for a NVMe NAS. Not enough PCIe lanes.

As for Thunderbolt, it might currently work, more or less, but this is not an official feature, is certainly not tested and might break at any point if iX removes or inadvertently breaks some part of Thunderbolt.

8

It has a “sled” (their word) that is holds four NVMe drives that sit behind a PCIe 3.0 x4 chip. If only one drive is being accessed, it gets full bandwidth x4 bandwidth, while if all four are being hammered, each drive gets x1 bandwidth. It also has the PCIe 4.0 x4 and PCIe x2 expansion slots so it would be pretty easy to have 10 NVMe drives.

I also have a Ugreen NAS 6800 Pro which can hold six NVMe drives. The layout is less flexible but it comes with two 10 Gbe NICs so 2x10Gbe SMB is easy. The ZimaCube has 2x2.5 Gbe + 1x10 Gbe. They both use the same CPU (i5-1235U) but made different choices on lane allocations.

One will be my primary NAS and my intent is to move most of the 10 TB of external NVMe storage on my Macs today to it without too noticeable a performance hit while the other will be a replication target for the primary. They are pretty equivalent except for expansion slots (the Ugreen NASync only has a single 3.0 x4), number and speed of “native” M.2 connections, and NICs). There are some relatively mechanical differences as well which effects the cooling.

So I’m in the “which box for which role” point of my project. The ZimaCube has more unknowns as I’d been banging on NASync for a month before the ZimaCube arrived plus the ZimaCube is a bit more “clever” in its layout and I’m always extra suspicious of “clever” as it usually has more known edge cases.

It’s a hobby not a job so I have more freedom over the choices I make (which is why I have lots and lots of backups).

9

I have one of those, and I see room for three–the OS and two others. Where do the other three go?

10

Sorry, my mistake. There are three on the system board and I’ve got a card that adds two more so call it five. But I should have been clearer that was with an expansion card. But it’s really four for me with two behind the removable cover and two on the card as I wouldn’t use the “boot” M.2 for a number of reasons.

11

I took closer look at what happens when the adapter is plugged in and dmesg says the system has identifies the adapter as an SSD rather than a network adapter perhaps because it’s using an m.2 adapter. Makes me wonder if I’d get a better result from one of the more mainstream adapters from OWC or Sonnet.

But I’ve already decided to utilize the ZimaCube as my secondary backup server where it mostly will talk to the primary server over the Thunderbolt Networking connection between the two boxes so I guess I’ll never know. Although I’ll check it with each new kernel that arrives to see if anything changes.

[147524.008914] thunderbolt 0-1: new device found, vendor=0x8086 device=0x1
[147524.010645] thunderbolt 0-1: Intel Thunderbolt3 LP SSD

12

Are you positive that that is an AQC113? Most such adapters come with the AQC107. Also, any idea what M.2 chp it uses?

13

It’s connected to a Mac Studio these days and macOS reports it as an AQC113

Aquantia AQC113:

Bus: PCI
Vendor ID: 0x1d6a
Device ID: 0x04c0
Subsystem Vendor ID: 0x1d6a
Subsystem ID: 0x0001
Revision ID: 0x0003
PCIe Link Speed: 8.0 GT/s
PCIe Link Width: x2
PCIe Slot: Thunderbolt@3,0,0
Firmware Version: 1.5.38
Driver: com.apple.driver.AppleEthernetAquantiaAqtion

Robbie on nascompares.com pulled it apart on a YouTube video and there was a long argument in the comments about the it that, like most of these arguments, settled nothing.

BSD Device Name: en20
MAC Address: 88:c9:b3:b3:10:ba
AVB Support: Yes
Maximum Link Speed: 10 Gb/s

14

Thanks. Now I remember: this is bottlenecked by the M.2 chip, which is actually an SSD M.2 chip. In other words, you won’t get more than 6Gbps in sustained transfers, in spite of the nice AQC113 chip–which, in this case, is just a waste.

Care to do some testing? Show us some benchmarks? What bandwidth are you routinely getting?

15

It can’t be an SSD M.2 chip because I get full 10Gbe speeds from both the ones I have. I use them constantly for 2x10Gbe SMB Multichannel connections to my Ugreen Pro 6800 Pro running TrueNAS Scale from my Mac Studio and my Mac Mini 2018.

I have a flat network with multiple overlayed subnets. The NICs in the Mini are the main network (192.168.0.x) and the IOCREST adapters are on a secondary network (192.168.10.x).

I just ran this iperf3 between the between the two Macs using 192.168.10.x subnet with these results. I also ran iperf3 against the 10 Gbe NIC on my ZimaCube Pro with exactly the same results.

[ 4] local 192.168.10.152 port 49452 connected to 192.168.10.163 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.09 GBytes 9.39 Gbits/sec
[ 4] 1.00-2.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 2.00-3.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 3.00-4.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 4.00-5.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 6.00-7.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 7.00-8.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 8.00-9.00 sec 1.10 GBytes 9.41 Gbits/sec
[ 4] 9.00-10.00 sec 1.10 GBytes 9.41 Gbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec sender
[ 4] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec receiver

1 Like
16

What is Robbie talking about then? (See the UPDATE at the bottom.)

17

Did you install scale with the adapter attached?

18

I did not. I suspect, but do not know for a fact, that it works with some versions of some distributions since their web site says so.

I figure I’ll just occasionally plug it in occasionally and see if the drivers get integrated. I don’t think iXsystems pays much attention to TB since it’s more hobbyist niche

19

More accurately, iX pays no attention at all to Thunderbolt since it is a desktop-class interface and TrueNAS is a server.

20

I can answer this one by this ticket which has been directly closed by TrueNAS after shipping the necessary information (they don’t have time for this, so this will not be fixed, unless we are lucky with another kernel module that is going to be shipped by default).
https://ixsystems.atlassian.net/browse/NAS-130969

I am still running on 23.10, but perhaps I will move over to Proxmox soon, because this is annoying.

The Thunderbolt 10G Ethernet device does work in TrueNAS Scale 23.10 but updating to later version will break the driver support.

Perhaps recompiling the kernel driver / module with an older version will work, but then you need to apt-get stuff, which is not supported by the storage appliance (nor you should do this normally). Not sure why the new Linux kernel has no support for this specific network interface anymore. See technical output below.

OWC Thunderbolt 3 10G Ethernet not working in TrueNAS Scale 24.04 or 24.10 but works in 23.10.2

Apps
Description
After upgrade from Scale 23.10.2 to Scale 24.04.2 or Scale 24.10.BETA.1 the network ethernet interface stops working.

Atlantic driver is not working properly anymore for OWC Thunderbolt 3 10G Ethernet device and ails with a bad FW message.

Within Scale 24.04.2 or 24.10.BETA.1

Sep 3 00:11:02 truenas kernel: thunderbolt 1-0:1.1: new retimer found, vendor=0x8087 device=0x15ee
Sep 3 00:11:03 truenas kernel: thunderbolt 1-1: new device found, vendor=0x5a device=0xde35
Sep 3 00:11:03 truenas kernel: thunderbolt 1-1: Other World Computing Thunderbolt 3 10G Ethernet Adapter

root@truenas[/]# dmesg | grep atlantic
[ 1.846884] atlantic 0000:2e:00.0: enabling device (0000 → 0002)
[ 2.137879] atlantic: Bad FW version detected: ffffffff
[ 2.138027] atlantic: probe of 0000:2e:00.0 failed with error -95

The device is not listed in ifconfig -a anymore.

While in Scale 23.10.2 I am getting the following output and the network interface is working properly:

root@truenas[/home/admin]# dmesg | grep atlantic
[ 1.916111] atlantic 0000:2e:00.0: enabling device (0000 → 0002)
[ 2.372096] atlantic 0000:2e:00.0 enp46s0: renamed from eth0
[ 34.960884] atlantic 0000:2e:00.0 enp46s0: atlantic: link change old 0 new 10 000

root@truenas[/home/admin]# dmesg | grep thunderbolt
[ 1.858388] ACPI: bus type thunderbolt registered
[ 4.540980] thunderbolt 1-0:1.1: new retimer found, vendor=0x8087 device=0x15ee
[ 5.352627] thunderbolt 1-1: new device found, vendor=0x5a device=0xde35
[ 5.354528] thunderbolt 1-1: Other World Computing Thunderbolt 3 10G Ethernet Adapter
root@truenas[/home/admin]#

root@truenas[/home/admin]# ethtool -i enp46s0
driver: atlantic
version: 6.1.74-production+truenas
firmware-version: 3.1.106
expansion-rom-version:
bus-info: 0000:2e:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes
root@truenas[/home/admin]#

Steps to Reproduce

upgrade from 23.10.2 to a later TrueNAS Scale version
Expected Result

working network interface as being possible in 23.10.2.
Actual Result

no network interface.

root@truenas[/]# dmesg | grep atlantic
[ 1.846884] atlantic 0000:2e:00.0: enabling device (0000 → 0002)
[ 2.137879] atlantic: Bad FW version detected: ffffffff
[ 2.138027] atlantic: probe of 0000:2e:00.0 failed with error -95

21

Thanks.

That’s unfortunate but not a totally unexpected outcome. Limited resources mean hard decisions about priorities and I can understand why external Thunderbolt 10Gbe adapters are not a priority item for ixSystems.