Try testing with Debian… same issue?
They advertise Linux compatibility with a driver so I suspect any recent vanilla Linux distribution would be fine. Unfortunately, I don’t have any spare HW with a TB port to use to test.
Yes. And if that were the case you’d expect it to work in TrueNAS as well which is using a fairly up to date version Linux Kernel 6.6.
Since it doesn’t you need to prove that it works on kernel 6.6 on Debian, or a newer kernel etc. or that it doesn’t. TrueNAS Scale is based on Debian.
You could install Debian on a different boot disk, and use the NAS…
that is something I can try as well, since it is a Thunderbolt device.
Let us see if it might work on a Live image in Debian.
If not, then the issue needs to be fixed here as well.
Where did you end up with this?
I got mine working, but it was overcomplicated and not obvious.
I moved on to other things as it was for a backup TNS server that already had one 10Gbe NIC. I decided that life would be better if I avoided unsanctioned changes to my TNS servers.
I do love hearing about non-obvious, complicated solutions though…
I’d be interested in anyone who’s got some time to test the latest nightlies, they have kernel 6.12 and this may “just work”.
tl;dr I can make it work, but getting it to work reliably for more than a few hours at a time involves enough rigamarole that I probably can’t actually do it on this particular server with SCALE.
Shorter version:
- You need to be able to control Thunderbolt DMA and ASPM on the motherboard; or
- You’re probably not getting across the finish line because Scale makes doing these sorts of low-level kernel/boot tweaks confusing and scary.
(That’s not a knock on Scale or iX. Locking down the kernel to the degree they have is a sane design decision for an appliance OS. It just happens to make troubleshooting Thunderbolt a sort of ridiculous experience. As Thunderbolt is not really a first-class bus on anything but a Mac, and most PCs/SCALE servers won’t even have a Thunderbolt bus, that’s a reasonable if unfortunate (for me and my Thunderbolt-enabled hardware) decision.
Short version:
-
By default, SCALE does not authorize Thunderbolt devices. Which means they don’t get PCIe lane allocations when they’re plugged in and
lspcican’t even see them. Already getting off to an awesome start for troubleshooting. See: Thunderbolt - ArchWiki -
Theoretically, your BIOS should allow you to overcome this by disabling “DMA Protection” either on your PCIe bus as a whole or Thunderbolt in particular.
2.1. In Thunderbolt’s case, DMA Protection is intended to protect you from a malicious actor with physical access plugging in an evil Thunderbolt device. That is not something I’m particularly concerned with in my bedroom home office, and in any case no such protection is enforced anywhere for hostile USB sticks (which actually exist), so … this just feels annoying.
2.2. My motherboard doesn’t even expose Thunderbolt settings in the BIOS, so that brings us to trying to make the kernel do what we want. -
Thunderbolt Kernel Settings. Again, see the article I linked above.
3.1. For testing, you can useechoto manually flip an authorization bit via the/sys/bus/thunderbolt/device tree. As soon as you do that, the Thunderbolt device should just appear inlspci, as if it’s been plugged in at that moment.
3.2. Pushing values into/sys/bus/thunderboltwon’t persist after reboot, which is where the trouble begins.
3.3. Ordinarily, you’d use audevrule to set it up so a specific Thunderbolt device or all Thunderbolt devices are authorized on plug-in, just like USB. (See the wiki link, above.) -
This is where the madness begins to set in.
4.1. TrueNAS SCALE does not persist text config files added to/etc/udev/rules.d/across reboots, even when you create the.conffile as root.
4.2. On one hand, yay security, on the other … this is the part where TrueNAS actively starts fighting you, which is a step past just not supporting Thunderbolt, IMHO.
4.3. The next most correct Linux way to configure this would be asysfs.conffile in/etc/to manually run the activationechocommand on boot.
4.4.sysfsutilsis not installed in TrueNAS Scale and I’m not inclined to activateaptjust for this. -
At first, it looks like TrueNAS has a mechanism for this. See:
System Advanced Settings > Init/Shutdown Scripts.
5.1. Add a pre-init command here toechothe activation bit intosys/bus/thunderbolt…
5.2. It works! Except …
5.3. If for any reason you lose connection to the device, that script only runs on pre-init, so it won’t run when the device gets plugged back in.
5.4. This is a problem if your Thunderbolt device gets dropped because of “retimer” disconnects (Linux and or the motherboard freak out and reset the “retimer” on the Thunderbolt bus.) This is happening to me because … reasons? That is unclear, but I suspect having ASPM active on the Thunderbolt bus (the only place it’s active on my PCIe bus, for some reason) is causing the bus to reset from time to time when the NIC gets too idle (or whatever other trigger makes ASPM decide to … help). -
This is where I hit a wall. There are options to fix this, but none of them are workable either for me specifically or in TrueNAS Scale’s case in particular.
6.1. Easiest option: Disable ASPM on the motherboard for Thunderbolt. My BIOS does not expose any options related to Thunderbolt, so that’s right out.
6.2. Kernel command line options. It should be trivial to add a flag to the kernel command line to disable ASPM for testing …
6.3. I have no idea how to safely add arguments to the boot command in TrueNAS Scale. I’ve never seen a grub config that looks like the one on this system and I’m afraid to touch it. -
In sum: I think I could get this working if I had the ability to set udev rules that persist across reboots and add a kernel command flag for ASPM. But really the most that would do is get Thunderbolt devices working automatically on boot and keep ASPM from interfering. No guarantee something else wouldn’t go wrong at that point. That realization made me accept that this was a rabbit hole not worth further pursuit at this time.
But, I did learn a lot about how Linux implements Thunderbolt and how to set custom udev rules, so I guess that’s a plus. 
1 Like
Thanks for a very comprehensive write-up. Terrific job.
I had Thunderbolt networking working with a direct connection between two TNS servers with on-board Thunderbolt 4 and, while the connection would survive a reboot by either server, when both servers were down after a power failure, the connection did not come back up so I threw up my hands and moved on to other things.
That was on 24.04 so maybe I’ll give TB networking another try on 24.10. I’ve got a couple more of the TB-10 Gbe adapters coming so maybe try them out as well when they arrive.
1 Like
Glad that was helpful. First time in a while I’ve tried to write up something like that, and I was a bit afraid that I was either going into too much detail or not enough, or that my frustration was bleeding through too much.
As a lifelong Mac user, the instability of Thunderbolt on anything but a Mac has been a confusing, aggravating disappointment. It’s hard not to feel like you’ve made an expensive mistake every time you look at your hardware and see ports that you can’t use.
Aside: There are user-land Linux tools mentioned in the ArchWiki article that would make troubleshooting easier, but those aren’t included in Scale. I’m considering asking for them to be bundled in along with whatever package contains lsusb, which I’ve found myself needing a few times.
My motherboard doesn’t even expose Thunderbolt settings in the BIOS, so that brings us to trying to make the kernel do what we want.
Just for completeness, I’ve since discovered that my BIOS has an undocumented Advanced BIOS mode, that exposes a lot more options, including ASPM settings. I need to get into it and look around (and investigate enabling C-states, which are apparently not on and would explain part of the high power draw on this system), and see what Thunderbolt options are available, if any.
That’s specific to my use case, but I would say that if your BIOS doesn’t expose Thunderbolt or ASPM options, or C-state settings, or anything else that you think should be there on a modern system, it’s worth checking to see if there’s a hidden/advanced mode.
There are tools to investigate BIOSes to find their hidden features, so odds are someone has poked at the thing and posted about it somewhere.
Indeed… I suggest you go through the stages of grief and accept that Thunderbolt is pretty much a Mac-only interface.
1 Like
Yeah. I got there a while ago, actually.
At this point, I occasionally go through a flurry of trying to make it work not so much because I want Thunderbolt per se (though it’d be convenient), but because it not working reminds me there are PCIe lanes on this board locked up on a bus I can’t use.
This board is already starved for PCIe lanes. Like, don’t sell me a board meant for Linux with hardware Linux hates when you could have given me an extra actual PCIe slot. Grr.
2 Likes
One of my servers (ZimaCube Pro) exposes TB setting but their meanings are a complete mystery to me. The other (UGreen DXP6800 Pro) doesn’t admit that they exist.
The one consistent thing was that TB networking would only “wake up” on the server, if the connection on the other end was already working. The way I made that work was to plug my MBA M3 with networking already setup on the TB port into one of them. I could then change theTB cable to the other server and everything worked until both servers were off simultaneously. I imagine there is some secret incantation that will bring up TB networking on a server at boot but I lack the knowledge needed to cast the spell.
ixSystems lack of interest in TB networking was the primary reason I stopped working on it. I want my TNS servers to be vanilla and stable
2 Likes
Welcome to modern desktop boards. Depending on which one, you may lose 8-16 lanes to NVME storage.
I just upgraded my gaming rig to a board that has 3x x16 slots (5.0x16, 4.0x4, 4.0x2), which will allow me to hand down this hardware to my nas in 3-5 years (10gbe NIC, HBA and ARC gpu). There are a ton of $300-700 AM5 boards that only offer 2 x16 slots.
This is the truth. TrueNAS is not meant to be tinkered with, and Thunderbolt on Linux is an exercise in tinkering. I don’t feel like I can really count on any TB3 device on Linux at this point, even disks.
I’ve got a Realtek USB 1 Gbps NIC in a powered hub hooked up to my TrueNAS server. I’m using it as a management port, and it’s rock solid.
As a Mac user, a USB Realtek NIC being more solid than a TB3 device feels like I stepped through the mirror that the wizard tells you not to go near.
One thing I’ve been thinking about though:
USB 4 is a thing. Is USB 4 going to end up neglected bus on Linux the same way Thunderbolt is? My server’s TB ports identify as USB4 ports, too, so I can’t help but wonder if USB4 hardware would work better.
I don’t own any USB4 hardware and since my Mac is great with Thunderbolt, I’m not inclined to replace any of the TB3 docks/NICs I already own. Catch 22. -_-
I doubt they will work consistently with USB. Perhaps better than Thunderbolt on Linux, but using USB to host boot pools, etc. has been frowned up here for a while because of how flakey USB can be. Never mind USB + port multipliers + hardware RAID 5 like my Oyen Digital Mobius 5s.
I was psyched to get my first QNAP server that claimed combined NAS/DAS operation via Ethernet and TB2. What QNAP left out was that their NAS/DAS would commandeer ALL Thunderbolt devices on its bus for itself, so I suddenly I couldn’t use my monitor, dock, etc anymore (my MacBook air only had one TB2 port). Back to Amazon it went.
Good news is that the between my great experiences with QNAP, Synology, and ReadyNAS I finally saw the light and came to FreeNAS with a MiniXL. I’ve had a few bumps along the way but as far as I can tell no data loss or bit flips.
I suspect that Thunderbolt works as reliably as it does on Mac platforms because Apple has put in considerable time to ensure the OSX drivers work ok (much like IXSystems allegedly worked hard to make the C4750D4i board reliable re: FreeNAS).
Apple also only use a limited number of chipset SKUs, IIRC. At work, some docks will work with Apple and some cheaper docks won’t work.
Part of that is likely due to cost. IIRC, Thunderbolt chipsets are more expensive than USB4 to implement and hence rare outside of laptops in windows-land. Can’t any TB4 chipset do USB 4 but the reverse is not true?
And USB4 (no space…) includes a Thunderbolt 3 compatibility layer. But USB sub-standards are a dog’s breakfast.[1] I would expect the USB part of USB4 to work; but the Thunderbolt 3 part may not get more attention and better support on Linux than TB 3 got in the first place.
Then, shortly after Maple Ridge TB4 controllers were released, Intel damaged its own standard by removing TB 1 and TB 2 support from Maple Ridge through a Thunderbolt firmware update (NVM 36), packaged in motherboard BIOS updates and not downgradable[2].
So, while Apple users can now enjoy their legacy TB1/2 devices on their Apple Silicon Mac (with adapters) because Apple makes its own Thunderbolt controllers and still maintains full compatibility, Intel users now have to interpose a Titan Ridge (TB 3) dock between their Maple Ridge TB 4 controller and their legacy TB 2 audio interface… I suppose Intel saves on compatibility testing and that no Windows user complained because Thunderbolt is not used on Windows. And how’s that going to evolve with Thunderbolt 5 is anybody’s guess.[3]
That’s all the point: Thunderbolt 4 controllers do USB4; USB4 controllers do basic Thunderbolt 3, but not Thunderbolt 4 (=Thunderbolt 3 with all optional features).
ASM4242 is a “USB4” controller that is actually designed as a full TB 4 controller but was released without official Thunderbolt certification. Whether ASMedia eventually gets certification is an open question.
USB 3.0 (with space…), USB 3.1 Gen1, USB 3.2 Gen1, USB 3.2 Gen 2, USB 3.2 Gen2x1, USB 3.2 Gen2x2… Fancy a quizz? Which modes are not supported by the USB compatibility layer of Thunderbolt controllers? The x2 modes. ↩︎
well, not without a special downgrading BIOS from the manufacturer (9901 for some Asus Z690 boards). ↩︎
My guess: Windows users will, eventually, get Thunderbolt 5 docks; all other Thunderbolt devices will be designed and marketed solely or primarily to Mac users. ↩︎
1 Like