USB to SATA adaptors and SWAP space nightmare

TrueNAS General
,
1

2024-09-13 10.45.45
2024-09-13 10.45.451920×1440 232 KB

I’m building my first TrueNas server using Scale 24.04.2. Everything has gone smoothly apart from getting the boot pool to work using a pair of USB to SATA adaptors connected to the internal 19 pin header. The rest of the system is;

Motherboard: ASRockRack AM5D4ID-2T/BCM (Deep Mini-ITX)
RAM : Kingston 2 x 48GB 5600MT/s DDR5 ECC (MemTest86 for a couple of hours, all good)
CPU: AMD 7600
HBA: Broadcom 9500 16i
Drives: 8 x Kingston DC600M SSD 7.68TB
Drives: 3 x Micron 7450 800GB NVMe M.2 (1 on the motherboard, 2 on a PH-43 PCIe riser card)
Boot drives: 2 x Kingston DC600M SSD 480GB (I know they are way too big, couldn’t find anything smaller that actually cost less)

Everything is brand new. I also have another 8 x 4TB SATA SSDs in my old Synology NAS that will move over once I’m happy this new TrueNAS build is working reliably.

The motherboard has no SATA ports and I really want to keep all the ports on the HBA free for data drives. To do that I have to boot from something attached to the USB header on the motherboard. I know using the USB bus isn’t ideal but it’ll be mirrored with decent SATA drives and I’ll keep backups of the config, I’ve accepted this risk. I thought this would be straight forward, how wrong was I.

Here is what I’ve tried so far:

2x Sabrent SATA to USB 3.2 (https://www.amazon.co.uk/dp/B011M8YACM)
No idea what chip this is using. Everything installed and seemed to be working but the UI was having issues as this adaptor reports the same serial number for both the boot drives. Also reports the make and model of both drives as “Sabrent”.

So I connected the drives to my laptop one at a time and SMART tests are passing, formatted with ext4 and copied some big files over and back with no problems (checking file hashes as i went). So I did some research and various blogs/forum posts said that adaptors based on ASMedia chips would work best.

2x StarTech SATA to USB Cable - USB 3.0 (https://www.amazon.co.uk/dp/B00HJZJI84)
Uses the ASM225CM chip. One of them reports the correct model, make and serial number and the other does not. I can install TrueNas but during the first boot it go to the GRUB menu, carries on then gives “cannot import pool ‘boot-pool’”.

2x StarTech SATA to USB Cable - USB 3.1 (https://www.amazon.co.uk/dp/B00XLAZODE)
Uses the ASM235CM chip. Wont boot, hangs on the ‘B4’ POST test which is means a problem with a USB device. Interestingly, it will boot with just one of the adaptors plugged in, either of them. The ‘B4’ POST test takes a couple of minutes though. I can install TrueNAS on a single drive but not boot TrueNAS as one of them keeps dropping the drive and the other gives “error: compression algorithm inherit not supported” before the GRUB menu and then “cannot import pool ‘boot-pool’”. I also spun up a Windows 10 VM to try and update the firmware for these, as StarTech provides an update. Both of them failed to upgrade with a ‘configuration mismatch’ error. I also tested these by connecting them both to my laptop running Fedora. I believe that one is faulty as it also kept disappearing here. At least they were reporting the correct model, make and serial number.

2x SanDisk 64GB Luxe Flash Drives (https://www.amazon.co.uk/dp/B0842P231G)
Seems to work fine, installed TrueNas, updated it, setup pools and installed an app etc… Both drives visible in the UI. Obviously not a long term solution but this gives me more confidence that the USB ports on the motherboard are working. They got really super hot!

2x UGREEN USB 3.0 to SATA (https://www.amazon.co.uk/dp/B07Y825SB8)
Uses the ASM225CM chip and claims to support UASP, SMART and TRIM out of the box. Everything installed fine but TrueNAS wouldn’t come up. I got several errors about “error: compression algorithm inherit not supported” before the GRUB menu then “cannot import pool ‘boot-pool’” and left at the initramfs promt.

AH HA!

I did some more testing and found that the 16Gb swap partition that I’m asked to create during the install could be causing some of the issues. In all the cases above, I did not create the swap partition. If I do create it, everything seems to work. I’ve confirmed this behaviour on both the UGreen adaptors and the StarTech 3.0 adaptors.

Have I found a bug!

At this point I’m completely out of ideas. I don’t want a swap partition on the USB connected drives! Will probably spend another day searching the internet for ideas and then try a pair of industrial USB sticks. This “error: compression algorithm inherit not supported” with no swap seems odd to me and I not convinced it’s a problem with the adaptors. I can get the Innodisk USB Drive 3ME but they’re not cheap and not a great solution, sigh. If anyone has any suggestions, I’d really appreciate it.

2 Likes
2

Hey @cgfrost

Thanks for the comprehensive testing and details!

One path I’d wonder about going down is trying these adapters (on a test system) to see if they’re passing the serial numbers and product data correctly. If they’re all using the same single set of identifiers (as seems to be the case with your Sabrent adapters) then TrueNAS might be getting confused trying to differentiate.

You can do this from a booted TrueNAS system with:

sudo sg_vpd -p sn /dev/sdX
sudo sg_vpd -p di /dev/sdX

Compare data across the adapters from the same brand with each other.

3

I was thinking the same thing, make sure the drives are not being shown with the same serial numbers.

The issue with needing the swap space seems odd to me. With all the RAM you have, you shouldn’t need any swap partition.

Lastly, can you use a single voot device and it all works? All you need to do is make a backup of your TrueNAS config file and restoration is fairly easy.

4

Hi @HoneyBadger, thanks for the quick reply.

On the occasions when I’ve been able to get to the UI, both the UGreen and StarTech 3.0 adaptors have reported different serial IDs. I assume Sabrent is reporting matching serial IDs but in the UI I can only see once of the SSDs. I tested the Sabrents on Fedora and saw the matching IDs there.

The documentation says that it won’t try and create a 16GB swap if any of the boot drives are connected by USB. I suspect Sabrent had no issues as it detected that it’s connected over USB and didn’t ask to add a swap; but the other adaptors are doing a good job of being transparent and the installer can’t tell they are USB and wants to add the swap. Then it goes wrong when I don’t let it have a swap.

I’m unable to run those commands when I’ve selected no swap as I get left at the busybox built-in-shell which doesn’t understand them. In the working case, where I’ve accepted the swap, this is what the Ugreen adaptor reports.

UGreen-with-swap
UGreen-with-swap744×634 62.4 KB

5

Hi @joeschmuck

I’ve just tested this out. Using a single UGreen adaptor and with the second USB cable unplugged. When I accept the swap option, everything works as expected and I can boot in to TrueNAS. When I decline it breaks when booting in to TrueNas for the first time with “cannot import pool ‘boot-pool’”. Interestingly I didn’t get the errors before the GRUB menu about “error: compression algorithm inherit not supported”. Possibly there is more than one thing going on here.

6

That is very odd. I am curious about something. When you boot and are able to log into TrueNAS, how much ram does it say you have? I suspect it will all be there however this should not be happening. And I don’t understand the compression issue.

I am perplexed.

Have you tried Legacy BIOS booting? Not UEFI.

7

Carrying on from the single boot drive test with a UGreen adapter, I enabled CSM (Compatibily Support Module) in the BIOS and put everything in legacy mode. I hit the same problem, “cannot import pool ‘boot-pool’”.

When I have an install that works, with swap, the UI reports all 96Gb of RAM.

There is at least one problem, the installer shouldn’t offer to add a swap when drives are connected over USB. I have no idea why not having that swap would cause an issue though. Does TrueNas do anything special when it knows the boot pool is running over USB?

8

I don’t know if it’s relevant but when booting in to the installer, before the first menu is displayed I get some error messages.

hub 6-0:1.0 config failed, hub doesn’t have any ports! err -19

and

systemd[1]: Failed to open /dev/shm device, ignoring,: Inappropriate ioctl for device

9

Not that I’m aware of. Using USB is generally fine, it is using USB Flash Drives that is not so good as they tend to die quickly.

Guess you will be running with a swap file on the boot drive for a while. Very strange. But you have lots of space and you have lots of RAM so you should never need to use Swap in the first place.

You may want to file a bug report because that is definitely strange.

10

It’s not a problem with USB. I moved the two drives to the HBA and did a fresh install again. It asked me to add a swap, I did not. Install finished successfully and I rebooted. I got the same error, unable to find the “cannot import pool ‘boot-pool’” .

I’m completely stuck at this point unless I accept the swap partition. Any suggestions?

11

I don’t see why you can’t just delete the swap partition.

You can get to a shell from an installer iso if that’s the issue.

And anyway, 24.04.2 doesn’t use swap. Even if there is a swap partition.

12

You can create the swap partition, it will be fine. It is just odd how your system is reacting to not having that swap created.

13

Thank you @Stux. if I delete the swap partition the boot pool not found error comes back. I believe the swap partition is causing a slight change in how long it takes other things to come up and that’s the problem.

Thank you @joeschmuck. Yes, I’ve realised this having read about the recent changes to the swapiness value from the default to 1. Unfortunately I’m unable to let something like this go, I’ll get to the bottom of it eventually. :sweat_smile:

Based on reading through a lot of forum posts I’m going to try adding a slight delay to the GRUB config and see if that helps. I believe it will, will find out tomorrow morning when I can spend some more time on this. I’ll report back once I have.

14

Let us know if you are able to fix the strange issue.

15

I’m unable to run either update-grub or update-initrd while in the initramfs busybox cli. Is that correct or am I missing something? I’ve looked in /bin and /sbin but can’t find it.

16

I may have been mistaken before. Everything is working perfectly when the boot drive is attached to the HBA. I think I had multiple boot pools setup and it was getting confused.

I still want to try adding the sleeps to the boot scripts but I’m less confident that they will help. I’m unable to manually import the boot-pool from the initramfs promt. zpool is also reporting “no pools available”.

fdisk can see the actual drives, they are /dev/sdi and /dev/sdj.

Stuck at this point.

17

I’m beginning to suspect that one of the drives had a hardware problem even though I have got a bootable TrueNas install on them a couple of time. Also, when I got everything to work when booting from a device on the HBA I was using a different device.

I’ve done a fresh install, mirror on the two Kingston drives connected over USB, with a swap and now this isn’t working. The error is a different one about too many pools named boot-pool. When I try to import them by id I get this.

Screenshot from 2024-09-18 14-35-44
Screenshot from 2024-09-18 14-35-441637×583 118 KB

@HoneyBadger or @joeschmuck does this look like hardware problems to you?

18

Usually this results from improperly wiped disks, causing both drives to advertise a name of boot-pool but with different unique IDs.

But a pool that self-corrupts itself is likely bad hardware, or possibly a missing USB storage quirk on the adapter.

19

Thank you @HoneyBadger, we have a solution!

I booted up a Fedora live CD and wiped both my boot drives with zeros, took about an hour. I also wrote 100MB of zeros to the beginning of every other drive, being impatient.

Installed TrueNas with no swap and everything is working. Boots up no problems, setup pools, all disks report healthy and so on. I believe the root cause of this problem was unhelpful 1s and 0s on the boot drives that the installer wasn’t wiping. Having a swap partition meant that the unhelpful 1s and 0s didn’t cause an issue. What an annoying bug.

1 week later and an unhealthy collection of USB/SATA adaptors, I’ll carry on setting up my TrueNAS server. lol.

Finally, I do still get these errors coming out just before the GRUB screen. I believe they are unrelated to my boot problem. There are quite a few threads about similar messages that just blame GRUB and say not to worry about it. Do you think it’s safe to ignore these?

TrueNAS-Scale-Boot
TrueNAS-Scale-Boot1623×1224 62.3 KB

Thank you so much for your help. Easily one of the nicest open source forums I’ve been on.

20

Happy to have been of some assistance :slight_smile:

The compression issues do appear to be related to GRUB trying to iterate through snapshots and not knowing how to deal with the ZFS inherit setting for compression - it doesn’t appear to be an actual bug in GRUB understanding compression (although I don’t know if it supports zstd yet, but generally one shouldn’t go to the command prompt and muck with the boot-pool settings) as the boot itself works.