So after recently upgrading my TrueNAS Scale system, and migrating from internal SATA to external SATA-over-USB, I thought I’d share my experience and information that might help others, including some things which I could not find anywhere on the internet.
So to start I had TrueNAS Scale running on a really old CPU that was struggling and needed upgrading. I wanted to run ownCloud, but I couldn’t even get the containers running because the CPU just couldn’t handle it.
The system was a micro-ATX with internal SATA drive bays. However this time around I wanted to change design. I’m just tired of having to rebuild an entire system (not just TrueNAS, but any of my systems) any time I want to upgrade any component. Want to replace the CPU? Well, you’re going to need a new motherboard because the socket is out of date. Oh and the RAM too because it’s also out of date. And going to need a bigger PSU now. And if you change from mini-ATX to micro-ATX or whatever, going to need a new case too. There goes several hundred dollars. And the PITA of building it all.
The point of using TrueNAS was to be simple and have it just work. Well the software may be simple, but the hardware is anything but.
Even if you say “oh, you can run on an old CPU for a while, so upgrading is rare”, there are other reasons that might involve all this. Maybe I want 2.5gbe. Maybe I want NVMe. There’s always new features coming around.
So my solution was to change the design and go with everything external. Though to be fair that just means the drives really. Though I do have an external KVM unit (PiKVM). This way I can just upgrade individual components (and things like mini PCs are cheap). And while right now I’m using SATA drives, I want to switch to NVMe drives in the future (price is really coming down). There are external USB-to-NVMe adapters (I use them for other things), so the idea is feasible. And with USB4’s PCIExpress-over-USB, I figured that would just make things even easier. There are units like the Flashstor or the LincStation which are NVMe NAS units. But again I wanted upgrade flexibility, and plus they wouldn’t support my current SATA drives anyway. I googled the subject and saw comments saying “TrueNAS doesn’t care about how the drives are connected, it’ll just work” (that turned out to be hilariously wrong, but will get to that shortly).
I went with an N100 Mini PC (one of those all-in-one units where you just plug it in). Has 5x USB 3.2 ports, so plenty of connectivity. I also went with a QNAP TR-002 for the drive enclosure.
I put the QNAP in “individual” mode so it would export the individual disks to TrueNAS, starting with just one disk plugged in to make sure it wouldn’t do anything stupid like wipe it (I have 2 disks in mirror). I also moved the OS drive from the old system to the new one so I wouldn’t have to reinstall everything. After booting everything up, I did have to fight with TrueNAS a bit to get it to forget about the old network interface so it would use the correct IP addressing. But after resolving that, I booted it up again and everything appeared to work fine. The system was operating as if nothing had changed. Both drives were online, the network mounts from my systems were all working, etc.
But then I started poking around and noticed that while everything was functioning, the TrueNAS UI was not happy, and some parts of it were showing 2 disks, while other parts only showed 1 disk. After researching, it turns out this is a well known issue because it’s common for external multi-disk enclosures to use the same serial number for all the disks. And while the underlying OS didn’t care and ZFS was happy, the UI apparently doesn’t like that. And here’s where I have some beef with TrueNAS on the subject. When I looked at the Udev info on the disks, they did have different serial numbers. However they also export a “ID_SERIAL_SHORT” attribute, and that was the same. So it seems that instead of using “ID_SERIAL” TrueNAS is using “ID_SERIAL_SHORT”. I would think that if TrueNAS is going to care about serial numbers, it would use the whole serial number, not a shortened version of it. But nooooo. Seriously guys?
However I do somewhat understand the issue in that the serial number wasn’t the actual serial number of the drive, but the serial number of the adapter/controller.
I went researching on how to solve this problem and could not find much helpful information. Just lots of things saying “don’t do SATA over USB” over and over. I refused to accept that, and dug more. I was originally thinking I could use an eSATA enclosure (the mini PC does have a single SATA port on it), but I was less than thrilled with the idea of hacking it up to turn the internal SATA port into an eSATA port (not very flexible and future-proof). I eventually ran across UAS (USB Attached SCSI), which is a protocol meant for attaching disks over USB. However I could not find any information whatsoever about whether this solved the serial number problem. The QNAP device does not support UAS, so it meant I would need to buy another enclosure that did, and find out the hard way. I also couldn’t find any definitive information whether the mini PC I bought supported UAS, but UAS is old enough to where I guessed it did. I ended up getting a StarTech enclosure that supports both UAS and eSATA (model S352BU33RER). So even if the UAS route failed, I could go the eSATA route.
It turns out that UAS does indeed solve the serial number problem. After plugging the drives into the StarTech, setting it in “individual” mode so it would export individual disks, I connected it to the mini PC and booted it up. Again everything came up just fine, but this time the UI was happy. I inspected the serial numbers, and the serial numbers of the drives were coming through. And I confirmed UAS was in use.
So lesson learned. TrueNAS can work properly with USB drives, but you must use UAS.
The StarTech enclosure is unfortunately a bit louder (the fan) than the QNAP. But it’s still moderately quiet, and is shoved in a closet anyway. I also don’t like how the StarTech has a power button on the front that is easy to accidentally press (the mode switch too). The QNAP would have been perfect if it supported UAS.
All said and done, the new setup, with all the external components (pc + drives + kvm), is now quieter, and takes up about 1/3 as much space as the old setup.