Why you should avoid USB attached drives for data pool disks

This subject has been coming up quite often since about 2022. Perhaps because of TrueNAS SCALE has brought forth some attention to TrueNAS in general and ZFS in specific.

Please note that this is about USB attached storage for ZFS data pools. On occasion, people use USB drives, (flash, USB->SATA, USB->NVMe, etc…), for booting. In general, these do work acceptably, (when not using the cheapest USB flash drives). Though, it is always recommended to backup your configuration in case of boot drive failure.

Some of the points also apply to Thunderbolt enclosures. But, Thunderbolt might even be worse because the drivers would be even less tested than USB on TrueNAS Core or SCALE.

Here is my take on why USB attached storage would make a poor selection for ZFS data pool device. Not all of them will apply in all circumstances.

  • The connector is not latched, which may lead to inadvertent dis-connects.
  • Multi-drive USB enclosures funnel all traffic through a single USB port, which may limit drive speed as they share bandwidth. This becomes especially problematic during scrubs or disk replacements. Worse, if there is inadequate cooling.
  • USB attached SATA disks may use the same serial number, confusing software that attempts to detect multipathed disks.
  • Some multi-drive USB enclosures use a very light weight RAID controller chip, which sometimes can be set to JBOD. This is NOT the same as direct pass-through from a HBA.
  • Using USB 2.0 is too slow for many NAS functions.
  • Using USB without UASP feature, (on BOTH sides), wastes CPU time and reduces performance.
  • Some USB enclosures are not intended for continuous or extended use. Which ZFS will do in a scrub or re-silver, potentially causing the drive to get so hot, it / they may fail.
  • USB device chips tend to be as cheap as possible, potentially having unknown bugs, either firmware or logic, that high throughput systems like TrueNAS with ZFS can expose.
  • Some USB controllers are funneled using hubs on the system board. This hub may have other functions attached to it, like your keyboard or mouse USB port. Unless you know which ports do this, you may inadvertently use the “wrong” USB port.
  • Many USB to SATA adapters do not pass through SMART data, so monitoring a USB attached SATA drive for preemptive failure is problematic.
  • A host side powered USB port can hit 500ma on a standard 2.0 type A plug and it simply turns off. So if you use a hub, and mix powered and un-powered devices, it’s possible for the hub to exceed 500ma and your disks get disconnected despite having their own power source.

Power for external USB devices, that can’t be powered from host:

  • External power bricks usually have press fit power connections, not latched. Thus, they can come loose, or accidentally pulled.
  • External power bricks may have long cables that get in the way, so they may get un-plugged.
  • USB disk chassis that need external power, likely don’t support redundant power. But, an actual server computer may.
  • If using a UPS on the server side, it would be necessary to include any USB power bricks on to the UPS, for any USB pool member disks.
  • When using multiple external USB disks that require their own power supplies, you increase the potential for failure with each power supply.

All that said, USB attached storage does have a place with TrueNAS:

  • Occasionally USB has to be used as the boot device connection method, (or better yet, boot Mirror).
  • Importing data to a NAS via USB storage may be faster than network copies.
  • Backup disks attached via USB for temporary connection can be used successfully.

For the last, a failed backup can often be repeated without data loss. Coupled with planning and best practices, ZFS snapshots can preserve the previous backup, limiting the loss if it can’t be repeated.


See related problems with SATA port multipliers:

Multiply your problems with SATA Port Multipliers and cheap SATA controllers

In the last year or two, we’ve had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers. Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research. SATA controllers and SATA…

www.truenas.com www.truenas.com

It is possible that a USB to SATA adapter chip, then feeds into a SATA port multiplier, increasing the chances of unreliable disk access.

3 Likes

ty Arwen for covering this.

Also by now most experienced users would know, not to use usb flash sticks for their truenas boot drive.

I’m using a usb external enclosure with M.2 NVME ssd. because my TS-877 which has m.2 sata in them via a pcie addon card doesn’t allow boot apparently for truenas. I think installing a 2.5’’ m.2 sata ssd might work, but i don’t have any on me to confirm that.

This is the link to my project as a case study for this thread

I just wanted to share my experience when using a usb m.2 nvme enclosure as the truenas scale dragonfish latest as the boot drive.

it works stable. however, i noticed that smart test does not work.

As for caution because it’s connected via usb and can risk getting disconnected, one solution for this is to add a double sided stick adhesive tape or sorts, to then stick the ssd enclosure to the side of your nas, problem solved (yes it can still be easily pulled out of usb, but that normally would have to be intentional).

But as others pointed out, just be sure to save your latest working truenas config, so if anything does happen, you can at least bring things back online by importing that config.

If i had a choice, yes i too would preferably want to install on a drive installed into the nas directly (preferably a m.2 nvme ssd) x2 in mirror for a boot drive. But for some NAS e.g. qnap converting to truenas, they may not support these (some models are even limited to m.2 sata or some not even that). So for those models their only option is through a usb enclosure like mine (m.2 nvme)

This is the specific enclosure i am using, and tested as a working for booting truenas from

The other issue i had was the alerts

#### 'boot-pool' is consuming USB devices 'sdg' which is not recommended.

I appreciate the reminder and know why it’s there, so i dismissed. In RC1 it kept spaming me about it. The devs know about this issue and are gonna do something about it in future, so don’t want to keep harping on this. thought it deserved at least a mention in this thread. The warning i’m told was supposed to be mean for the usb flash stick users. But i am a usb nvme ssd user. So i’m basically collateral damage because truenas doesn’t know this, it only see it’s booting off usb only so it can’t make the distinction apparently.

Also on this topic i saw one user post their usb setup for truenas that may be of interest

1 Like