This resource was originally created by user: jgreco on the TrueNAS Community Forums Archive. Please DM this account or comment in this thread to claim it.
In the last year or two, we’ve had a resurgence of users asking about SATA Port Multipliers and cheap SATA controllers.
Please, do NOT use port multipliers, and use cheap SATA controllers only after extensive research.
SATA controllers and SATA Port Multipliers are some of the cheapest hardware, designed to make a number of hard drives accessible to Windows. A port multiplier takes a single port of a SATA controller and multiplexes it between several drives.
Windows typically only accesses one or two drives at a time, which means that the needs of Windows users are often relatively limited. However, ZFS often accesses data on all drives simultaneously, such as during scrubs and resilvers. This creates a lot of stress on I/O controllers, and tends to expose any problems.
Cheap SATA controllers are often built as PCIe x1 cards, to make them compatible with the largest number of mainboards. On PCIe 2.0, a single PCIe lane is capable of 500MBytes/sec, or at best, two HDD’s worth of transfer speed. Yet, many cheap SATA controllers (Syba SI-PEX40064 example) have four SATA ports on them. There is insufficient capacity to talk to all four drives simultaneously.
Worse, often, this is combined with SATA Port Multiplier to offer cards handling as many as 10 drives, so clearly there isn’t enough bandwidth there.
So far, all of this assumes that all the hardware works swimmingly well, and even so, it already sucks.
Unfortunately, many cheap SATA controllers are a bit dodgy, and do NOT work well. Because they are price-driven by the PC market to produce the cheapest product, many of these cards use dodgy controllers which may be knockoffs of already poorly-designed controllers by companies who have put out something that is just good enough to work acceptably under Windows. Often, these chipsets do not work particularly well under FreeBSD and Linux, especially when stressed by the massive I/O loads imposed by ZFS.
Worse, SATA Port Multipliers require both a SATA controller that works perfectly, AND a host driver that understands how to control the Port Multiplier correctly. The number of working, trustable combinations of these that exist are relatively few. The fine folks over at BackBlaze and 45Drives built their first generations of storage appliances with these after exhaustive testing that only found a few good combinations. They have posted significant discussion of their experiences over at Our Biggest Design Decision - Direct Wired or Backplane? - 45Drives Blog
which is worth reading before you choose to go down the Port Multiplier path.
Next, transistors generate heat, meaning that many critical components in a computer have a heatsink to help dissipate the heat. Many cheap SATA controllers lack the heatsink, and if the airflow and surface area available are insufficient to dissipate the heat, ZFS can and will keep your SATA silicon very busy especially during heavy activity, scrubs, or resilvers. Much busier than Windows, which typically focuses on a single drive for a small number of files for only seconds or minutes. ZFS scrubs, for example, can take days, and also usually involve multiple drives. This generates a lot more heat than some controllers were really designed for.
Further, when you get your cheap SATA controller, it may come with cheap SATA cables, with inexpensive SATA connectors, and the signal integrity may not be quite up to snuff. This may lead to occasional corruption of the data as it transfers between the drive and the controller. Most SAS cabling used with the recommended LSI HBA disk controllers are very carefully shielded and resistant to signal degradation, and SAS has other design elements helping to protect your data. The typical bargain bin SATA controller at your local Fry’s Electronics has multiple points where signal corruption may be a concern, and you have to remember that if you have an array of these, you need EACH SATA connector at EACH drive and EACH controller connection, along with the cable itself, to be performing perfectly.
Almost all of these “solutions” have been more expensive to buy new than it costs for a used LSI HBA on eBay, where the PERC H200’s and PERC H310’s are often available for $30, and provide eight fully supported lanes and can use SAS expanders for up to dozens (or even hundreds!) of additional drives beyond that.