On a moderation note, I will treat any further statements of fact on port multipliers which are not accompanied by relevant citations as being made in bad faith.
For those with an ASM1166 based controller, may want to delay upgrading to Dragonfish
1 Like
Not entirely clear to me if it’s a controller issue or if there’s a port multiplier in the mix. I think some folks are starting to see port multipliers where there only crappy AHCI controllers, which makes things hard to follow…
I am the author of the post @Stux referenced above (although now I’ve seen others reporting the same issue), and in my case the product is the “MZHOU PCIe SATA Card 8-port” (link) which I’ve come to understand (mainly from this blog post) is based on the ASM1166 SATA controller, which is a cheap SATA controller with less bandwidth than some but no port multiplier, combined with a port multiplier to get the extra ports.
So I’m not sure that the issue would occur with an ASM1166 card without the port multiplier. There are now multiple bugs filed by different people (NAS-128478 being the main one, the others are duplicates of it) so hopefully that will help disambiguate the specific hardware affected.
FWIW, though, in this and the other thread, as a “casual” user of TrueNas (shared storage and backups for my kids, etc) I found the snarky comments around port multipliers rather helpful. I’ve always thought that if they work, they work, and they just might be slower and less reliable than enterprise gear. I’m not gonna ever have ECC RAM or enterprise SSDs in my home TrueNAS server, because it is always going to based on my kids’ last gaming PC that finally got too old to work for games. (Currently, I’m rocking ASRock Z390 with a 9th-gen Intel CPU.) So I didn’t think I needed an HBA either.
But the “well, duh, you’re using a port multiplier, bro” comments did spur me to dig into the docs/blogs a little more and — TL;DR — after reading various info and experiences, I did, in fact, order a used LSI HBA. (Or what purports to be one anyway — I live in Japan, and the eBay ones plus shipping are quite expensive for us here, especially with the collapse of the JPY, so what I ordered might be a fake — we’ll see.)
EDIT: Also, if the problem does happen to your machine, it is immediately obvious and you can simply downgrade using the web UI to 23.10 and all the data is fine and usable again. So, not a big risk in updating.
4 Likes
@axhxrx thankyou for the candid reply
I’m not trying to disparage people’s hardware, but rather work out what works and what doesn’t. Unfortunately there are very good reasons why Port Multipliers are a bad idea, what I’m not so convinced about is that all SATA controllers are inherrently flawed.
At least by having this thread, we can try and get some up-to-date experiences from people now that TrueNAS is based on Linux, rather than BSD, which does make a difference to its hardware support.
Hopefully we can get to the bottom of it… Personally, if I needed a couple of extra SATA ports in my build I would probably use a SATA M.2 card… but it seems that Port Multipliers are simply a no-go, especially at the moment with Dragonfish
Sure. To be clear, I didn’t find your comments snarky at all, I was more referring to the memes like this. (But even those kinds of things are actually pretty helpful to help figure out Hmm should I allocate time to understanding this better?)
1 Like
The point is that none is playing around but truly using those adapters: one thing is saying “everyone, here Is this new thing I will test and come back to you” another is “Hey, why things don’t work? I’m using this cheap thing btw”.
We welcome everyone willing to test and give us feedback, even when it has been proved multiple times it does not; we do not welcome people making baseless claims.
If you see things that way you have an issue with how you view things. Speaking from experience and knowledge and suggesting using proper, reliable hardware its one of the strenght of this community; are you expecting a tech forum to not help and not educate?
You do not build a NAS with a mini pc here.
Either you have not looked into the old forum or you don’t know how to search.
I would argue that choosing unsuitable hardware is bad planning.
Everything has already been answered by eric, and I will just say that your attitude is totally wrong if you care about this community (which you clearly do not) and want to make a change.
That’s a frequent dabbler’s honest reaction when seeing someone it’s using port multipliers, I would not see that as snarky.
2 Likes
JGreco’s resource about port multipliers does state that some, but not all, Marvell or ASMedia SATA controllers are acceptable. If you’re absolutely desperate to add a couple of SATA ports in the smallest possible footprint, you may want to dig further with that but it’s going to be more validation trouble than a $50 M.2 card is really worth. That energy would have been better spent trying to find a small footprint motherboard with enough native SATA ports.
All occurences of port multipliers are bad. Period.
Chipset ports are good.
LSI 2008/2308/3008 and their OEM versions from Dell/HP/IBM/Lenovo are the way to go to add more ports.
These are simple and safe rules. HBAs can be found second-hand for less than what a “good” (or “acceptable”) SATA-ports-on-M.2-stick costs new.
3 Likes
This is true. I read the specs. It’s incompatible with ZFS’ idea of a reliable and queuable block device.
1 Like
Allow me to jump in here.
We can certainly encourage/discourage certain component choices, but ultimately each user is responsible for their own decisions, their own level of acceptable risk tolerance, and their desired system form factor/budget.
It’s important to note that M.2 is only a form factor decision - bite-sized PCIe is still PCIe - but the crux of the problem rests on three major issues:
- The chipset used to provide SATA connectivity (including driver support)
- Manufacturing quality of the device itself
- Presence or absence of a port multiplier
While SCALE has expanded the playing field relative to CORE (where “Just Buy LSI” was the rhyme of reason) - there are still definitely “tiers” of driver support and functionality within the Linux kernel, and the vast majority of M.2 SATA controllers use a “lower tier” chipset. Intel and AMD are preferred over all others, Marvell and ASMedia share the next space, and JMicron brings up the rear (but has made improvements recently.)
A poorly manufactured or improperly cooled device can of course contribute to instability - whether the chip responds by temporarily throttling commands or running to thermal failure though is another question. M.2 slots traditionally do not get the same quantity of airflow as traditional PCIe, and this is exacerbated by the “cable spaghetti” that comes from these cards. Prefer cards with heatsinks, if possible.
Finally, port multipliers - the thorniest piece of this argument. Quite simply, they are to be avoided for multiple reasons.
The first is the bandwidth argument - the most common multipliers are 1:5 - so one SATA port worth of bandwidth being spanned across five downstream devices. That means your 600MB/s of theoretical SATA3 bandwidth is cut into 120MB/s per device - a definite limitation for SATA SSDs, but it can also be a limiter on sequential I/O to spinning disks. With sequential resilver and scrubbing, this is a very real thing to hit.
The second is the nature of the port multiplier itself - if it’s using “Command-Based Switching” then you can basically have only a single command queued against any of the devices behind the port multiplier, and all others are blocked. This means that your performance tanks even more than the bandwidth cap, because it’s like having a common media akin to a “hub” vs a “switch” in the old networking sense. If it’s using FIS-based (hardware) switching, then it’s able to queue up against multiple devices, but it’s still sharing the bandwidth.
And finally, if a single device behind a SATA port multiplier hangs up or fails to respond to a command, it could (“will” in the command-based switching model) cause all devices behind the PM to be non-responsive. Not exactly a good thing as RAIDZ6 doesn’t yet exist. 
6 Likes
This is absolutely the key.
I have a few machines that are 1 liter or smaller and have some data they share with other machines on the network. I would never consider calling them a “NAS”. Maybe someday when the bugs on the multi-M.2 micro motherboards are worked out, we can have a real NAS in a tiny form factor.
1 Like
Linking to a similar thread on old forum for posterity
2 Likes
I think something many of the moderators here ignore is that there are plenty of people who use Truenas in unsupported unreccomended configurations at home who also use it in a professional enterprise setting. I was an Isilon Integrator before eventually using Truenas in many projects. My issue isn’t that people have differing opinions.
My issue is with the often condecending attitude by some. This insessant whining not to do something a particular way in someone’s home lab experiment. Sometimes the responses by moderators are plain rude, if not helpful at all. I mean I was threatend action by @ericloewe, for merely stating my opinions supported by facts, for not citing my sources. Despite those facts being pretty easy to find with a simple google search, I didn’t see anyone else citing anything. I was then promptly suspended from this forum, for which I appealed and got reinstated pretty quickly. Granted he blacklisted my IP too so now I have to login with a VPN. I’m not holding my breath that I won’t be prompty suspended again just for posting this reply, but I feel that it needs to be said.
At this point I am not going to respond to every single point in your reply because honstly it seems it will just be a neverending back and forth. I will respond to one point though…
You do not build a NAS with a mini pc here.
LOL. “You don’t”. I do and I will. I have every intention of doing so for my own home built DIY solution for non critical data. I’ve ordered a Dell Optiplex 3000 thin client N6005 which I will be using a ASM1166 m.2 to 6xSata adapter in a custom 3D printed case with 6 sata drives in a RAIDZ2 pool. Why? Because I want something low power for home lab use and it’s a fun project that I will document on my youtube channel. Don’t worry, I will document and post on here too so I can listen to a whole bunch of ridicule me on why I shouldn’t be dong that. Assuming I don’t get suspended or banned again, LOL.
There are ones that work reliably, it just requires some research and thorough testing. Plemty of people have home lab builds that they use for non-critical data. Home lab or not, ciritical data or not, it is still a NAS. I have done several non-supported, non-reccomended builds for personal use that are still working great many years later and proved to be far more resilient and reliable than your standard off the shelf NAS RAID solutions. What it all really boils down to is people like you saying Truenas is not for DIY home build use unless done the same way one would deploy an enterprise solution. The reality is that neither you, nor anyone else gets to decide that. You are of course welcome to your own opinion and to caution all day long as why people shouldn’t.
Also, I just want to say a big thankyou to @HoneyBadger for responding with something informative, thoughtful and helpful! While I have come acrosss most of this in my own research I never really got to that point of expaining any of this through the barage of fairly condecenting messages saying Truenas is not for that. Not to mention being suspended unjustly for having disagreeing opinions with a moderator. We need more people like you offering helpful advice and information instead of just condecendingly telling people not to do unsupported builds in their home lab environments. Sure, caution against it poltiely if you will, but the rudeness needs to stop. Thanks again for a breath of fresh air on here.
1 Like
I have to say I agree with @William_Steele here.
There is nothing wrong with someone building a TrueNAS system with 4 18TB disks on the oldest, slowest, 64-bit Intel processor you can find, only 8GB of non-ECC memory, an ancient RAID HBA with SATA 1 channels, and without any redundancy whatsoever i.e. a terrible spec. which is asking for trouble PROVIDING that they understand just how badly it will perform, just how at-risk their data is and just how little support they will get when it all goes wrong.
Of course if money was no object, and physical space was no object, and electricity usage was no object, we would all have Rolls-Royce rack-mounted bespoke-built enterprise-server-based solutions with RAIDZ3 and hot spares and dual power supplies and dual network infrastructure, and building-wide UPS. But home users, small businesses, hobbyists, generally have constraints on the money, space, time, effort etc. and are willing to make some compromises on security to get what they need within the cash and physical space etc. they have.
These compromises are mostly going to mean less redundancy than you might like, perhaps single points of temporary failure, and requiring double failures to lose data. Since many of us store e.g. media files which are non-critical and still have much better redundancy than a non-redundant single disk that the data was previously on, these can easily be levels of risk that we can accept because they are already significantly lower risk that we previously accepted.
There is nothing wrong with making compromises to get a useable system with acceptable levels of redundancy vs. risk within the budget and space you can afford - just so long as you understand and accept the risks created by the compromises you choose.
2 Likes
P.S. That said, I think it is probably quite easy for inexperienced users to cobble together a system that has the right number of ports but which is going to perform poorly or put their data at risk of corruption. In other words, they won’t understand the risks they are creating.
Indeed, despite having been in IT since before DOS PCs existed, and despite having done masses of projects including huge RAID subsystems and complex networks - even I almost succumbed to the "throw together whatever hardware has the right number of ports) and it was only the fortunate fact that the motherboard I first planned to use was dead-dead-dead that pushed me to buy a RAID appliance instead and have a couple of limited compromises rather than something that was architecturally flawed.
It is far easier to give a simple rule-of thumb to use LSA HBAs flashed to IT mode for SATA ports and avoid a whole bunch of unknown potential gotchas resulting in data loss at a later date.
1 Like
Thanks for the kind words @William_Steele - I agree with both you and @Protopia and I even had to challenge a few of my own ingrained hardware biases when SCALE released with significantly better support for hardware that was a no-go under CORE, including the ASMedia cards and Realtek NICs.
3 Likes
Just throwing my 2C in here as well. We should remember to all be professional and respectful at all times. Regardless of what particular hardware choices we all choose to make. At the end of the day we each have different risk tolerances and opinions about everything. I suspect most of you would be horrified at what I run for my own personal home-brew rigs as well 
P.S: I’m happily using a Realtek 2.5Gb onboard NIC here and it’s been working flawless on SCALE for quite some time now.
4 Likes
Well, if @kris thinks it’s OK to use compromise solutions (and if anyone knows the risks it will be him) then that is pretty much an expert endorsement that it’s OK (so long as you understand the compromises and risks you are taking).
But…
I still think that these can be distilled down to a set of rules of thumb, but these rules of thumb need to be checked every so often.
-
I still read in various places a rule of thumb that you should either use MB SATA ports or LSA HBAs flashed to IT mode. But is this still the case?
-
I also still read in various places a rule of thumb that you should only use Intel Ethernet ports. But is this still the case?
Some of us are intentially not looking to do things the easy way. I think most using Truenas for DIY homebuilds understand the risks. Of course there will be those that don’t and I think as long as they are politely reminded or informed of the risks and not scolded or talked down to, which seems to happen alot on this forum, I think this will be a more welcoming open source community. What can I say? Some of us are just masochists 
1 Like