Cannot saturate disk IO or CPU on writes

Fittingly, on the subject of swap and why it has been dropped from Truenas.

Upfront disclaimer: I never looked that deeply into TrueNAS, FreeBSD, or ZFS. Simply because for my TrueNAS Core system I am pretty much an ordinary user. So the following statements are guesswork, when it comes to what is actually implemented in TrueNAS et al.

From an overall systems design standpoint it makes a lot of sense to introduce such a ceiling. Because the architect has no way of knowing what workload people will be throwing at the system. My expertise is mainly in high-end transaction processing and there you never(!) fully load the system. You need to keep some capacity to respond to other incoming requests. Especially to privileged ones like an emergency shutdown. Or a high-priority job that must be executed right away.

Plus many sub-systems tend to become less stable when fully saturated. That especially includes networking, where you often see package drop rates go up under load.

Also, you need to keep in mind that ZFS was designed for high-end enterprise storage systems. So we are talking about tens of thousands of parallel requests that the system must be able to handle. So in that sense your workload (if I understand things correctly) is not something that ZFS was ever designed for.

Hope that helps.