I did PCIe pass-through of my NIC (Intel X710) through to the TrueNas VM, to ensure it wasnt proxmox being slow (even though the original iperf results didnt show issues). Same 45-50MB/s file copy.
I’m not sure, but maybe:
- is there a proxmox disk limit set?
- does the vm running truenas need multiple virtual controllers (say 3?) to make a tank virtual lvm span? I remember this for a vmware virtual server was a thing.
Grasping….hope you find a solution.
Good ideas, but I am passing through the HBA card directly to the VM, so truenas has direct access to the disks.
I believe it is an issue related to the pool’s sync setting. When I set it to disabled, I got 2.5gbps without SMB multi-path.
I tried adding a NVME SSD I had laying around as an SLOG, but that didn’t seem to help? Maybe I’m misunderstanding something, but I thought SLOGs would speed up synchronous writes and not have any affect on async writes.
I’m glad you found disabling sync allowed it to be faster. Now I wonder:
- if your Synology comparison was using sync (comparing apples to apples)?
- if your use-case requires dataset sync? E.g., maybe a stable power source (UPS) is enough to ensure your writes are committed? I am fairly certain that “canceled or interrupted” transfers at the client will not corrupt the dataset.
Your findings led me to read a lot of posts about TrueNAS SLOG devices and sync. Thanks! For example, I read that truenas Time Machine smb backup datasets use sync…seems reasonable to allow slow writes to ensure a good backup. Also I read that even separate SLOG performance will always trend closer to your original sync tests than to async. (Lots of discussions about faster SLOG devices such as Optane). Anyway, sorry I don’t have more direct empirical information.
Kind regards.
What was it set to before you changed it to Disabled? Always?
Speeding up sync writes is a whole science, but I will say that you should think of a SLOG less as something that speeds up sync writes, and more that something that makes sync writes somewhat less slow, assuming you have a device fit for the job.
Latency is an important factor of what makes a device suitable for SLOG duty, powerloss protection (PLP) is another.
I was testing between sync=always and sync=disabled, so I always knew exactly if it was snyc or async writes. I think I had assumed that when sync was disabled, it was writing only to RAM, or something. I’m still not an expert exactly how this all works.
Now I have my sync set to standard, to let the file transfer decide. I purchased an Optane P4801X which has fantastic latency, extreme flash durability, and also power loss protection. Once I get that in, I will be using that for my SLOG device instead of a consumer NVME SSD (which has neither durability or plp), but until I test, I am not sure if I will even benefit from it.
The use case of this zpool is really just NAS file storage for media servers (jellyfin), backup targets for servers/PCs, and a photo library to edit raw photos off of. I don’t know which use case and/or SMB/NFS/etc will be best, but that’s the point of a homelab 
I would keep sync=Standard if your use case is a media server. That lets TrueNAS provide sync if the client requests it. Windows clients typically do not request sync, MacOS does. Your SLOG is unlikely to see much use at all.
Databases and VM storage are when sync=Always may be warranted, or any other situation where data integrity during a sudden powerloss is paramount.
oh, wow. Your comment on windows vs macos sync request makes so much more sense now on some of my testing. I was using both windows and macos during different rounds of testing. I didn’t think there would be a difference since both were using SMB. That explains why a lot of my testing I was only able to get the 50MB/s.
Definitely no databases or VMs on my truenas zpool. For my proxmox host, I do have two optane drives in a mirror configuration. Looks like proxmox defaults to sync=standard. I might have to experiment setting that to always. I’m not worried about a powerloss event, as I have a UPS and have strict shutdown policy, but there is always the case of hardware failure or kernel faults that could cause a sudden shutdown.