Extremely bad disk performance

TrueNAS General
, ,
1

Hey! My read/write speeds and IO performance is terrible on a newly built setup. When using fio I get “write: IOPS=36, BW=37.5MiB/s” on sequential write and “read: IOPS=27, BW=28.5MiB/s“ on sequential read with multiple streams. When scrubbing, it takes about 10-15 days.

My setup is using a Truenas scale 25.10.0.1 VM on Proxmox (with cpu: “host”, on a ryzen 9 7900) with 20GB dedicated RAM, an L2ARC device with 128GB and an SLOG device with 32GB (both virtual disks from proxmox, from a zfs mirror pool on two enterprise SSDs). I am using PCIe passthrough of a sata expension card connected to 4x28TB drives (ST28000NM000C) in a RAIDZ2 pool with ZFS native encryption.

Any help would be appreciated! I do not know how to troubleshoot this.

2

Remove the slog and l2arc. You probably dont need them and adding a virtualisation layer to them totally defeats their purpose.

If the SATA expansion card is not a HBA in IT mode this could also add to your bad performance.

1 Like
3

I removed the slog and l2arc and the issue persists (I get “write: IOPS=31, BW=33.2MiB/s” for sequential read). The SATA expansion card is a HBA in IT mode.

4

edit: meant write*

5

Is this a new problem ? Did it work before ?
What type of HBA ? Does it get cooled enough ?

6

New problem but also a new setup. I believe you were right about the problem being with my SATA expansion card, it uses the AMS1062 controller which seems to have many issues. Logs show “hard resetting link” and “port multiplier”. I have ordered a proper LSI SAS 9300-16I 16 Port HBA card and will update the post when I’ve tried it. Thanks!

3 Likes
7

Oh yeah yuck, that chip only provides two actual SATA interfaces. Most likely the first port is real and the other three are on a cheap splitter chip, which works fine if you only need to access one or two at a time, but not so much with simultaneous access on all ports.

Getting a real controller should definitely get you to rational usability.

8

Just a reminder to anyone pondering building their own server. If your motherboard doesn’t have enough connectors, don’t grab just any sata expander. Get an hba, lsi are not expensive and are plentiful. Most come with their own cables. They are pcie devices so they get enough bandwidth to let all the drives run full speed.

Average about $50 USD on ebay or Amazon but that depends heavily on the model and number of ports. Need to be flashed to IT mode and then they are pretty much plug and play. Many sellers advertise this and even Truenas in the descriptions.

Last caveat, they can get hot. They expect to live in a server case where it’s usually a wind tornado all day so keep that in mind. Yours may or may not need additional cooling.

2 Likes