Write performance on mirrored drives

[caligulaa]

Hello, new user coming from unraid.

I setup a TrueNAS core machine to store some videos for work.

-2x Seagate IronWolf 12TB mirrored
( NAS Internal Hard Drive HDD – CMR 3.5 Inch SATA 6Gb/s 7200 RPM 256MB Cache for RAID Network Attached Storage (ST12000VNZ008/ST12000VN0008)
-intelCore i7 4770 cpu 8 threads
-8 gigs of ram
-SSD for OS

My max write speed is around 90MB/sec, reads are able to “saturate” the gigabit link. I know that these drives have faster writes than this. What is my bottleneck? The fact that they are mirrored, low ram, no slog drive?

Tried to find some answers online, saw people with worse performance than I had. Don’t want to fall in yet another rabbit hole!

Would like to eventually setup a 10 gigabit network to have a couple of editors editing off an SSD array and use the two mirrored drives for archival purposes.

Thanks for your input!
A

[etorix]

SLOG is only for sync writes, which are slower than async anyway.
The main suspect would be low RAM.

[Johnny_Fartpants]

RAM is rather low. Have you tried any local pool benchmarks like fio? That would be a good place to start before benchmarking clients over the network.

[caligulaa]

don’t know if I’m doing this the correct way. Entering this command in shell where the dataset is mounted:

[Johnny_Fartpants]

224MB/s which sounds about right for a 2 drive mirror. So this suggests your pool and drives are working fine. How is your client connected to your network? Could you run an iperf from client to NAS to confirm network speed?

If all that checks out then I’d be focusing on your RAM and trying to bump that up to at least 16GB and see if that improves things.

[Stux]

Mirrors write at the speed of a single disk in the mirror.

(Read at single speed * number of disks)

HD perf between 100-250MB/s based on where the data is located on the disk is perfectly reasonable.

Then factor in fragmentation.

How full is your pool?

More mirrors. More speed.

[caligulaa]

Upped the ram to 32gigs. Still getting the same speed unfortunatly.

truenas iperf server:
with -P8 getting 808Mbits/sec
single threaded getting 766

truenas client:
with -P6 939Mbtis/sec
929Mbits/sec

Which reflects what i’m seeing when copying files via SMB from windows to the truenas and vice-versa.

I will be adding a 10gig card next week and keep you updated. Looks like the built in nic in this HP computer is not performing at full capacity?

The only thing in between the client and the truenas is a netgear switch ProSafe GS728TPP in the current configuration.

Keep you posted once i change the config to leverage 10gig on the truenas and 2.5gig with clients…

Knock on wood!

Thanks for the help.

A

[caligulaa]

20%, I also have hourly snapshots turned on… Don’t think that should impact write speeds. From the iperf tests it looks like the current nic is maxxing out at the speeds I’m seeing when moving files around.

A

[MBILC]

You do only have a 1GB NIC so in theory your max speeds would be around that 115MB/s (little more or less)

And then depending on the source and back, are you moving large files or many smaller files?

[Sara]

Do you use smb? Maybe too low single core performance with that old CPU.

[caligulaa]

I’m using smb,

I just got a 10gig card setup on the server, and 2.5gig on a client.
Iperf3 shows 1.99Gbits/sec speed bandwith with -P8. 1.5GB otherwise. Which seems to mach up with my read and write performance. Why am I leveraging multiple threads when writing but not when reading? Could it be a firewall issue? My client has the firewall on for the 10gig network…?

Is there a way to configure more threads on SMB with truenas?

SMB performance: (Video files 1-3GB & up)
Write performance is good at 225MB/sec now
read is capping at 190MB/sec.

Any idea what could be happening with the reads? I thought that I should be able to get higherspeeds maxing out my 2Gbits/sec bandwith with arc ?

Local performance:
When running fio locally on the truenas on read test I get around 9 gigs for small files when bypassing arc 232 MB/sec

I’m using a mikotik switch (CRS305-1G-4S+IN) and on the client
Plugable 2.5GB USB Ethernet Adapter for Laptop, 2-in-1 Cable, USB 2.5GB

[caligulaa]

I posted an update after changing some networking equipment, curious to see what you thought. Thanks, have a nice weekend!

A

[NickF1227]

@caligulaa

SMB performance: (Video files 1-3GB & up)
Write performance is good at 225MB/sec now
read is capping at 190MB/sec.

For two spinning hard drives in a single mirror vdev, this is about what I would expect.

[Sara]

@caligulaa

Iperf3 shows 1.99Gbits/sec speed bandwith

Strange that you are unable to reach 2.5G, but as far as I know, iperf is also not multithread.

@caligulaa

Is there a way to configure more threads on SMB with truenas?

I don’t think so.

@caligulaa

Any idea what could be happening with the reads? I thought that I should be able to get higherspeeds maxing out my 2Gbits/sec bandwith with arc ?

That is pretty close to maxing out your 2.5G already.

To be honest, I am surprised that you get that good of a performance with such a shitty USB NIC.
What controller does that thing use?

[MBILC]

@caligulaa

Iperf3

iPerf3 had a beta out in 2023 (Multi-threaded iperf3 public beta 1 · esnet/iperf · Discussion #1495 · GitHub) for multi-threaded, some other just run multiple instances:

iperf3 at 40Gbps and above

iPerf performance will not rely on any I/O from storage, so if those are the max speeds, then look likely to your USB network dongle if anything…could just be getting too warm and cant max out 2.5Gbps