Hi guys,
Today I found that copying files to my TrueNAS server starts out with full bandwith (113MB/sec) for a few seconds, and then drops to 2MB/sec (!).
This is only when transfering files TO the server. Copying files FROM the server has no issues and gives me a stable 113MB/sec.
I copy from my Windows 11 PC and it happens with all sorts of files (video, ISO etc.)
My specs are:
HP MicroServer gen8
Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz
16GB RAM
Media Subtype: 1000Mb/s Twisted Pair
DATA VDEV: 1 x RAIDZ1 | 4 wide | Mixed Capacity
TrueNAS version: Dragonfish-24.04.2
I have no idea where to look for the issue. I’ve searched online and on this forum but without any results. Any help is greatly appreciated.
Cheers,
Rob
Can you tell us more about the drives making up that VDEV? Specifically model numbers would be handy.
If you continue the copy process, does it stay at 2 MB/s throughout or does change?
1 Like
Hi @robnijhof
A couple things jump to mind here.
First off, I would check your drive health - have you configured or run manual SMART tests on any of them? A failing but not failed drive can cause this kind of slowness, where it will “eventually respond.”
The second is, as @neofusion points out, potentially related to SMR, or Shingled Magnetic Recording based drives. These are known to have pathological performance problems under ZFS when their “re-shingling” needs to take place to write new data.
And finally, are you by any chance using ZFS deduplication to attempt to reduce your data footprint?
1 Like
Hi neofusion,
Thanks for the quick reply.
My DATA VDEV consists of 2 models:
Model:
2x Western Digital HDD: WDC_WD20EARX-00PASB0
2x Seagate HDD: ST3000DM001-1E6166
After the initial 133MB/sec (which is obviously the limit of my Gb LAN network) it stays on ~2MB/sec throughout.
Hi,
Drive health is good. I did S.M.A.R.T. tests and they all come out fine.
I don’t know what SMR is, but I’d say with whatever drives, a 2MB/sec transfer speed is ridiculous, right?
I don’t remember turning on ZFS deduplication. Where can I find such a setting? [edit]] I found the setting: it’s off.
4/4 SMR being that they are WD Greens and Seagate Barracudas unfortunately.
Not convinced that is the only reason for possible poor performance in your case though.
Edit: As noted by HoneyBadger below it seems my assumption that all Greens and Barracudas are SMR is incorrect, perhaps it’s linked to storage size and only the bigger models turn SMR?
1 Like
How full is your pool?
You should choose NAS rated drives that are CMR.
Used Space: 17%
I have no other drives available.
Have you looked at the GUI disk reports for all four drives at the same time? Maybe you can see if any one drive is misbehaving or if they are all behaving about the same.
Not SMR, but not particularly good drives - the WD Green series is known for aggressive head-parking, and the Barracuda 3TB specifically was the subject of a class-action lawsuit due to outsized failure rates vs. other 3TB drives.
Using the “Hide Details” option under the Gear menu, or the code [details="Text] This text will be hidden [/details]
can you post the output of smartctl -A /dev/sdX
for your drives (changing the X for the actual letter of sda/sdb etc)
Look for elevated values for Reallocated_Sector_Ct
, Load_Cycle_Count
, and Current_Pending_Sector
among others.
2 Likes
Running anything else on the SCALE? Just wondering
I can’t really figure out what you mean here. I cannot find a “hide details” option in the gear menu and the code doesn’t work…
In the forums, like this:
[details=Click to view my pasted text]
Here is a bunch of pasted text!
Here is a bunch of pasted text!
Here is a bunch of pasted text!
Here is a bunch of pasted text!
Here is a bunch of pasted text!
Here is a bunch of pasted text!
[/details]
It will appear like so in your post:
Click to view my pasted text
Here is a bunch of pasted text!
Here is a bunch of pasted text!
Here is a bunch of pasted text!
Here is a bunch of pasted text!
Here is a bunch of pasted text!
Here is a bunch of pasted text!
3 Likes
namespace smartctl not found
I couldn’t figure out how to start the shell as root, so I did it from iLO:
Anything you can see from this output?
That drive is 10.5 (!) years old and is showing a few read and multi-zone errors.
With the full output we would likely be able to see how recent the errors were.
If you go to System Settings → Shell and then prefix your smartctl commands with sudo
it ought to give you the output that you can then select and paste here.
Example:
sudo smartctl -a /dev/sda
Edit:
If you specifically want a root shell you can type sudo -i
into a shell and it will give it to you.
Shame we won’t be able to ask users to do that on CORE anymore
This didn’t work. With sudo I get
namespace sudo not found
With sudo -i
I just get a login prompt for my admin account which then again isn’t able to run the smartctl
command.
namespace smartctl not found
For my root user I set the TrueNAS CLI shell, but that doesn’t run the CLI as root either.