10gb first fast than drops to 0

[Gerco_van_der_Boon]

Hi,

I have two TrueNas Scale servers. One contains the data, the other the backup.
I have connected the two servers using 10gb fiber optical cable.
I use rsync for backing up the data.
It used to work fine but now this happens:

• when rsync job starts the data is transferred at 6gb speed. I can see the zfs cache fill up until there is 0.6gb ram free left. Then the transfer falls back to 0. Like it suffocates… ?

I did upgrade Ilo and Bios recently on the back server (ml310e gen 8).

Does any1 have an idea what I could try?

[LarsR]

we need more information about the system, pool layout, how much ram?
You’re most likely bottlenecked by your pool speeds, but without more information we can only guess.

[Gerco_van_der_Boon]

Thank you for your reply, @LarsR

The server is an HP Proliant ML310e, 32GB RAM (HP smart-memory; 1600 mhz), disks are 4 * 10 TB Nas Drives (WD Red) and the Pool layout is a zfs-pool, stripe (I know, I know …).

[Gerco_van_der_Boon]

I think I can’t edit my post but in addition:
the pool is 36TB in size and uses128kb blocks.

[HoneyBadger]

@Gerco_van_der_Boon

ml310e gen 8

I suspect the storage controller; specifically the HP onboard B120i likes to disable drive cache for SATA drives.

Can you drop me the results of

for disk in /dev/sd?; do; hdparm -W $disk; done

in a command prompt?

[Gerco_van_der_Boon]

@HoneyBadger

Thanks for the suggestion. Here’s the result of the query, and I have also included a list of the drives.

/dev/sda:
write-caching = 0 (off)

/dev/sdb:
write-caching = 1 (on)

/dev/sdc:
write-caching = not supported

/dev/sdd:
write-caching = 0 (off)

/dev/sde:
write-caching = 0 (off)

/dev/sdf:
write-caching = not supported

/dev/sdg:
write-caching = not supported

/dev/sdh:
write-caching = 0 (off)

/dev/sdi:
write-caching = not supported

/dev/sdj:
write-caching = 0 (off)

root@truenas2[~]# ls -ls /dev/disk/by-id
total 0
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 ata-WDC_WD100EFAX-68LHPN0_JEHTJ67N → …/…/sde
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 ata-WDC_WD100EFAX-68LHPN0_JEHTJ67N-part1 → …/…/sde1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 ata-WDC_WD100EFAX-68LHPN0_JEHU4HDN → …/…/sdd
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 ata-WDC_WD100EFAX-68LHPN0_JEHU4HDN-part1 → …/…/sdd1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 ata-WDC_WD101KFBX-68R56N0_1EJU0YWZ → …/…/sda
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 ata-WDC_WD101KFBX-68R56N0_1EJU0YWZ-part1 → …/…/sda1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0680442 → …/…/sdb
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 ata-WDC_WDS500G1R0A-68A4W0_23464V802193 → …/…/sdh
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 ata-WDC_WDS500G1R0A-68A4W0_23464V802193-part1 → …/…/sdh1
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 ata-WDC_WDS500G1R0A-68A4W0_23464V802193-part2 → …/…/sdh2
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 ata-WDC_WDS500G1R0A-68A4W0_23464V802193-part3 → …/…/sdh3
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 ata-WDC_WDS500G1R0A-68A4W0_23464V802193-part4 → …/…/sdh4
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 ata-WDC_WUS721010ALE6L4_VH0A393M → …/…/sdj
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 ata-WDC_WUS721010ALE6L4_VH0A393M-part1 → …/…/sdj1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 ata-hp_DVD-RAM_GH82N_K4PD4CD4757 → …/…/sr0
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 dm-name-sdh4 → …/…/dm-0
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 dm-uuid-CRYPT-PLAIN-sdh4 → …/…/dm-0
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 usb-ASMedia_USB_3.0_Device_000000000410-0:0 → …/…/sdb
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 usb-HP_iLO_LUN_00_Media_0_000002660A01-0:0 → …/…/sdf
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 usb-HP_iLO_LUN_00_Media_0_000002660A01-0:0-part1 → …/…/sdf1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 usb-HP_iLO_LUN_01_Media_0_000002660A01-0:1 → …/…/sdg
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 usb-HP_iLO_LUN_01_Media_0_000002660A01-0:1-part1 → …/…/sdg1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 usb-HP_iLO_LUN_02_Media_0_000002660A01-0:2 → …/…/sdi
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 usb-HP_iLO_LUN_02_Media_0_000002660A01-0:2-part1 → …/…/sdi1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 usb-Seagate_Game_Drive_PS4_NZ0D0R75-0:0 → …/…/sdc
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 usb-Seagate_Game_Drive_PS4_NZ0D0R75-0:0-part1 → …/…/sdc1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 wwn-0x5000cca0d8c49813 → …/…/sdj
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 wwn-0x5000cca0d8c49813-part1 → …/…/sdj1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 wwn-0x5000cca267d93d3a → …/…/sde
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 wwn-0x5000cca267d93d3a-part1 → …/…/sde1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 wwn-0x5000cca267d985a9 → …/…/sdd
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 wwn-0x5000cca267d985a9-part1 → …/…/sdd1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 wwn-0x5000cca27ee78fe7 → …/…/sda
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 wwn-0x5000cca27ee78fe7-part1 → …/…/sda1
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 wwn-0x50014ee209bcf475 → …/…/sdb
0 lrwxrwxrwx 1 root root 9 Oct 30 15:42 wwn-0x5001b448b353f771 → …/…/sdh
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 wwn-0x5001b448b353f771-part1 → …/…/sdh1
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 wwn-0x5001b448b353f771-part2 → …/…/sdh2
0 lrwxrwxrwx 1 root root 10 Oct 30 15:42 wwn-0x5001b448b353f771-part3 → …/…/sdh3
0 lrwxrwxrwx 1 root root 10 Oct 30 15:43 wwn-0x5001b448b353f771-part4 → …/…/sdh4

[HoneyBadger]

That’s likely the culprit.

Picking out your 10TB disks from the mix here:

@Gerco_van_der_Boon

/dev/sda:
write-caching = 0 (off)

/dev/sdd:
write-caching = 0 (off)

/dev/sde:
write-caching = 0 (off)

/dev/sdj:
write-caching = 0 (off)

None of them have write caching enabled.

Let’s hit them all in sequence here, with a bonus of enabling it on your boot WD Red SSD:

hdparm -W1 /dev/sda
hdparm -W1 /dev/sdd
hdparm -W1 /dev/sde
hdparm -W1 /dev/sdj
hdparm -W1 /dev/sdh

If this helps, then the “permanent fix” is to check that your storage controller is set to “AHCI/SATA” or “Non-RAID” mode in the BIOS, and then look for a setting in your BIOS for “Physical Drive Write Cache State” and set it to Enabled. ZFS is able to control the cache on individual drives and will send them flush commands for data safety.

If there’s no BIOS setting, then you could set up a scheduled task to re-enable write caching after each boot, but let’s see if this fixes your issue.

I also see that you’ve got a couple USB-connected drives (the 3TB WD and the PS4 Seagate Game Drive) - USB attached drives aren’t generally encouraged with TrueNAS and the PS4 one specifically is a “shingled” or SMR drive which is doubly suspect. What are these used for?

[Gerco_van_der_Boon]

@HoneyBadger
In the SETUP I saw ‘drive write cache’ was disabled. I have enabled it and ran the query again:

root@truenas2[~]# for disk in /dev/sd?; do; hdparm -W $disk; done

/dev/sda:
write-caching = 1 (on)

/dev/sdb:
write-caching = 1 (on)

/dev/sdc:
write-caching = not supported

/dev/sdd:
write-caching = 1 (on)

/dev/sde:
write-caching = 1 (on)

/dev/sdf:
write-caching = not supported

/dev/sdg:
write-caching = 1 (on)

/dev/sdh:
write-caching = 1 (on)

[HoneyBadger]

Looking good. Let the replication job fire off again, see if you don’t get significantly improved results!

[Gerco_van_der_Boon]

@HoneyBadger

hdparm -W1 /dev/sdh

Sorry, I think me previous message arrived at the same time as you sent yours.

root@truenas2[~]# hdparm -W1 /dev/sda

/dev/sda:
setting drive write-caching to 1 (on)
write-caching = 1 (on)
root@truenas2[~]# hdparm -W1 /dev/sdd

/dev/sdd:
setting drive write-caching to 1 (on)
write-caching = 1 (on)
root@truenas2[~]# hdparm -W1 /dev/sde

/dev/sde:
setting drive write-caching to 1 (on)
write-caching = 1 (on)
root@truenas2[~]# hdparm -W1 /dev/sdj
/dev/sdj: No such file or directory
root@truenas2[~]# hdparm -W1 /dev/sdh

/dev/sdh:
setting drive write-caching to 1 (on)
write-caching = 1 (on)

[Gerco_van_der_Boon]

@HoneyBadger

I got the same results, alas.
For a minute it’s working fine, but then it sort of suffocates, again.

Maybe I have an incorrect network setup? Let me explain.
I have a HP Fiber network controller that has two SFP+ ports. One of the ports is connected to the internet and in my network (192.168.1.x), this to make my backups available for every device connected to the 192.168.1.x-network.
The other SFP+ port is connected directly to the server using 192.168.3.5 as IP address.

I have mounted /mnt/backup/video to 192.168.3.6/hpprox/video (server IP address that has the same HP network controller).
I use:
rsync -rau --no-perms /mnt/video /mnt/backup/video
to synchronise my movies and tv-series every three hours (job in crontab) and once a month I I have the same job, plus the --delete parameter to mirror (if I accidentally delete a file I have until the 1st of the month to restore if needed).

The weird thing is, this has been working for a year, so I am a bii puzzled what has happened, other than the ILO and firmware update. I can’t seem to access Smart Provisioning either. Could this be related to the issue I have? Should I do a server reset for example?

[Gerco_van_der_Boon]

Pressed SEND too soon.

Just so you know: when I use Rsync via TrueNas’ Data Protection / Rsync tasks, it uses the 192.168.1.x network and I notice that when ZFS cache is full and 0.3 GB memory is left, data must be written from cache to disk because free memory goes up and down like an accordion. This is good, I think.

Only when using the 192.168.3.x 10GB network it seems to suffocate.

[HoneyBadger]

10Gbps is probably beyond the capability of the destination system to receive, but I’d think you would be seeing a more gradual slope and “leveling off” of the transfer at the actual throughput speeds of your vdev as the ZFS write throttle starts to constrict things down.

However this is giving me some questions:

@Gerco_van_der_Boon

I have mounted /mnt/backup/video to 192.168.3.6/hpprox/video (server IP address that has the same HP network controller).

Are you saying that the source system (192.168.3.5) is mounting the filesystem from the destination (192.168.3.6) as a local path (potentially over NFS or SMB?) rather than sending an rsync job, ZFS replication stream, or other method of transfer?

[Gerco_van_der_Boon]

Yes, I am very used to using Rsync in this fashion. Bad habit maybe…
And I am pulling from the backup system. Should I give pushing from the source system a go and/or are you suggesting to use ZFS replication?

[Gerco_van_der_Boon]

And thank you for all your replies and effort. I really appreciate that a lot.

[HoneyBadger]

I’d definitely look at setting up a regular ZFS replication job (as a PULL from the backup system) as it will likely be much faster and more efficient, as well as letting you preserve your snapshot chains.

[Gerco_van_der_Boon]

I am using TrueNas replication now and have setup a SSH-connection through the 10GB network card. It seems to be working fine now.
And indeed, that replication works pretty nice!