Large SMB write freezes all other client writes on TrueNAS Scale server

TrueNAS General
,
1

This issue has been plaguing us since we switched from Core to Scale.

When copying large amount of large files to the server (+50GB) via SMB, the data rate starts at +200MB/s as expected, then fall gradually bellow 5MB/s and prevents any other clients from saving files, causing programs like LibreOffice and Acrobat to crash when we try to save projects but can’t.

The server is a Supermicro X10DRU-i+ loaded with 12x 8GB Sata HDD. The server is running TrueNAS Scale 24.04.2.2. The drives are configured as a single RaidZ2 pool 12 wide.

The server handles large video files for video editing. A 10GBe QNAP QSW-M2108R-2C network switch handles communication between the server and 5 PCs (Windows 10-11) benefit from the fast transfer speeds. Below that is an old standard 1GBe business switch that handle traffic from other client with more mundane office needs.

The issue occurs during large write evens, like dumping the content of a 256GB memory card to the server via SMB with a PC using OffShoot. During this event, the file transfer starts at +200MB/s as expected, but may suddenly decline to 5MB/s or less and remain there indefinitely, until the copy is canceled. While the server is in this “overwhelmed” state, no other write task can be completed by other clients, even for small file sizes.

I have noticed that those events coincide with a low level of free memory on the system. All disks will be writing at around 50MB/s, and then suddenly stop writing. Reporting data shows the CPU never goes above 25% usage.

We only had 128GB of ECC DDR4 on the system. So we upgraded to 384GB, hoping this would resolve the issue. We also added a 500GB SSD ARC2 Cache on the pool, for good measure. But the cache just fills up over time and once it’s full, the issue occurs again, usually when free memory is around 20GB. The issue never occurs with a lot of free memory is available.

Current used space on the server is at around 83%. I know that’s high. But freeing space is the first thing we tried months ago (getting it bellow 75%), and that did not resolve the issue.

The only workaround is to reset the server once a week to empty the cache. The cache doesn’t clear on it’s own and when it’s full now leaves only 35GB of free memory. I don’t know if that’s normal behavior.

As you can see, my investigation so far has been leaning towards the memory and cache. I have found no reason to suspect the network or the drives themselves.

I’m running out of ideas and I’m open to new insights.
Thanks

2

Did you put in a Bug ticket along with a debug dump when it is occurring?

Top of GUI there is a smile icon. Feedback / Report A Bug.

3

  1. What do you mean by “low free memory”? In TrueNAS Scale, low free memory is absolutely normal because most of it is taken up by ARC. 35GB of free memory sounds completely normal to me. So the real question is what the size of your ARC cache is?

  2. 128GB of memory sounds OK in principle, especially if you are not running apps or VMs on this system.

  3. Initial writes from Windows via SMB are asynchronous, and they run at network speeds and are effectively written into memory until you have reached the maximum amount of memory allowed for storing these pending writes. The memory for this is taken from ARC, and the limit is there to protect ARC from being completely trashed. 200MB/s+ is roughly equivalent to 1.6Gb/s which is substantially less than the 10Gb/s network speed, but it may be limited by Windows. Once this memory is full, then network transfer speeds will slow down to match the rate that data can be written from memory to disk.

  4. Adding the L2ARC will not do anything to help write speeds, and may actually have a negative impact on read and possibly write performance. My advice is to remove it.

  5. Rebooting to “empty the cache” is absolutely NOT the thing to do - the only impact will be to lower your read speeds whilst the cache is rebuilt.

  6. So the thing we need to focus upon is what we can assume is 5MB/s disk write speeds, which is extremely low, and these are definitely worth worrying about. But aside from knowing that the HDDs are 12x 8GB we don’t know anything about your disk subsystem (controller, disk models, SMART status, pool status etc.).

To get us started on this, can you please run the following commands:

  • lsblk -bo NAME,MODEL,ROTA,PTTYPE,TYPE,START,SIZE,PARTTYPENAME,PARTUUID
  • lspci
  • sas2flash -list
  • sas3flash -list
  • sudo zpool status -v
  • smartctl -x /dev/sdX for one of the 12 HDDs.

Can you also take a look at the Report and Netdata screens and see if you can spot anything that looks odd?

Thanks.

4

Thank you for your interrest in my issue.

What do you mean by “low free memory”?

I’m refering to what is identified in the memory report as “free” (pink). It’s currently at 25GB during off-hours with the system doing nothing.
“Used” memory is always stable at 25GB.
“Cached” memry is at 328GB.
I don’t know what normal behaviour is. I just know that when a write task is started, Cache goes up until Free reaches a floor of about 22GB, then writing to the drives stops (we can see it in the report) and all client experience a loss of service until we cancel the task on the client PC that created the issue with the large write task.

200MB/s+ is roughly equivalent to 1.6Gb/s which is substantially less than the 10Gb/s network speed

The server is on 10GBe, but the client PCs have 2.5GBe. Write speeds between 250 and 350MB/s may be limited by CFast reader memory card read speed on the PC. This is not at issue here. I just put +200 to signify this is the fast speed that we expect during normal operations.

Adding the L2ARC will not do anything to help write speeds, and may actually have a negative impact on read and possibly write performance. My advice is to remove it.

It’s new from last week, so I want to give it more time and see what happens. Considering the amount of data in a average video project (+500GB per project) and that we sometimes have 3 or 4 projects open on different clients at once, I feel like we are probably a good use case for an L2ARC cache, based on my understanding of what it does, and the amount of memory already on the system.

Rebooting to “empty the cache” is absolutely NOT the thing to do - the only impact will be to lower your read speeds whilst the cache is rebuilt.

When the server becomes unresponsive, what else can I do but reboot?

Can you also take a look at the Report and Netdata screens and see if you can spot anything that looks odd?

I’m not familiar with Netdata, but I will install it and have a look at it. I’ve looked at the reports extensively and the behaviour with memory is the only correlation that caught my attention. That and the fact that the drives stop writing.

Here is the report dump the requested commands:


lsblk -bo
NAME        MODEL                ROTA PTTYPE TYPE    START          SIZE PARTTYPENAME             PARTUUID
sda         SATA SSD                0 gpt    disk            64023257088                          
├─sda1                              0 gpt    part     4096       1048576 BIOS boot                12befc97-06b9-495d-bf6b-26e1489884de
├─sda2                              0 gpt    part     6144     536870912 EFI System               07cf627c-f016-44e2-9008-2f5bd29ee0fd
└─sda3                              0 gpt    part  1054720   63483223552 Solaris /usr & Apple ZFS e62de453-f0d0-455a-b418-4f5c05e94b5b
sdb         HGST HUS728T8TALE6L4    1 gpt    disk          8001563222016                          
├─sdb1                              1 gpt    part      128    2147483648 FreeBSD swap             580bf001-da35-11ec-b915-002590f482d6
│ └─md125                           1        raid1            2144337920                          
│   └─md125                         1        crypt            2144337920                          
└─sdb2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              58d4b355-da35-11ec-b915-002590f482d6
sdc         ST8000VN004-2M2101      1 gpt    disk          8001563222016                          
├─sdc1                              1 gpt    part      128    2147483648 FreeBSD swap             57e0cc86-da35-11ec-b915-002590f482d6
│ └─md127                           1        raid1            2144337920                          
│   └─md127                         1        crypt            2144337920                          
└─sdc2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              587c9134-da35-11ec-b915-002590f482d6
sdd         ST8000VN004-2M2101      1 gpt    disk          8001563222016                          
├─sdd1                              1 gpt    part      128    2147483648 FreeBSD swap             582413f0-da35-11ec-b915-002590f482d6
│ └─md126                           1        raid1            2144337920                          
│   └─md126                         1        crypt            2144337920                          
└─sdd2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              589b3ac7-da35-11ec-b915-002590f482d6
sde         HGST HUS728T8TALE6L4    1 gpt    disk          8001563222016                          
├─sde1                              1 gpt    part      128    2147483648 FreeBSD swap             57eda0e7-da35-11ec-b915-002590f482d6
│ └─md127                           1        raid1            2144337920                          
│   └─md127                         1        crypt            2144337920                          
└─sde2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              587a03f3-da35-11ec-b915-002590f482d6
sdf         ST8000VN004-2M2101      1 gpt    disk          8001563222016                          
├─sdf1                              1 gpt    part      128    2147483648 FreeBSD swap             58137f00-da35-11ec-b915-002590f482d6
│ └─md127                           1        raid1            2144337920                          
│   └─md127                         1        crypt            2144337920                          
└─sdf2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              58f69e05-da35-11ec-b915-002590f482d6
sdg         WDC WD80EFAX-68KNBN0    1 gpt    disk          8001563222016                          
├─sdg1                              1 gpt    part      128    2147483648 FreeBSD swap             57dc3fba-da35-11ec-b915-002590f482d6
│ └─md125                           1        raid1            2144337920                          
│   └─md125                         1        crypt            2144337920                          
└─sdg2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              58308e18-da35-11ec-b915-002590f482d6
sdh         HGST HUS728T8TALE6L4    1 gpt    disk          8001563222016                          
├─sdh1                              1 gpt    part      128    2147483648 FreeBSD swap             5810deae-da35-11ec-b915-002590f482d6
│ └─md124                           1        raid1            2144337920                          
│   └─md124                         1        crypt            2144337920                          
└─sdh2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              58a7a4f1-da35-11ec-b915-002590f482d6
sdi         ST8000DM004-2CX188      1 gpt    disk          8001563222016                          
├─sdi1                              1 gpt    part      128    2147483648 FreeBSD swap             5800cf91-da35-11ec-b915-002590f482d6
│ └─md124                           1        raid1            2144337920                          
│   └─md124                         1        crypt            2144337920                          
└─sdi2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              59049f0e-da35-11ec-b915-002590f482d6
sdj         WDC WD80EFZX-68UW8N0    1 gpt    disk          8001563222016                          
├─sdj1                              1 gpt    part      128    2147483648 FreeBSD swap             580e00a4-da35-11ec-b915-002590f482d6
│ └─md126                           1        raid1            2144337920                          
│   └─md126                         1        crypt            2144337920                          
└─sdj2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              58d8c4f7-da35-11ec-b915-002590f482d6
sdk         HGST HUS728T8TALE6L4    1 gpt    disk          8001563222016                          
├─sdk1                              1 gpt    part      128    2147483648 FreeBSD swap             58100b1e-da35-11ec-b915-002590f482d6
│ └─md125                           1        raid1            2144337920                          
│   └─md125                         1        crypt            2144337920                          
└─sdk2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              5901a464-da35-11ec-b915-002590f482d6
sdl         WDC WD80EFAX-68KNBN0    1 gpt    disk          8001563222016                          
├─sdl1                              1 gpt    part      128    2147483648 FreeBSD swap             581f584b-da35-11ec-b915-002590f482d6
│ └─md126                           1        raid1            2144337920                          
│   └─md126                         1        crypt            2144337920                          
└─sdl2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              58d6b3d3-da35-11ec-b915-002590f482d6
sdm         WDC WD80EFAX-68KNBN0    1 gpt    disk          8001563222016                          
├─sdm1                              1 gpt    part      128    2147483648 FreeBSD swap             582169c9-da35-11ec-b915-002590f482d6
│ └─md124                           1        raid1            2144337920                          
│   └─md124                         1        crypt            2144337920                          
└─sdm2                              1 gpt    part  4194432 7999415652352 FreeBSD ZFS              58a58401-da35-11ec-b915-002590f482d6
nvme0n1     WDS500G3X0C-00SJG0      0 gpt    disk           500107862016                          
└─nvme0n1p1                         0 gpt    part     4096  500105740800 Solaris /usr & Apple ZFS 9f2a921d-cda9-4f95-bb20-6d84a115addb


lspci
00:00.0 Host bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 (rev 02)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:04.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 (rev 02)
00:04.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 (rev 02)
00:04.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 (rev 02)
00:04.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 (rev 02)
00:04.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 (rev 02)
00:04.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 (rev 02)
00:04.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 (rev 02)
00:04.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 (rev 02)
00:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
00:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 02)
00:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
00:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 02)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.2 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #3 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
02:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
7f:08.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
7f:08.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
7f:08.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
7f:09.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
7f:09.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
7f:09.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
7f:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
7f:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
7f:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
7f:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0d.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0d.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0d.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0d.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
7f:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
7f:0f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
7f:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
7f:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
7f:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
7f:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
7f:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
7f:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
7f:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
7f:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
7f:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
7f:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
7f:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
7f:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
7f:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
7f:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
7f:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
7f:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
7f:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
7f:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)
7f:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
7f:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)
7f:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)
7f:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)
7f:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)
7f:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
7f:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
7f:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
7f:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
7f:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
7f:16.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
7f:16.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)
7f:16.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)
7f:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)
7f:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
7f:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)
7f:17.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control (rev 02)
7f:17.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers (rev 02)
7f:17.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers (rev 02)
7f:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
7f:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
7f:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
7f:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
7f:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
7f:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
7f:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
7f:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
7f:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
7f:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)
7f:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)
80:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
80:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
80:04.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 (rev 02)
80:04.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 (rev 02)
80:04.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 (rev 02)
80:04.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 (rev 02)
80:04.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 (rev 02)
80:04.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 (rev 02)
80:04.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 (rev 02)
80:04.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 (rev 02)
80:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
80:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 02)
80:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
80:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 02)
81:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD
82:00.0 PCI bridge: PLX Technology, Inc. PEX 8624 24-lane, 6-Port PCI Express Gen 2 (5.0 GT/s) Switch [ExpressLane] (rev bb)
83:04.0 PCI bridge: PLX Technology, Inc. PEX 8624 24-lane, 6-Port PCI Express Gen 2 (5.0 GT/s) Switch [ExpressLane] (rev bb)
83:05.0 PCI bridge: PLX Technology, Inc. PEX 8624 24-lane, 6-Port PCI Express Gen 2 (5.0 GT/s) Switch [ExpressLane] (rev bb)
83:08.0 PCI bridge: PLX Technology, Inc. PEX 8624 24-lane, 6-Port PCI Express Gen 2 (5.0 GT/s) Switch [ExpressLane] (rev bb)
85:00.0 Serial Attached SCSI controller: PMC-Sierra Inc. Device 8001 (rev 05)
86:00.0 Serial Attached SCSI controller: PMC-Sierra Inc. Device 8001 (rev 05)
ff:08.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
ff:08.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
ff:08.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
ff:09.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
ff:09.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
ff:09.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
ff:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
ff:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
ff:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)
ff:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
ff:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)
ff:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)
ff:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)
ff:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)
ff:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
ff:16.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
ff:16.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)
ff:16.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)
ff:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)
ff:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
ff:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)
ff:17.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control (rev 02)
ff:17.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers (rev 02)
ff:17.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers (rev 02)
ff:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)
ff:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)


sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

        No LSI SAS adapters found! Limited Command Set Available!
        ERROR: Command Not allowed without an adapter!
        ERROR: Couldn't Create Command -list
        Exiting Program.


sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

        Adapter Selected is a Avago SAS: SAS3008(C0)

        Controller Number              : 0
        Controller                     : SAS3008(C0)
        PCI Address                    : 00:03:00:00
        SAS Address                    : 5003048-0-1b3d-af01
        NVDATA Version (Default)       : 0e.01.30.28
        NVDATA Version (Persistent)    : 0e.01.30.28
        Firmware Product ID            : 0x2221 (IT)
        Firmware Version               : 16.00.01.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : LSI3008-IT
        BIOS Version                   : 08.37.00.00
        UEFI BSD Version               : 18.00.00.00
        FCODE Version                  : N/A
        Board Name                     : LSI3008-IT
        Board Assembly                 : N/A
        Board Tracer Number            : N/A

        Finished Processing Commands Successfully.
        Exiting SAS3Flash.


sudo zpool status -v
  pool: StoragePool
 state: ONLINE
  scan: scrub repaired 0B in 15:17:01 with 0 errors on Sun Nov  3 14:17:13 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        StoragePool                               ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            58d4b355-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            587c9134-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            589b3ac7-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            587a03f3-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            58f69e05-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            58308e18-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            58a7a4f1-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            59049f0e-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            58d8c4f7-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            5901a464-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            58d6b3d3-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
            58a58401-da35-11ec-b915-002590f482d6  ONLINE       0     0     0
        cache
          9f2a921d-cda9-4f95-bb20-6d84a115addb    ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:11 with 0 errors on Sun Dec  1 03:45:12 2024
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors


smartctl -x /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.32-production+truenas] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     HGST Ultrastar HC310/320
Device Model:     HGST HUS728T8TALE6L4
Serial Number:    VDJAMW0K
LU WWN Device Id: 5 000cca 0bbe10550
Firmware Version: V8GNW460
User Capacity:    8,001,563,222,016 bytes [8.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Dec  6 19:44:17 2024 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Disabled
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   87) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 841) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   100   016    -    0
  2 Throughput_Performance  P-S---   132   132   054    -    96
  3 Spin_Up_Time            POS---   175   175   024    -    523 (Average 405)
  4 Start_Stop_Count        -O--C-   100   100   000    -    143
  5 Reallocated_Sector_Ct   PO--CK   100   100   005    -    0
  7 Seek_Error_Rate         PO-R--   100   100   067    -    0
  8 Seek_Time_Performance   P-S---   128   128   020    -    18
  9 Power_On_Hours          -O--C-   096   096   000    -    30940
 10 Spin_Retry_Count        PO--C-   100   100   060    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    142
192 Power-Off_Retract_Count -O--CK   099   099   000    -    1414
193 Load_Cycle_Count        -O--C-   099   099   000    -    1414
194 Temperature_Celsius     -O----   166   166   000    -    36 (Min/Max 19/58)
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O---K   100   100   000    -    0
198 Offline_Uncorrectable   ---R--   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O-R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL     R/O    256  Device Statistics log
0x04       SL      R/O    255  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x0c       GPL     R/O   5501  Pending Defects log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x12       GPL     R/O      1  SATA NCQ Non-Data log
0x13       GPL     R/O      1  SATA NCQ Send and Receive log
0x15       GPL     R/W      1  Rebuild Assist log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    256  Current Device Internal Status Data log
0x25       GPL     R/O    256  Saved Device Internal Status Data log
0x2f       GPL     R/O      1  Set Sector Configuration
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     30791         -
# 2  Extended offline    Completed without error       00%     30623         -
# 3  Extended offline    Completed without error       00%     30456         -
# 4  Extended offline    Completed without error       00%     28295         -
# 5  Extended offline    Completed without error       00%     28128         -
# 6  Extended offline    Completed without error       00%     27959         -
# 7  Extended offline    Completed without error       00%     27793         -
# 8  Extended offline    Completed without error       00%     27623         -
# 9  Extended offline    Completed without error       00%     27455         -
#10  Extended offline    Completed without error       00%     27287         -
#11  Extended offline    Completed without error       00%     27119         -
#12  Extended offline    Completed without error       00%     26951         -
#13  Extended offline    Completed without error       00%     26783         -
#14  Extended offline    Completed without error       00%     26615         -
#15  Extended offline    Completed without error       00%     26447         -
#16  Extended offline    Completed without error       00%     26279         -
#17  Extended offline    Completed without error       00%     26112         -
#18  Extended offline    Completed without error       00%     25943         -
#19  Extended offline    Completed without error       00%     25775         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
Device State:                        Active (0)
Current Temperature:                    36 Celsius
Power Cycle Min/Max Temperature:     29/39 Celsius
Lifetime    Min/Max Temperature:     19/58 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -40/70 Celsius
Temperature History Size (Index):    128 (80)

Index    Estimated Time   Temperature Celsius
  81    2024-12-06 17:37    37  ******************
 ...    ..( 59 skipped).    ..  ******************
  13    2024-12-06 18:37    37  ******************
  14    2024-12-06 18:38    36  *****************
  15    2024-12-06 18:39    37  ******************
  16    2024-12-06 18:40    37  ******************
  17    2024-12-06 18:41    37  ******************
  18    2024-12-06 18:42    36  *****************
  19    2024-12-06 18:43    36  *****************
  20    2024-12-06 18:44    37  ******************
  21    2024-12-06 18:45    36  *****************
 ...    ..(  7 skipped).    ..  *****************
  29    2024-12-06 18:53    36  *****************
  30    2024-12-06 18:54    37  ******************
  31    2024-12-06 18:55    37  ******************
  32    2024-12-06 18:56    36  *****************
 ...    ..( 46 skipped).    ..  *****************
  79    2024-12-06 19:43    36  *****************
  80    2024-12-06 19:44    37  ******************

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 1) ==
0x01  0x008  4             142  ---  Lifetime Power-On Resets
0x01  0x010  4           30940  ---  Power-on Hours
0x01  0x018  6    798615768880  ---  Logical Sectors Written
0x01  0x020  6      1912345285  ---  Number of Write Commands
0x01  0x028  6   1208495279193  ---  Logical Sectors Read
0x01  0x030  6     11892771804  ---  Number of Read Commands
0x01  0x038  6    111386243350  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           30501  ---  Spindle Motor Power-on Hours
0x03  0x010  4           30501  ---  Head Flying Hours
0x03  0x018  4            1414  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              36  ---  Current Temperature
0x05  0x010  1              36  N--  Average Short Term Temperature
0x05  0x018  1              38  N--  Average Long Term Temperature
0x05  0x020  1              58  ---  Highest Temperature
0x05  0x028  1              19  ---  Lowest Temperature
0x05  0x030  1              53  N--  Highest Average Short Term Temperature
0x05  0x038  1              24  N--  Lowest Average Short Term Temperature
0x05  0x040  1              48  N--  Highest Average Long Term Temperature
0x05  0x048  1              25  N--  Lowest Average Long Term Temperature
0x05  0x050  4               0  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4               8  ---  Number of Hardware Resets
0x06  0x010  4               2  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
0xff  =====  =               =  ===  == Vendor Specific Statistics (rev 1) ==
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c)
No Defects Logged

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            0  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            1  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS
5

Technically ARC should fill until ~90% of free memory is in use & then it should automagically lower itself if necessary. I kinda find that I have to manually set limits on TrueNAS Scale though because I don’t feel like it is working as flawlessly as it did on TrueNAS Core.

echo  **SETLIMITINBYTES**>> /sys/module/zfs/parameters/zfs_arc_max

Try setting arc to uhh… 295279001600 should be 275gb? echo 295279001600 >> /sys/module/zfs/parameters/zfs_arc_max See if that offers any improvement?

Headsup, this command will NOT survive reboots. I’ve almost noticed that launching or shutting down a VM also resets it back to default.

If getting an extra 50gb of RAM free for filetransfers doesn’t do the needful, I’m not sure if it is RAM relevant - though I guess you can play around with manually setting ARC limits. This would impact ARC readspeeds, but at 328gb worth of total RAM, I think you have some wiggle room to experiment a touch.

6

Shingled drive spotted, potential contributing factor here.

Do you have another ST8000VN004 or WD/HGST drive you could replace it with?

2 Likes
7

Shingled drive spotted

Thanks. I think you might have found the underling issue. I’ll swap the drive and let you know…
That is exactly why I posted here… I figured someone would come in with fresh insight and spot something obvious that I missed.

I thought I had been careful not to buy any SMR drives, and it didn’t even occur to me that one had slipped through. How frustrating…

8

As I said before this is normal. Cached memory includes memory put aside for writes, and free memory reducing is NOT any indication of a problem. From the perspective of the operating system, the non-write queue part of cached memory is also free. If you wanted to start a 64GB virtual machine, then the O/S would request ZFS to free up 64GB of cache space so this could happen, and whereas swapping would mean writing that 64GB out to the swap file in order to free it, since this is cache and can simply be freed up.

On a completely separate topic, I see that you have L2ARC - do you have any stats showing that this is beneficial or is it simply reducing the amount of memory you have for ARC?

9

So one thing to try is NOT to tweak anything. Remove all the old autotune settings from the previous versions, do not manually set anything - this assumes you are on ElectricEel-24.10.0.2. SMB was a big laggy before this where one system could make things slow to a crawl on my TrueNAS XL+ over SMB. I removed all the pre-ElectricEel release hand configured / autotune settings and things got much better.

My system has 64GB RAM, 8x14TB drives mirror/striped and 400GB cache (ssd) and 14G log (ssd).