Performance issues

TrueNAS 13.0-U6.1
For some time now, i have been experiencing significant performance drops with my TrueNAS. The system is used as a backup repository. Initially, i had transfer rates of 800-1000 MB/s from Veeam. Currently, i am only achieving a maximum of 25 MB/s. Additionally, the pool status is “Online (unhealthy),” but status checks show no corrupt files or read/write errors. Notably, i am seeing steadily increasing checksum errors on one drive. I tried various things to resolve the issue. Among other things, the drive has been replaced, the drive has been installed in another slot, and all contacts have been checked. During the replacement and resilvering, no anomalies occurred. Only after the process did the same error reappear. The pool is currently 68% utilized and configured in 6 RaidZ1 each 5x 12TB Disks and 2 NVME in Mirror VDev config.
Can you explain further test options to help me narrow down the problem ?

Can you provide some details on hardware?


its a Supermicro chassis running an AMD EPYC 7302 with 640GB of RAM, two 25G nics for ISCSI + 1g for management, the disks are mainly HGST HUH72121/12TB

That. And then

Keep it below 50% for block storage.