This is a follow up of NAS-129942.
In a short summary; I am unable to copy and upload data to my Hetzner Storage Box. The “Storage Box” is a simple SSH / Samba / WebDAV server where I’d like to upload data to.
Here is my system context:
Full Gigabit network connection to the storage box (50GB upload runs with avg 75MB/s)
I use PiHole for local DNS resolution. All clients use the the same DNS server.
Uploads via FTP from several different clients (AMD laptop, Intel desktop, ARM SOC) can reach speeds up to Gigabit when uploading via FTP to the Storagebox.
Because of these reasons, I have mostly ruled out Hetzner’s infrastructure as the problem. Hetzner’s support also confirmed upon my request that they do not see any performance issues with my instance (which I can confirm).
I have experimented with SSH and WebDAV in TrueNAS to upload data - both fail to backup my data.
SSH:
SSH is very fast initially and seems to upload the data correctly.
It starts with a speed of approx. 100MB/s but slows down to <1MB/s.
I upload about 15TB; hence the upload never completed with 1MB/s.
No errors or warnings visible in TrueNAS and the job log.
WebDAV:
Slower than SSH, also slows down to 0B/s or a few 100Kb/s.
No errors or warnings visible.
I have also performed tests using the rsync uploads. Since it does not allow to reuse the Cloud Backup connections, I set up SSH using private & public key. I can confirm that manually connecting via port 23 and the CLI works. But with the exact same parameters in the rsync task, it cannot start the upload.
As stated in the Jira ticket, I have now run out of options and next steps. I have upgraded to TrueNAS-24.10.0 (EE). Support stated that “We have reason to believe your issue has been fixed in the latest stable version available. Please upgrade and report back if the problem persists.”.
Old posts lead me to believe that this is an older problem / or the error message is unclear and is leading to confusion.
This is the thread I opened, which is also about that “403 Forbidden” errors, which I do receive when trying to copy with WebDAV. When trying to copy with SFTP I receive the error mentioned in that thread ( Failed to copy: Update Create failed: sftp: "Invalid filename" (SSH_FX_FAILURE)).
When disabling encryption, everything is fine, but this is actually not an option…
“Please post better details” → I’m well aware that the details are not sufficient. I took multiple screenshots carefully and expected to be able to upload them. As of today, I still cannot upload or link anything - so I’m unable to provide any further graphic information. I understand that this is to prevent spam and obscene content - but not being able to ask for help and providing basic information seems rather extreme. I’ll gadly post detailed screenshots as soon as I’m able to. For the time being, I attached more info as text below.
Thanks for the suggestions @AMiGAmann . I got a response from Hetzner in the mean time. They stated that “we do see in the log we think that the problem is the long file name”. They added “The length of the whole file path should be ok as the limit is 4 KiB (4096 chars) but the limit for a single file/directory name is only at 255 byte (255 chars) but here it was tried to create an directory with 256 chars.”. Disabling “Filename Encryption” did solve my uploads afterwards. However the slow upload speed (25MB/s) still remain for WebDAV. It still failed too because it failed to follow symlinks. As an example: “apps/ghost/data/themes/source: Listing error: symlink: stat /mnt/ssd-pool/apps/ghost/data/themes/source: no such file or directory”.
Disabling Encryption is not acceptable for me either for personal reasons. I store confidential data that I do not want to leak in case of unauthorized access to my Storage Box.
The pool in question uses high speed SDDs that should easily be capable of saturating Gigabit. I can approx. 1Gbps using iperf3 from my clients (tested from Dream Machine and personal PC).
System Data: OS Version: TrueNAS-SCALE-24.10.0 Product: B760 GAMING X DDR4 Model: 12th Gen Intel(R) Core™ i3-12100 Memory: 63 GiB System Serial: Default string
Pool being backed up:
1 x MIRROR | 2 wide | 931.51 GiB
WD_BLACK SN850X 1000GB
PCI devices
00:00.0 Host bridge: Intel Corporation Device 4630 (rev 05)
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-S GT1 [UHD Graphics 730] (rev 0c)
00:06.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 (rev 05)
00:14.0 USB controller: Intel Corporation Raptor Lake USB 3.2 Gen 2x2 (20 Gb/s) XHCI Host Controller (rev 11)
00:14.2 RAM memory: Intel Corporation Raptor Lake-S PCH Shared SRAM (rev 11)
00:15.0 Serial bus controller: Intel Corporation Raptor Lake Serial IO I2C Host Controller (rev 11)
00:15.1 Serial bus controller: Intel Corporation Raptor Lake Serial IO I2C Host Controller (rev 11)
00:15.2 Serial bus controller: Intel Corporation Raptor Lake Serial IO I2C Host Controller (rev 11)
00:15.3 Serial bus controller: Intel Corporation Device 7a4f (rev 11)
00:16.0 Communication controller: Intel Corporation Raptor Lake CSME HECI (rev 11)
00:17.0 SATA controller: Intel Corporation Raptor Lake SATA AHCI Controller (rev 11)
00:19.0 Serial bus controller: Intel Corporation Device 7a7c (rev 11)
00:19.1 Serial bus controller: Intel Corporation Device 7a7d (rev 11)
00:1a.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port (rev 11)
00:1c.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port (rev 11)
00:1c.1 PCI bridge: Intel Corporation Device 7a39 (rev 11)
00:1c.2 PCI bridge: Intel Corporation Device 7a3a (rev 11)
00:1c.3 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port (rev 11)
00:1c.4 PCI bridge: Intel Corporation Device 7a3c (rev 11)
00:1f.0 ISA bridge: Intel Corporation Device 7a06 (rev 11)
00:1f.4 SMBus: Intel Corporation Raptor Lake-S PCH SMBus Controller (rev 11)
00:1f.5 Serial bus controller: Intel Corporation Raptor Lake SPI (flash) Controller (rev 11)
01:00.0 Non-Volatile memory controller: Sandisk Corp Western Digital WD Black SN850X NVMe SSD (rev 01)
02:00.0 Non-Volatile memory controller: Sandisk Corp Western Digital WD Black SN850X NVMe SSD (rev 01)
04:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
06:00.0 SATA controller: ASMedia Technology Inc. 106x SATA/RAID Controller (rev 01)
07:00.0 Non-Volatile memory controller: Micron/Crucial Technology P2 NVMe PCIe SSD (rev 01)
CPU:
Vendor ID: GenuineIntel
Model name: 12th Gen Intel(R) Core(TM) i3-12100
CPU family: 6
Model: 151
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
Stepping: 5
CPU(s) scaling MHz: 19%
CPU max MHz: 4300.0000
CPU min MHz: 800.0000
Memory:
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online yes 0
0x0000000100000000-0x00000010ffffffff 64G online yes 2-33
(NAS-132472)
