2MB/sec SMB transfer speed to TrueNAS Scale Storage

Johnny_Fartpants · July 13, 2024, 5:17pm

Try sudo -s then enter and enter your admin password then try again.

robnijhof · July 13, 2024, 5:34pm

Thanks! That works.
Here is the output of one of the WD disks:

Summary

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Green
Device Model: WDC WD20EARX-00PASB0
Serial Number: WD-WCAZAF139425
LU WWN Device Id: 5 0014ee 20720ea35
Firmware Version: 51.0AB51
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database 7.3/5528
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Jul 13 19:32:53 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (37500) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 361) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 172 170 021 Pre-fail Always - 6400
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 197
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 001 001 000 Old_age Always - 94539
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 191
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 83
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 117
194 Temperature_Celsius 0x0022 118 100 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 28844 -

2 Short offline Completed without error 00% 28841 -

3 Short offline Completed without error 00% 28709 -

4 Short offline Completed without error 00% 28541 -

5 Short offline Completed without error 00% 15453 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

And here is the output of a Seagate drive:

Summary

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1CH166
Serial Number: Z1F2NB4E
LU WWN Device Id: 5 000c50 050301a33
Firmware Version: CC44
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Jul 13 19:34:41 2024 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 592) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 339) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.

SMART Attributes Vendor Specific ID# ATTRIBUTE_NAME 1 Raw_Read_Error_Rate 3 Spin_Up_Time 4 Start_Stop_Count 5 Reallocated_Sector_Ct 7 Seek_Error_Rate 9 Power_On_Hours 10 Spin_Retry_Count 12 Power_Cycle_Count 183 Runtime_Bad_Block 184 End-to-End_Error 187 Reported_Uncorrect 188 Command_Timeout 189 High_Fly_Writes 190 Airflow_Temperature_Cel 191 G-Sense_Error_Rate 192 Power-Off_Retract_Count 193 Load_Cycle_Count 194 Temperature_Celsius 197 Current_Pending_Sector 198 Offline_Uncorrectable 199 UDMA_CRC_Error_Count 240 Head_Flying_Hours 241 Total_LBAs_Written 242 Total_LBAs_Read Data Structure revision number: 10
SMART Attributes with Thresholds:
FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
0x000f 118 099 006 Pre-fail Always - 182095872
0x0003 091 091 000 Pre-fail Always - 0
0x0032 100 100 020 Old_age Always - 198
0x0033 100 100 010 Pre-fail Always - 0
0x000f 056 054 030 Pre-fail Always - 1189816967728
0x0032 043 043 000 Old_age Always - 50549
0x0013 100 100 097 Pre-fail Always - 0
0x0032 100 100 020 Old_age Always - 123
0x0032 100 100 000 Old_age Always - 0
0x0032 100 100 099 Old_age Always - 0
0x0032 100 100 000 Old_age Always - 0
0x0032 100 097 000 Old_age Always - 0 1 121
0x003a 096 096 000 Old_age Always - 4
0x0022 062 043 045 Old_age Always In_the_past 38 (0 43 38 35 0)
0x0032 100 100 000 Old_age Always - 0
0x0032 100 100 000 Old_age Always - 87
0x0032 100 100 000 Old_age Always - 1024
0x0022 038 057 000 Old_age Always - 38 (0 15 0 0 0)
0x0012 100 100 000 Old_age Always - 0
0x0010 100 100 000 Old_age Offline - 0
0x003e 200 200 000 Old_age Always - 0
0x0000 100 253 000 Old_age Offline - 49192h+31m+51.454s
0x0000 100 253 000 Old_age Offline - 17302167486
0x0000 100 253 000 Old_age Offline - 28937721516

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error

1 Short offline Completed without error 00% 50390 -

2 Short offline Completed without error 00% 50387 -

3 Short offline Completed without error 00% 50255 -

4 Short offline Completed without error 00% 50087 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

neofusion · July 13, 2024, 6:13pm

Apologies, I forgot that not everyone uses a normal shell as their shell. You can, if you want, change it to a basic shell in Credentials → Local Users and then edit the user of choice. All the ones ending with “sh” are shells.

The reason it couldn’t find sudo is because the binaries aren’t in the shell PATH’s, something I view as an oversight. You could fix that yourself either by changing your shell config/profile file or work around it by using the absolute path to the command you want to run.

neofusion · July 13, 2024, 6:15pm

Could you post the output of the full two other disks as well?
The one you posted an image of earlier wasn’t in the output you just posted.

Johnny_Fartpants · July 13, 2024, 6:25pm

and perhaps zpool status while you’re at it

robnijhof · July 14, 2024, 1:46pm

Sure,

Disk 3:

Summary

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 105 074 006 Pre-fail Always - 7453744
3 Spin_Up_Time 0x0003 091 091 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 253
5 Reallocated_Sector_Ct 0x0033 100 099 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 060 052 030 Pre-fail Always - 1452094109357
9 Power_On_Hours 0x0032 014 014 000 Old_age Always - 76122
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 252
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 63405
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 6 7 9
189 High_Fly_Writes 0x003a 096 096 000 Old_age Always - 4
190 Airflow_Temperature_Cel 0x0022 061 032 045 Old_age Always In_the_past 39 (6 96 39 35 0)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 69
193 Load_Cycle_Count 0x0032 089 089 000 Old_age Always - 22392
194 Temperature_Celsius 0x0022 039 068 000 Old_age Always - 39 (0 14 0 0 0)
197 Current_Pending_Sector 0x0012 100 074 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 074 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 51483h+51m+58.891s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 85638898724
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 121491805173

Disk 4:

Summary

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 2
3 Spin_Up_Time 0x0027 172 168 021 Pre-fail Always - 6391
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 192
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 001 001 000 Old_age Always - 94552
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 186
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 79
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 405
194 Temperature_Celsius 0x0022 115 100 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1

robnijhof · July 14, 2024, 1:48pm

Here it is:

Summary

pool: APPS
state: ONLINE
config:

    NAME                                    STATE     READ WRITE CKSUM
    APPS                                    ONLINE       0     0     0
      f5520d42-7b42-45e7-9b04-13e1fe8d71a3  ONLINE       0     0     0

errors: No known data errors

pool: STORAGE
state: ONLINE
scan: scrub repaired 0B in 02:03:55 with 0 errors on Sun Jul 14 02:03:57 2024
config:

    NAME                                      STATE     READ WRITE CKSUM
    STORAGE                                   ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        4a8d46dd-16dd-4cf3-a192-a6ed4f72c9fb  ONLINE       0     0     0
        df94f340-0527-402b-ad66-de7d7b209666  ONLINE       0     0     0
        fcf06a14-9944-4f35-85b9-18a880a4f624  ONLINE       0     0     0
        a6ec2541-7805-4f74-ab3b-128951e54798  ONLINE       0     0     0

errors: No known data errors

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:00:15 with 0 errors on Sat Jul 13 03:45:17 2024
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      sdf3      ONLINE       0     0     0

errors: No known data errors

Johnny_Fartpants · July 14, 2024, 2:08pm

Can you show us the dataset properties from the UI want to check you haven’t turned sync on or dedupe.

robnijhof · July 14, 2024, 2:24pm

prez02 · July 14, 2024, 2:37pm

Why today? Did you change anything? Did you set it up new?

Might not be related, but this has a hp 120i soft raid “controller”. I have been running it in SATA mode, which makes booting from the 5th sata port a bit of a hassle. It has worked without any problems over the years with Freenas, Core and Scale.

Some peope use the Intelligent Provisioning feature, which each disk being a single stripe, iirc. Generally using full raid controllers with zfs is recommended strongly against, but I do not think this is related to your problems, especially as it is not a “real” raid controller.

I can get some old disk from work, and use them to test things on Scale. My experience is, if the disk are that old, at some point they fail. Some have no smart errors, but produce zfs errors consistently, independed of cable etc. The same model of disk in the same “spot” works fine.

robnijhof · July 14, 2024, 2:41pm

No, after installation I did not try copying files to it, until a couple days ago.

The onboard SATA controller is in AHCI mode, so no RAID configuration from the BIOS.

I have a separate SATA card installed with the boot drive (Crucial SSD) and a separate Samsung SSD is installed for the apps. So I’m not using the CD-ROM port at all.

prez02 · July 14, 2024, 3:10pm

Okay, so a new install, basically, not something failing that did work before?!

Just wrote a few larger files to my Gen8, no drop in bandwith.

What type/model of SATA card are you using? Always thought of adding one, but there is not much space in the case.

neofusion · July 14, 2024, 3:11pm

So drive 4 is the one from your earlier image, the one showing clear errors, but you still haven’t posted the full smartctl output. Without more details I will fall back on generic suggestions.

Run a long smart test. Ideally this should be setup to run regularly automatically, weekly or bi-weekly is a common recommendation.

Test your drives IO performance locally on the server with fio to see if they perform any better outside of SMB over the network. Make sure you read up on how to use it as incorrect fio usage can destroy the pool.

Good luck.

robnijhof · July 14, 2024, 3:54pm

Thank you. I started a manual SMART test. It’ll take some time, but I will report back the results.

robnijhof · July 16, 2024, 6:29pm

Hi neofusion,

I did long SMART tests and all came out with SUCCESS. I think it’s safe to say that the disks are not the culprit.
Any other ideas why my transfer rate to my STORAGE pool is so slow?

neofusion · July 16, 2024, 7:44pm

No, sorry.

somethingweird · July 16, 2024, 7:46pm

Have you tried not running any of the apps - my gut feeling it’s a low memory issue. (I could be wrong.)

HoneyBadger · July 18, 2024, 3:45am

Riddle me this, from the same root command prompt:

 for disk in /dev/sd?; do; hdparm -W $disk; done

What you should see is a number of outputs similar to:

/dev/sda:
 write-caching =  1 (on)

/dev/sdb:
 write-caching =  1 (on)

/dev/sdc:
 write-caching =  1 (on)

/dev/sdd:
 write-caching =  1 (on)

If, on the other hand, you see any drives with it off - then that’s potentially our problem.

robnijhof · July 23, 2024, 6:47pm

Thank you, here is the output:

Write caching results

/dev/sda:
write-caching = 0 (off)

/dev/sdb:
write-caching = 0 (off)

/dev/sdc:
write-caching = 0 (off)

/dev/sdd:
write-caching = 0 (off)

/dev/sde:
write-caching = not supported

/dev/sdf:
write-caching = 1 (on)

/dev/sdg:
write-caching = 1 (on)

The first 4 disks are indeed my DATA storage…

HoneyBadger · July 23, 2024, 7:13pm

Let’s do:

hdparm -W1 /dev/sda
hdparm -W1 /dev/sdb
hdparm -W1 /dev/sdc
hdparm -W1 /dev/sdd

And then re-run the for loop and check to make sure the setting stuck. If it did, try a big transfer again.