Low performance migrating to Truenas

First and more important, thanks for maintaining the project and supporting the forums.

Context:
I have an old HP Gen8 with a Xeon E3-1265L V2, 16GB DDR3 ECC, 2xSPF+, 256GB SD for OS and 4x3TB WD Red for storage. The system was running with a Gentoo + ZFS with raidz1 (encrypted). After 8 years my SSD started to report some errors so decide to buy a new one, and at the same time, replace my 3TB WD Red for 8TB WD red plus (CRM). I did pass all the tests in Truenas to ensure the HD are ok.

As I didnt want to go again and rebuild from scratch the whole Gentoo I decided to try to TrueNas (Scale) as it has what I need and much easier to deploy.

Problem:
I used to get around 150 mb/s speed approx using NFS/SMB/SSH (it used to be a 20 mb/s diff between them) in transfers between the NAS and any of my servers using the older 3TB HDs. After Truenas migration I am not getting more than 40/50 mb/s.

What I have tested:

  1. Ensure it is not a network problem. Everything is wired with LC using SPF+ (10 and 28) and tested with iperf getting the 10Gbit and 20 Gbit/s
  2. I have connected the old 4x3TB HD to TrueNas, but the pool is not recognized as I encrypted the pool with Luks so I can really not test it.
  3. I have boot the NAS using a linux USB, created a raid 5 using mdadm for the 4x8TB, format with EXT4 and the transfer goes 180/190 mb/s.

At this point, I am assuming is something wrong in my set up with Truenas, something my fat finger has selected and I am not able to identify now.

Linux 6.12.15-production+truenas (truenas) 	06/01/25 	_x86_64_	(8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.61    0.11    1.50   35.72    0.00   62.07

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda              67.76     12979.57        28.85         0.00   20748880      46120          0
sdb             129.13     12701.75        28.88         0.00   20304756      46168          0
sdc              70.90         9.10     13042.87         0.00      14552   20850068          0
sdd             121.65     12976.76        28.98         0.00   20744392      46328          0
sde              11.85       183.87        48.80         0.00     293926      78004          0
              capacity     operations     bandwidth 
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
boot-pool   2.82G   219G      7      3   163K  48.8K
raid         427G  28.7T    239     46  38.8M  13.2M
----------  -----  -----  -----  -----  -----  -----
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
raid  29.1T   427G  28.7T        -         -     0%     1%  1.00x    ONLINE  /mnt
NAME  PROPERTY              VALUE                  SOURCE
raid  type                  filesystem             -
raid  creation              Sat May 31 14:20 2025  -
raid  used                  311G                   -
raid  available             20.7T                  -
raid  referenced            309G                   -
raid  compressratio         1.02x                  -
raid  mounted               yes                    -
raid  quota                 none                   default
raid  reservation           none                   default
raid  recordsize            128K                   default
raid  mountpoint            /mnt/raid              default
raid  sharenfs              off                    default
raid  checksum              on                     default
raid  compression           lz4                    local
raid  atime                 off                    local
raid  devices               on                     default
raid  exec                  on                     default
raid  setuid                on                     default
raid  readonly              off                    default
raid  zoned                 off                    default
raid  snapdir               hidden                 default
raid  aclmode               discard                local
raid  aclinherit            discard                local
raid  createtxg             1                      -
raid  canmount              on                     default
raid  xattr                 on                     local
raid  copies                1                      default
raid  version               5                      -
raid  utf8only              off                    -
raid  normalization         none                   -
raid  casesensitivity       sensitive              -
raid  vscan                 off                    default
raid  nbmand                off                    default
raid  sharesmb              off                    default
raid  refquota              none                   default
raid  refreservation        none                   default
raid  guid                  15405334406286279910   -
raid  primarycache          all                    default
raid  secondarycache        all                    default
raid  usedbysnapshots       0B                     -
raid  usedbydataset         309G                   -
raid  usedbychildren        1.80G                  -
raid  usedbyrefreservation  0B                     -
raid  logbias               latency                default
raid  objsetid              54                     -
raid  dedup                 off                    default
raid  mlslabel              none                   default
raid  sync                  standard               default
raid  dnodesize             legacy                 default
raid  refcompressratio      1.02x                  -
raid  written               309G                   -
raid  logicalused           317G                   -
raid  logicalreferenced     315G                   -
raid  volmode               default                default
raid  filesystem_limit      none                   default
raid  snapshot_limit        none                   default
raid  filesystem_count      none                   default
raid  snapshot_count        none                   default
raid  snapdev               hidden                 default
raid  acltype               posix                  local
raid  context               none                   default
raid  fscontext             none                   default
raid  defcontext            none                   default
raid  rootcontext           none                   default
raid  relatime              on                     default
raid  redundant_metadata    all                    default
raid  overlay               on                     default
raid  encryption            aes-256-gcm            -
raid  keylocation           prompt                 local
raid  keyformat             hex                    -
raid  pbkdf2iters           0                      default
raid  encryptionroot        raid                   -
raid  keystatus             available              -
raid  special_small_blocks  0                      default
raid  prefetch              all                    default
raid  direct                standard               default
raid  longname              off                    default

After trying and trying with different solutions, installing truenas in USB and using SSD for slog, ARC2, etc the issue was much more simple. For unknown reasons the cache write of my hard disk were set to 0 in Truenas whereas in Gentoo and Debian I was enabled by default, not sure the reason of this.

The raid moved from 40mb/s, and dropping to 5 mb/s after few minutes, to achieve 250 mb/s without issues via SCP with Rsync (already copied 1 TBs of data and no issues)

Command:

hdparm -W 1 /dev/sdX

Thanks!

2 Likes

Great catch and its an interesting observation.

If anyone notices something similar, let us know so that we can review the defaults . Sometime the defaults are set to mitigate an historical 3rd party vendor issue.