TRIM not enabling on supported SSD's (Scale-ElectricEel-24.10.2.2)

Hello guys!

I’m building and testing a new storage by using an old server with new SSD’s with trim capability but trim is unsupported on “Scale-ElectricEel-24.10.2.2” and I couldn’t fix it. I hope I can find someone experienced on this with a solution.

Let me tell you my hardwares:
Server: Supermicro 4U-X10DRH-iT-72BS-SAS3-2PS
HBA: AOC-S3008L-L8e

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------
0  SAS3008(C0)  16.00.14.00    0e.01.30.28    08.37.00.00     00:02:00:00

SSD: 40x Crucial CT4000MX500SSD1 (I hate it, do not buy EVER)

=== START OF INFORMATION SECTION ===
Model Family:     Crucial/Micron Client SSDs
Device Model:     CT4000MX500SSD1
Serial Number:    
LU WWN Device Id: 5 00a075 1e6ba7076
Firmware Version: M3CR045
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available
Device is:        In smartctl database 7.3/5528
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)

Lets check the SSD trim support:

hdparm -I /dev/sdd | grep -i trim
 *    Data Set Management TRIM supported (limit 8 blocks)

Lets check the SSD trim support in the pool:

zpool status -LP -t | grep sdd
            /dev/sdd1     ONLINE       0     0     0  (trim unsupported)

Lets check the Truenas (Scale-ElectricEel-24.10.2.2) mpt3sas module:

modinfo mpt3sas

filename:       /lib/modules/6.6.44-production+truenas/kernel/drivers/scsi/mpt3sas/mpt3sas.ko
alias:          mpt2sas
version:        43.100.00.00
license:        GPL
description:    LSI MPT Fusion SAS 3.0 Device Driver
author:         Avago Technologies <MPT-FusionLinux.pdl@avagotech.com>
srcversion:     E46E6B71C385F485C977A1B
depends:        scsi_mod,scsi_transport_sas,raid_class,scsi_common
retpoline:      Y
intree:         Y
name:           mpt3sas
vermagic:       6.6.44-production+truenas SMP preempt mod_unload modversions 
parm:           logging_level: bits for enabling additional logging info (default=0)
parm:           max_sectors:max sectors, range 64 to 32767  default=32767 (ushort)
parm:           missing_delay: device missing delay , io missing delay (array of int)
parm:           max_lun: max lun, default=16895  (ullong)
parm:           hbas_to_enumerate: 0 - enumerates both SAS 2.0 & SAS 3.0 generation HBAs
                  1 - enumerates only SAS 2.0 generation HBAs
                  2 - enumerates only SAS 3.0 generation HBAs (default=0) (ushort)
parm:           diag_buffer_enable: post diag buffers (TRACE=1/SNAPSHOT=2/EXTENDED=4/default=0) (int)
parm:           disable_discovery: disable discovery  (int)
parm:           prot_mask: host protection capabilities mask, def=7  (int)
parm:           enable_sdev_max_qd:Enable sdev max qd as can_queue, def=disabled(0) (bool)
parm:           multipath_on_hba:Multipath support to add same target device
                as many times as it is visible to HBA from various paths
                (by default:
                         SAS 2.0 & SAS 3.0 HBA - This will be disabled,
                         SAS 3.5 HBA - This will be enabled) (int)
parm:           host_tagset_enable:Shared host tagset enable/disable Default: enable(1) (int)
parm:           max_queue_depth: max controller queue depth  (int)
parm:           max_sgl_entries: max sg entries  (int)
parm:           msix_disable: disable msix routed interrupts (default=0) (int)
parm:           smp_affinity_enable:SMP affinity feature enable/disable Default: enable(1) (int)
parm:           max_msix_vectors: max msix vectors (int)
parm:           irqpoll_weight:irq poll weight (default= one fourth of HBA queue depth) (int)
parm:           mpt3sas_fwfault_debug: enable detection of firmware fault and halt firmware - (default=0)
parm:           perf_mode:Performance mode (only for Aero/Sea Generation), options:
                0 - balanced: high iops mode is enabled &
                interrupt coalescing is enabled only on high iops queues,
                1 - iops: high iops mode is disabled &
                interrupt coalescing is enabled on all queues,
                2 - latency: high iops mode is disabled &
                interrupt coalescing is enabled on all queues with timeout value 0xA,
                default - default perf_mode is 'balanced' (int)
parm:           poll_queues:Number of queues to be use for io_uring poll mode.
                This parameter is effective only if host_tagset_enable=1. &
                when poll_queues are enabled then &
                perf_mode is set to latency mode. &
                 (int)

Lets check the kernel assigned “DISC-GRAN, DISC-MAX, MIN-IO”:

lsblk -SDt
NAME HCTL       TYPE VENDOR   MODEL               REV SERIAL          TRAN   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED       RQ-SIZE  RA WSAME
sdd  0:0:4:0    disk ATA      CT4000MX500SSD1    046  2312E6BD96A0    sas           0        0B       0B         0         0   4096      0    4096     512    0 mq-deadline     256 128    0B
sdao 6:0:0:0    disk ATA      Samsung SSD 870 EV 1B6Q S6PTNZ0RB03121J sata          0      512B       2G         0         0    512      0     512     512    0 mq-deadline      64 128    0B

As you can see above the boot pool Samsung drive plugged on-board sata controller (Intel C610/X99 SATA Controller) and it has DISC-GRAN=512B DISC-MAX=2G values assigned and TRIM supported.
/dev/sdap3 ONLINE 0 0 0 (untrimmed)

I use the same drive on a different server with ubuntu 22.04 drive plugged on-board sata controller (Intel C610/X99 SATA Controller) and it is working.

NAME HCTL       TYPE VENDOR   MODEL            REV SERIAL          TRAN   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED       RQ-SIZE  RA WSAME
sdc  2:0:0:0    disk ATA      CT4000MX500SSD1 046  2312E6BD97FD    sata          0        4K       2G         0         0   4096      0    4096     512    0 mq-deadline      64 128    0B

Because of this I believe the issue related to my HBA card “AOC-S3008L-L8e”.

BUT!!!

On a different truenas storage I have identical HBA card but Samsung drives and everything works with “TrueNAS-SCALE-23.10.2”

lspci -v | grep -i lsi -A 1
82:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
        Subsystem: Super Micro Computer Inc AOC-S3008L-L8e
lsblk -SDt
NAME HCTL       TYPE VENDOR   MODEL                     REV SERIAL          TRAN   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED       RQ-SIZE  RA WSAME
sda  0:0:1:0    disk ATA      Samsung SSD 870 EVO 4TB  1B6Q S6BBNJ0RA02825A sas           0      512B     128M         0         0    512      0     512     512    0 mq-deadline     256 128    0B
sdf  11:0:0:0   disk ATA      SAMSUNG MZ7WD480HCGM-000 203Q S1G1NYAF503106  sata          0      512B       2G         0         0    512      0     512     512    0 mq-deadline      64 128    0B
./sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02) 
Copyright (c) 2009-2018 Avago Technologies. All rights reserved. 

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.37.00.00
  Firmware version                        : 16.00.14.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 255
  Concurrent commands supported           : 3072
  Slot                                    : 3
  Segment                                 : 0
  Bus                                     : 130
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 0
  SAS Address                             : 5003048-0-1f73-ef80
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 3815447/7814037167
  Manufacturer                            : ATA     
  Model Number                            : Samsung SSD 870 
  Firmware Revision                       : 1B6Q
  Serial No                               : S6BBNJ0RA02825A
  Unit Serial No(VPD)                     : S6BBNJ0RA02825A
  GUID                                    : 5002538f31a159c8
  Protocol                                : SATA
  Drive Type                              : SATA_SSD

After all of my research I started to think there should be a missmatch between LSI-3008 firmware/driver ↔ Crucial CT4000MX500SSD1 firmware.

And I have only one last finding is: TRIM and SG_UNMAP support for LSI HBAs and RAID controllers

So please tell me… Which part is the root cause and how can I solve it?

  • Thanks

I guess I find the root cause:

Disk Capabilities

  • Samsung SSD 860 (sda/sdb)
    • Support TRIM.
    • Support Deterministic Read Zero after TRIM (DRZAT) → after trimming, reading the block always returns 0. This is ideal for ZFS autotrim because it guarantees consistent reads.
  • Crucial MX500 (sdc/sdd)
    • Support TRIM, but no DRZAT (only plain TRIM).
    • After TRIM, data read may return indeterminate patterns (not guaranteed zeros). ZFS still works with this, but it can sometimes cause checksum errors if firmware misbehaves.

AHCI onboard Intel SATA (works fine with TRIM)

  • Controller: 00:1f.2 Intel C610/X99 AHCI SATA
  • lsblk shows:
    • DISC-GRAN = 2G
    • DISC-MAX = 2G
    • TRIM (Data Set Management) reported by hdparm.
  • Behavior: Kernel exposes the discard capability to the OS, so autotrim=on or fstrim will actually send TRIM down to the SSD.
lsblk -SDt | grep "NAME\|sdan\|sdam\|sdap"
NAME HCTL       TYPE VENDOR   MODEL                    REV SERIAL          TRAN   DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED       RQ-SIZE  RA WSAME
sdam 0:0:39:0   disk ATA      CT4000MX500SSD1         046  2317E6CFCD88    sas           0        0B       0B         0         0   4096      0    4096     512    0 mq-deadline     256 128    0B
sdan 0:0:40:0   disk ATA      CT4000MX500SSD1         045  2311E6BA4D23    sas           0        0B       0B         0         0   4096      0    4096     512    0 mq-deadline     256 128    0B
sdap 3:0:0:0    disk ATA      Samsung SSD 870 EVO 1TB 1B6Q S6PTNZ0RB03184H sata          0      512B       2G         0         0    512      0     512     512    0 mq-deadline      64 128    0B

hdparm -I /dev/sdam | egrep -i 'Deterministic|read zeros|trim'
           *    Data Set Management TRIM supported (limit 8 blocks)

hdparm -I /dev/sdan | egrep -i 'Deterministic|read zeros|trim'
           *    Data Set Management TRIM supported (limit 8 blocks)

hdparm -I /dev/sdap | egrep -i 'Deterministic|read zeros|trim'
           *    Data Set Management TRIM supported (limit 8 blocks)
           *    Deterministic read ZEROs after TRIM

LSI SAS3008 HBA (TRIM not exposed)

  • Controller: 03:00.0 Broadcom / LSI SAS3008 (mpt3sas driver)
  • lsblk shows:
    • DISC-GRAN = 0B
    • DISC-MAX = 0B
  • Even though hdparm shows TRIM support at the drive level, the controller does not pass through TRIM/UNMAP to SATA devices behind it.
  • Broadcom confirmed this in the KB you linked: SAS3008 firmware + mpt3sas does not translate ATA TRIM → SCSI UNMAP.

Why

  • AHCI speaks ATA natively → TRIM works directly.
  • SAS HBAs (3008/3108) speak SCSI. SATA drives are tunneled (STP → SATA Tunneling Protocol).
  • TRIM is an ATA command, but SAS uses UNMAP.
  • Most LSI HBAs (SAS3008/3108, SAS2008, etc.) do not translate TRIM ↔ UNMAP, so the OS only sees “discard not supported.”

So the solution looks like adding " UNMAP" trim support to the “Crucial CT4000MX500SSD1” or TRIM support to LSI SAS3008 HBA card.

I wonder is it possible via firmware upgrade or is this change requires hardware change too.

Another solution is finding compatiple new HBA card for MX500 drives. Advises?