Hello guys!
I’m building and testing a new storage by using an old server with new SSD’s with trim capability but trim is unsupported on “Scale-ElectricEel-24.10.2.2” and I couldn’t fix it. I hope I can find someone experienced on this with a solution.
Let me tell you my hardwares:
Server: Supermicro 4U-X10DRH-iT-72BS-SAS3-2PS
HBA: AOC-S3008L-L8e
Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------
0 SAS3008(C0) 16.00.14.00 0e.01.30.28 08.37.00.00 00:02:00:00
SSD: 40x Crucial CT4000MX500SSD1 (I hate it, do not buy EVER)
=== START OF INFORMATION SECTION ===
Model Family: Crucial/Micron Client SSDs
Device Model: CT4000MX500SSD1
Serial Number:
LU WWN Device Id: 5 00a075 1e6ba7076
Firmware Version: M3CR045
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Lets check the SSD trim support:
hdparm -I /dev/sdd | grep -i trim
* Data Set Management TRIM supported (limit 8 blocks)
Lets check the SSD trim support in the pool:
zpool status -LP -t | grep sdd
/dev/sdd1 ONLINE 0 0 0 (trim unsupported)
Lets check the Truenas (Scale-ElectricEel-24.10.2.2) mpt3sas module:
modinfo mpt3sas
filename: /lib/modules/6.6.44-production+truenas/kernel/drivers/scsi/mpt3sas/mpt3sas.ko
alias: mpt2sas
version: 43.100.00.00
license: GPL
description: LSI MPT Fusion SAS 3.0 Device Driver
author: Avago Technologies <MPT-FusionLinux.pdl@avagotech.com>
srcversion: E46E6B71C385F485C977A1B
depends: scsi_mod,scsi_transport_sas,raid_class,scsi_common
retpoline: Y
intree: Y
name: mpt3sas
vermagic: 6.6.44-production+truenas SMP preempt mod_unload modversions
parm: logging_level: bits for enabling additional logging info (default=0)
parm: max_sectors:max sectors, range 64 to 32767 default=32767 (ushort)
parm: missing_delay: device missing delay , io missing delay (array of int)
parm: max_lun: max lun, default=16895 (ullong)
parm: hbas_to_enumerate: 0 - enumerates both SAS 2.0 & SAS 3.0 generation HBAs
1 - enumerates only SAS 2.0 generation HBAs
2 - enumerates only SAS 3.0 generation HBAs (default=0) (ushort)
parm: diag_buffer_enable: post diag buffers (TRACE=1/SNAPSHOT=2/EXTENDED=4/default=0) (int)
parm: disable_discovery: disable discovery (int)
parm: prot_mask: host protection capabilities mask, def=7 (int)
parm: enable_sdev_max_qd:Enable sdev max qd as can_queue, def=disabled(0) (bool)
parm: multipath_on_hba:Multipath support to add same target device
as many times as it is visible to HBA from various paths
(by default:
SAS 2.0 & SAS 3.0 HBA - This will be disabled,
SAS 3.5 HBA - This will be enabled) (int)
parm: host_tagset_enable:Shared host tagset enable/disable Default: enable(1) (int)
parm: max_queue_depth: max controller queue depth (int)
parm: max_sgl_entries: max sg entries (int)
parm: msix_disable: disable msix routed interrupts (default=0) (int)
parm: smp_affinity_enable:SMP affinity feature enable/disable Default: enable(1) (int)
parm: max_msix_vectors: max msix vectors (int)
parm: irqpoll_weight:irq poll weight (default= one fourth of HBA queue depth) (int)
parm: mpt3sas_fwfault_debug: enable detection of firmware fault and halt firmware - (default=0)
parm: perf_mode:Performance mode (only for Aero/Sea Generation), options:
0 - balanced: high iops mode is enabled &
interrupt coalescing is enabled only on high iops queues,
1 - iops: high iops mode is disabled &
interrupt coalescing is enabled on all queues,
2 - latency: high iops mode is disabled &
interrupt coalescing is enabled on all queues with timeout value 0xA,
default - default perf_mode is 'balanced' (int)
parm: poll_queues:Number of queues to be use for io_uring poll mode.
This parameter is effective only if host_tagset_enable=1. &
when poll_queues are enabled then &
perf_mode is set to latency mode. &
(int)
Lets check the kernel assigned “DISC-GRAN, DISC-MAX, MIN-IO”:
lsblk -SDt
NAME HCTL TYPE VENDOR MODEL REV SERIAL TRAN DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME
sdd 0:0:4:0 disk ATA CT4000MX500SSD1 046 2312E6BD96A0 sas 0 0B 0B 0 0 4096 0 4096 512 0 mq-deadline 256 128 0B
sdao 6:0:0:0 disk ATA Samsung SSD 870 EV 1B6Q S6PTNZ0RB03121J sata 0 512B 2G 0 0 512 0 512 512 0 mq-deadline 64 128 0B
As you can see above the boot pool Samsung drive plugged on-board sata controller (Intel C610/X99 SATA Controller) and it has DISC-GRAN=512B DISC-MAX=2G values assigned and TRIM supported.
/dev/sdap3 ONLINE 0 0 0 (untrimmed)
I use the same drive on a different server with ubuntu 22.04 drive plugged on-board sata controller (Intel C610/X99 SATA Controller) and it is working.
NAME HCTL TYPE VENDOR MODEL REV SERIAL TRAN DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME
sdc 2:0:0:0 disk ATA CT4000MX500SSD1 046 2312E6BD97FD sata 0 4K 2G 0 0 4096 0 4096 512 0 mq-deadline 64 128 0B
Because of this I believe the issue related to my HBA card “AOC-S3008L-L8e”.
BUT!!!
On a different truenas storage I have identical HBA card but Samsung drives and everything works with “TrueNAS-SCALE-23.10.2”
lspci -v | grep -i lsi -A 1
82:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Subsystem: Super Micro Computer Inc AOC-S3008L-L8e
lsblk -SDt
NAME HCTL TYPE VENDOR MODEL REV SERIAL TRAN DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME
sda 0:0:1:0 disk ATA Samsung SSD 870 EVO 4TB 1B6Q S6BBNJ0RA02825A sas 0 512B 128M 0 0 512 0 512 512 0 mq-deadline 256 128 0B
sdf 11:0:0:0 disk ATA SAMSUNG MZ7WD480HCGM-000 203Q S1G1NYAF503106 sata 0 512B 2G 0 0 512 0 512 512 0 mq-deadline 64 128 0B
./sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.
Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS3008
BIOS version : 8.37.00.00
Firmware version : 16.00.14.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 255
Concurrent commands supported : 3072
Slot : 3
Segment : 0
Bus : 130
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0
Device is a Hard disk
Enclosure # : 2
Slot # : 0
SAS Address : 5003048-0-1f73-ef80
State : Ready (RDY)
Size (in MB)/(in sectors) : 3815447/7814037167
Manufacturer : ATA
Model Number : Samsung SSD 870
Firmware Revision : 1B6Q
Serial No : S6BBNJ0RA02825A
Unit Serial No(VPD) : S6BBNJ0RA02825A
GUID : 5002538f31a159c8
Protocol : SATA
Drive Type : SATA_SSD
After all of my research I started to think there should be a missmatch between LSI-3008 firmware/driver ↔ Crucial CT4000MX500SSD1 firmware.
And I have only one last finding is: TRIM and SG_UNMAP support for LSI HBAs and RAID controllers
So please tell me… Which part is the root cause and how can I solve it?
- Thanks