25.10: 90-minute SMART poll (smartctl -x) stalls SATA drives behind a SAS expander – reproducible, isolated to the expander

TrueNAS General
,
1

Summary

On TrueNAS CE 25.10.7, the middleware’s periodic SMART poll runs smartctl -x on every disk. For SATA drives attached through my SAS expander, the SCT log reads included in -x hang whenever the drive is busy. The drive stops answering all commands (including ZFS reads/writes) until the kernel aborts and resets it after 30 s. The poll walks the disks one by one, so the pool stalls for up to ~10 minutes, every 90 minutes and shortly after boot. The same drives on the same HBA and cable work perfectly when connected directly to the HBA, bypassing the expander.

This looks like the same issue as the thread “Drives start timing out every 90 minutes” (March 2026), where the problem followed one particular backplane across several HBAs.

System

  • TrueNAS CE 25.10.7 (kernel 6.12.105-production+truenas, smartmontools 7.4)
  • HBA: Supermicro-branded LSI SAS3008, IT mode, firmware 16.00.12.00 (also seen on 16.00.01.00)
  • Expander: Intel RES3FV288, firmware B057 (latest; SES reports serial “PMCSIERAD”)
  • Disks: 12× Seagate Exos X16 16 TB SATA (ST16000NM001G), firmware SN03 (10) and SB40 (2), as 2× 6-wide RAIDZ2
  • HDD Standby: Always On, APM: Disabled

Symptoms

  • Transfers drop to 0 MB/s; iostat shows one disk at 100 % util with no I/O.
  • Kernel log, one disk at a time:
  sd 1:0:2:0: attempting task abort!scmd(...), outstanding for 31588 ms & timeout 30000 ms
  sd 1:0:2:0: task abort: SUCCESS scmd(...)
  sd 1:0:2:0: Power-on or device reset occurred
  • Episodes start on a fixed 90-minute clock from boot (8 episodes fit a 5404 s period within ±13 s), plus one ~3 min after boot. That is why boots take 10–15 minutes before shares/iSCSI are usable.
  • Episodes only happen when the pool is busy at that moment; idle polls pass silently.
  • zpool status stays clean (0/0/0), so no data errors so far.

What triggers it

  • alert/source/smart.py: SMARTAlertSource uses IntervalSchedule(timedelta(minutes=90)); check_sync() loops over all disks from disk.get_disks() and calls disk.smartctl_info().
  • utils/disks_/disk_class.py line 320: cmd = ["smartctl", "-x", self.devpath]
  • There is no per-disk opt-out that I can find (togglesmart is gone in 25.10).

Live capture during an episode (ps every second). All processes are children of middlewared, in D state at blk_execute_rq:

Disk smartctl -x running
sdg < 1 s
sdl 11:42:10–11:43:14 (65 s)
sdj 11:43:15–11:44:20 (66 s)
sdk 11:44:21–11:45:58 (98 s)
sdf 11:45:59–11:47:36 (98 s)
sdc 11:47:37–11:48:41 (65 s)
sdn 11:48:42–11:50:20 (99 s)
sdm 11:50:21–11:51:25 (65 s)
sde < 1 s

65 s and 98 s equal 2 and 3 abort/reset cycles of ~32 s. The full poll took 9 min 17 s.

Reproduction (will briefly stall the pool)

dd if=/dev/sdX of=/dev/null bs=1M count=6000 iflag=direct status=none &
sleep 3; time smartctl -l scterc /dev/sdX
  • Idle disk: 0.06 s.
  • Busy disk behind the expander: 30.4 s, “Read SCT Status failed: Input/output error”, then a task abort and device reset. -l scttemp: 32.8 s, same failure.
  • With /sys/block/sdX/device/timeout raised to 120, smartctl’s own 60 s pass-through timeout expired instead. The drive still never answered. The hung command:
  CDB: ATA command pass through(16) 85 08 0e 00 d5 00 01 00 e0 00 4f 00 c2 00 b0 00

That is SMART READ LOG, log E0h (SCT status); the next one was E1h (SCT data).

  • smartctl -a and smartctl -H under the same load: 0.26 s and 0.11 s, no errors.

Isolating the component

I moved one breakout cable (4 disks) from the expander to the HBA’s second port. Same disks, cable, HBA, firmware and driver; only the expander is removed from the path.

Path SCT query under load
Direct on HBA (SN03 and SB40 disks) 17/17 OK, 0–1 s
Behind expander 10/10 failed, ~30 s + reset

Moving the expander’s power from a powered riser to a motherboard PCIe slot changed nothing.

Ruled out: cables (phy error counters 0 on all disk links; uplink counters only increase with link resets), PSU, HBA firmware, disk firmware revision, HDD standby/APM, smartd (not running), other scheduled tasks (cron, systemd timers, apps).

Requests

  1. Use smartctl -a in the periodic poll (safe here under load), or at least drop the SCT queries (-l scttemp, -l scterc) that -x includes.
  2. Make the poll configurable (interval, off, or per-disk exclusion), or skip disks that have I/O in flight.

Has anyone seen the same with other expanders (LSI/Broadcom backplanes, Adaptec etc.)? The reproduction above takes a minute. Happy to run further tests.

2

That 30 s abort/reset pattern would make me test the expander’s SAT path next.

-while the poll is running, save dmesg -T and the expander SES page with sg_ses -p 0x2 /dev/sgX if that sg node is exposed. sg_logs -a /dev/sgX may show a link or PHY event too.
-run the same disk with the default device type, then with explicit smartctl -x -d sat and smartctl -a -d sat. If only -x hangs behind the expander, that gives iX a smaller SAT/SCT reproduction.
-until this is fixed, disable the 90-minute SMART task or keep these disks on the direct HBA path. Raising the device timeout probably just makes the pool wait longer.

3

Thanks a lot for taking the time to look at this, and for the concrete suggestions. I’ve now run through them, and the results are below.

1. Testing the expander’s SAT path

That’s where I ended up after the last round of tests. I moved one breakout cable from the expander to the HBA’s second port, so everything was equal except for the path (some through the expander and some direct):

Path smartctl -l scterc under load
Directly on the HBA OK, 0–1 s
Behind the expander Failed, ~30 s, then task abort + reset

As far as I understand, the SCSI-to-ATA translation itself happens in the HBA, and it handles these commands fine when the drive is direct-attached. So the part that fails seems to be the STP connection through the expander.

2. Capturing dmesg and the expander’s SES/log pages during a hang

I took sg_ses -p 0x2 from the expander before, during and after a hang. The expander answered normally during the hang, so its management side doesn’t freeze, only the connection to the drive that received the SCT command. Apart from the enclosure’s INFO summary flag being set after the drive reset, nothing in the status page pointed to a fault.

sg_logs -a isn’t supported by this expander (LOG SENSE returns Illegal request). The PHY error counters for the drive links stay at 0 throughout, so I don’t see any link-level events.

dmesg from the -d sat run (sdj is the drive behind the expander). The command that times out is one of dd’s normal reads, so the drive stops serving ordinary I/O while the SCT query is outstanding:

[Sun Oct  4 18:29:11 2026] sd 1:0:8:0: attempting task abort!scmd(0x00000000e80728a0), outstanding for 31308 ms & timeout 30000 ms
[Sun Oct  4 18:29:11 2026] sd 1:0:8:0: [sdj] tag#1611 CDB: Read(16) 88 00 00 00 00 00 00 18 48 00 00 00 08 00 00 00
[Sun Oct  4 18:29:11 2026] scsi target1:0:8: handle(0x000f), sas_address(0x50000d1f035cbf0b), phy(11)
[Sun Oct  4 18:29:11 2026] sd 1:0:8:0: task abort: SUCCESS scmd(0x00000000e80728a0)
[Sun Oct  4 18:29:12 2026] sd 1:0:8:0: Power-on or device reset occurred

3. Default device type vs. explicit -d sat

smartctl already uses SAT for these drives by default (earlier kernel logs show ATA PASS-THROUGH(16) for the hung command), but I ran your test with -d sat on the same drive under the same load:

  • smartctl -d sat -x: 100 s in total (three ~32 s abort/reset cycles, the same pattern as during the middleware’s poll)
  • smartctl -d sat -a: under 1 s

So yes, only -x hangs, and smartctl -l scterc alone is enough to reproduce it, with both pass-through sizes:

  • -d sat: 31 s, “Read SCT Status failed: Input/output error”
  • -d sat,12: 32 s, same failure

4. Workarounds until it’s fixed

  • Disabling the 90-minute task: To be clear, this isn’t something I’ve set up. It’s built into TrueNAS 25.10: the middleware’s SMART alert source (SMARTAlertSource, IntervalSchedule(timedelta(minutes=90))) runs smartctl -x on every disk (disk_class.py line 320). It’s separate from the scheduled S.M.A.R.T. test tasks, and I haven’t found a supported way to turn it off.
  • Direct HBA path: Unfortunately not an option for the whole pool. I have 12 drives, and the HBA has physical connections for 8.
  • Device timeout: I tried 120 s: the pass-through command then hit smartctl’s own 60 s timeout (the kernel log shows timeout 60000 ms), and the drive still didn’t answer before it was reset.