The affected interface is an Aquantia/Marvell 10GbE NIC using the atlantic driver at 10Gbps. Firmware reports as ATL2FW 103001f.
25.10.5: kernel 6.12.95-production+truenas
25.10.4: kernel 6.12.91-production+truenas
I rolled back to 25.10.4. The same server, NIC, configuration, and workload have now remained stable for more than 15 hours. All pools are healthy, overnight replication completed, and the full PBS backup job completed successfully.
I submitted a detailed bug report to TrueNAS with the debug bundle, journals, and pstore records. It was closed 11 minutes later because they do not have the hardware and said they would not invest time in fixing the issue, instead suggesting that I report it upstream.
Posting this so other Aquantia/Atlantic users are aware. Has anyone else seen similar crashes after upgrading to 25.10.5? For now, I am staying on 25.10.4.
Currently ‘functional’ (in the sense that I have a script on boot to delete & remake my network settings because otherwise I can’t even access lan after a reboot), but at least not getting any crashes.
Our NICs have always been firmly in the ‘not recommended’ section
Edit: I’ve had issues with my NIC since like… v24? maybe v23? I can’t remember when Kubs were removed in favor of docker.
Also I just read the Jira notes - that is rough man. I have no useful advice.
Yep…
So TrueNAS does not have each and every piece of hardware available. Fair enough.
Since TrueNAS cannot investigate and test, rolling back some driver or kernel changes at random isn’t an option.
All they could do would be to pass the whole thing upwards—and then they think that OP can do it himself. Let’s shoot into the dark in the general direction of Linux maintainers…
So there’s a NAS appliance which hopes for broad public distribution of its free Community Edition, but does not work beyond v.25.10.4 on certain consumer-grade 10G NICs which may be expected to be used by regular consumers and the only solution would be to upgrade to one of the server-grade NICs which are duly tested and validated on the paid Enterprise product.
Not to mention that the change from FreeBSD to Linux was supposed to bring more drivers for more hardware.
What went wrong here?
Following up on my earlier report with bisection-adjacent findings that narrow the search space.
Recap: two panics within ~40 min of moving from kernel 6.12.91 to 6.12.95 (TrueNAS SCALE 25.10.4 → 25.10.5), both under sustained RX load (ZFS replication + Proxmox Backup Server traffic) on an onboard AQC113 (fw ATL2FW 103001f):
BUG: kernel NULL pointer dereference, address: 0000000000000128
RIP: 0010:memcpy+0xc/0x20
Call Trace:
aq_ring_rx_clean+0x15b/0xe80 [atlantic]
aq_vec_poll+0x81/0x1d0 [atlantic]
Kernel panic - not syncing: Fatal exception in interrupt
Rolling back to 6.12.91 with identical hardware, config, and workload restored stability (multi-day uptime since).
New findings from diffing the trees:
drivers/net/ethernet/aquantia/ is byte-identical between v6.12.91 and v6.12.95 stable. The driver itself did not change in this range.
I also diffed the actual distro kernel trees ( GitHub - truenas/linux: Linux kernel for TrueNAS SCALE · GitHub , tags TS-25.10.4 and TS-25.10.5). The atlantic driver is stock/unpatched in both, and the distro’s own networking patches (ixgbe, ntb_netdev) are unrelated. So distro patches are ruled out; the regression window is the upstream stable v6.12.91..v6.12.95 backports.
Within that range, the only changes I can find in the RX path this trace lives in are in net/core/gro.c (skb_zcopy rejection in skb_gro_receive; pskb_may_pull guard in skb_gro_receive_list) and net/core/skbuff.c (pskb_trim_rcsum_slow restructuring). Nothing in dev.c, page_pool, NAPI, or the ipv4 GRO offload handlers.
Working hypothesis: a latent race in aq_ring_rx_clean exposed by changed GRO merge/flush behavior, rather than a new driver bug. Consistent with this, another AQC113CS user on the same kernel reports no panic, so the trigger appears traffic- or config-dependent.
I can test on 6.12.95 with GRO/LRO disabled (ethtool -K gro off lro off) under the same workload that reproduced the panic in under an hour, and report back. Happy to test patches;
I have EFI pstore capture working and full crash archives from both events available on request.
I don’t know if this is a useful data point or not- just wanted to mention that I have the same hardware and so far () have not hit the crash. System is a UGREEN DXP4800+. I’m currently only using the 10GbE interface as a dedicated iSCSI link for VM storage. A 10-minute iperf3 test was run with TN as the server and was stable. The client has an X710-DA2 on-link. Summary below.
Will keep an eye on this thread. Not great news, as there is no expansion option on this appliance.
Just an update. Reverting back to 25.10.4 apparently has stopped the crashes. My system has been online for more than 40 hours and has successfully accomplished two backup replication tasks and received data from two of my nightly Proxmox backup tasks from my PBS server. Clearly something in the 25.10.5 upgrade has caused the crashes that is not present in 25.10.4.
Being another Ugreen User (DXP6800 Pro) who has upgraded to 25.10.5 I’ve also tried bombarding my NAS with iPerf from three Gigabit Clients for 10 Minutes. Network interface of the NAS is running with 10G.
No crash here. Also the system’s total uptime is fine counting from the moment it got upgraded.
In my experience, the Atlantic driver is just not good and has never truly been stable enough for reliable 24/7 server use. I use it on client devices still, but only when I have no other choice.
) have not hit the crash. System is a UGREEN DXP4800+. I’m currently only using the 10GbE interface as a dedicated iSCSI link for VM storage. A 10-minute iperf3 test was run with TN as the server and was stable. The client has an X710-DA2 on-link. Summary below.