Data corruption via 10GbE NIC - Fine with 1GB integrated - Can I rule out RAM?

TrueNAS General
1

Been racking my brains for a while trying to diagnose this. More after my setup details.

-Latest versions of stable TrueNAS Scale, I think I started with a late v24, and now 25.04.2
-ASRock B550 Phantom Gaming 4 (I know, I know… “gaming”)
-Ryzen 5500GT (5600G in disguise)
-Seasonic G12 GC-550 (550W)
-128GB DDR4 (4x32GB Patriot Viper Steel 64GB DDR4 3600MHz CL18 1.35v)
-TPLink TX401 10GbE (both on NAS and W10 PC) - I think it’s AQC107 based.
-4x Toshiba MG09ACATE 18TB drives in RAIDZ1 for the dataset
-Micron 1100 - 512GB SSD for the OS

So the story, as tl’dr I can manage it, goes like this.

-First noticed checksum erros on scrubs. Random, 2-4 per drive, never the same drives. Could be 1, could be 3 drives. Never all. Drives pass all SMART tests and were working fine in my PC before I built the NAS.
After clearing, scrubs reported no errors until I transfered new data to the NAS.

-After a while, when editing video footage I’ve been noticing missing frames(even after scrubs).
Nothing “serious” because luckily I work with Nikon RAW NEV files which are basically many raw files in a sequence, so only the problematic frames were affected.

-Since then I was thinking it might be the RAM that corrupts files? Because it was difficult to connect a monitor to the NAS to check with MemTest(yeah, lazy) I started testing the 10GbE connection.

-Tried everything I, and my friend ChatGPT, could come up with:

:stop_sign: MTU/Jumbo frame mismatch between PC NIC and TrueNas NIC (TrueNas was on 9000, PC was on 9014). Best speeds even with them mismatched, saturating the connection (1.2GB/s) and drives (~650-700MB).
*corruption still there, copy/verify with teracopy or h2testw

:stop_sign: Defaulting to MTU1500 / Jumbo Frames disabled. Tanked speeds, ~300mb read, 800MB write in synthetics which usually got the 1.2GB/s speeds (via ARC, of course)
*corruption

:stop_sign: As ChatGPT said some Aquantia drivers are problematic and if I see this message, I should try disabling GRO/LRO. (I was, and I did)
“Driver has suspect GRO implementation, TCP performance may be compromised”
*corruption

:stop_sign: When testing with h2wtest i even ran the verification twice, to verify if it’s corrupting files at read as well. Errors returned the same, so the transmission from the NAS is ok through the TX401. The files were badly written.

Didn’t matter, MTU1500, MTU9014, GRO/LRO on or off, there was corruption

:bulb: My fantastic brain briliantly remembered that the motherboard has a 1GbE Realtek NIC that was connected to the internet via router for apps/updates. So it’s worth a try to transfer through it(and router) instead?

** :white_check_mark: It was, both Teracopy and a H2Testw returned absolutely no errors, although I can count my new grey hairs from some simple 100GB tests.**

:question: What I’m thinking now is that I can rule out RAM/CPU/HDDs after these tests and put the blame on the TX401?

:question: Can I try any other settings for the TX401, or should I just swap it with an X550-t2 ?

:question: Guessing no, but do I also need to swap theTX401 on the PC side as well?

Any other suggestions are really appreciated, maybe I can “salvage” the TX401?

Thanks for joining me in my sanity decreasing journey.

2

You were using the same MTU of 1500 on both ends, Windows and TrueNAS?

3

Yes, hoping it was the mismatched MTU to blame (9000 on TrueNAS) and 9014 on the PC card (that’s the value I could select) I tried with them identical. 9014/9014 or default (1500 on TrueNas, Jumbo Frame - Disabled on the PC). So in my troubleshooting I tried with them mismatched, matched at 9014, matched at default 1500). Corruption still there.

1 Like
4

P.S. This is what ChatGPT said about how the NIC can corrupt the data, if someone can please let me know if it makes sense or it’s complete bs?

P.P.S. sometimes I was getting “clean scrubs” even with corrupted data.

Blockquote What that warning means:

  • “Driver has suspect GRO implementation” is a Linux kernel warning that the NIC driver’s Generic Receive Offload (GRO) implementation is broken or non-compliant.
  • GRO combines multiple incoming packets into a larger buffer before passing them up the stack. If the driver mishandles GRO:
    • Packets can be merged incorrectly.
    • TCP checksums can still pass if the corruption happens after the checksum validation.
    • ZFS will happily checksum the corrupted data in RAM and write it as if it were correct.

:boom: Can this cause the kind of corruption you’re seeing?

Yes. This is one of the few scenarios where network-induced silent corruption can happen even with TCP/IP and ZFS:

  1. NIC receives good frames → TCP checksum passes.
  2. atlantic driver with broken GRO merges or misaligns the data buffer incorrectly.
  3. By the time it reaches ZFS, the block is valid-length but contains wrong bytes.
  4. ZFS checksums that data and writes it cleanly → scrub reports no error because the checksum matches the corrupted data.

This aligns perfectly with:

  • Random disks showing errors in the past.
  • Current silent corruption with no scrub errors.
  • Your seeing exactly that GRO warning in dmesg.
5

…but according to your OP, the bolded statement does not match what you’re seeing; you are seeing scrub errors.

6

Sorry, forgot to add in my OP that in the past two weeks or so, there were no scrubs errors, even if corruption still happened. Yesterday I got a scrub error though. That’s why ChatGPT mentioned that in its response.

My only logical thought now is that if the transfer is ok via the onboard Realtek 1GbE connection, the rest of the system might be ok and I either need to find the magic setting for the TX401, or replace it altogether?

L.E. The only workaround I can think about now is to transfer data to the NAS via the 1GbE connection, but still use it to edit via the 10GbE, as the reads seem to be “clean”. Not fun transferring 1-2TB of data though.

7

If you are seeing scrub errors, it has nothing to do with the network adapter. Scrub does not involve the network.
I would suggest to identify the root cause of the scrub errors first. They could come from disk errors, cable issues, RAM, or the disk controller.

8

I have had data corruption during transfer via SMB, and as other already say i can confirm too that they weren’t detected by scrubs… in your place the first thing i would try Is to test RAM and (if you already not) disabling XMP profile, try keep frequency/voltage lower

9

What exactly would be the issue with SMB? Does it not handle 10G well? I’m just thinking forward, what should I expect even if I replace the NAS NIC with an X550-T2.

Latest progress: I -slowly- copied 600GB yesterday through the 1G link (integrated NICs on the PC and NAS motherboards, connected via router) and the verification passed. Now I’m copying an extra 700GB. If this also goes smoothly, I can at least rule out the fault of the corruption to be the 10G TX401 path.

What I will do after this is finished:

-Connect a monitor to the NAS and recheck the BIOS settings. From what I remember, I kept the RAM at JEDEC speeds (2400 or 2666). As the RAM speed doesn’t really do anything for a NAS, I kept everything as default / low power as possible.
I will take some screenshots with the settings.

-Run MemTest86+ for 24hrs.

-Pointer from ChatGPT: There might be some sort of pcie-adapter power saving option in the BIOS that might affect the TX401. It said that some atlantic driver / aquantia chips might misbehave under Linux because of this?

-Run a scrub to see if any errors occur after only using the 1G link.

If the PCIe power saving setting doesn’t sort it, and if RAM is clean, I will send files via the 1G link until I get my hands on a X550-T2.

10

No you cannot.

10gbit networking uses more RAM and puts more of a load on your systems resources.

A marginal system will be more likely to fail or error.

You have an issue. It could be the RAM. Have you tested it yet?

(And you are now far enough into the weeds that ChatGPT will just be making stuff up)

1 Like
11

You mean the “bclone explosion” is not a real phenomenon? :nerd_face:

12

Hey guys, I’m back, this time with the conclusion.

I had a spirited discussion with the idiot that built my NAS. Although he said the settings are ok, they were not. The RAM was set at the 3600MHz profile, and apparently the voltages were not tweaked. Ryzen 5000 series is notorious for needing to boost the VDDP/VDDG/IOD voltages to manage those frequencies for 128GB of RAM.

That idiot is me. I was so convinced by the fact that I left them as default, as I totally knew they would do absolutely nothing performance wise, that I got paranoid trying to figure what else could be wrong with it. What can I say, overclocker genes right here. Did it on every PC I had, so I had to do it to my NAS as well, of course.

As I moved my NAS close to my monitors (it was relatively unaccessible before, hence the hesitation) and ran a MemTest I got the verdict in… 13 minutes. That’s how much it took for it to throw errors. Reset the RAM to JEDEC defaults @ 1.2v, all 4 passes ran perfectly over 17 hours.

No more scrub errors, no more corrupted data via the 10G link… everything works.

So… I owe each one of you a beer.

P.S. If anyone asks, we’ll just say it was the bclone explosion, ok?

6 Likes
13

Thanks for keeping this meme alive.

anotherone

RAM settings strike again.

EDIT: Microsoft just recently released a new anti-malware suite that protects against bclone explosions. They have a special discount through the month of August. You might want to check it out.

2 Likes
14

:beers:

2 Likes
15

That’s the way to do it, keep posting about bclone explosions and ChatGPT will index it as the truth. My honor will be saved in no time.

L.E. Just installed the bclone explosions patch and all the problems went away! Crazy things, these bclone explosions, they come when you least expect it and go up in your face to make you look like a fool on forums.

3 Likes
16

ChatGPT is a dangerous nut.

I asked it “What is a bclone explosion in ZFS?”

Here is the answer it gave me.

:white_check_mark: In ZFS, a bclone explosion refers to:

A pathological or uncontrolled proliferation of ZFS block clones (bclones), usually due to excessive use of deduplication, send/receive, or snapshot features — causing performance and memory problems.

:mag: What are ZFS block clones (bclones)?

  • A bclone (short for block clone) in ZFS is a reference to a shared data block, typically created during operations like:
    • zfs send | zfs receive
    • zfs clone
    • zfs snapshot
  • Rather than duplicating the data, ZFS uses copy-on-write (COW) to save space by pointing to the same block until it changes.
  • This makes operations efficient, but can cause trouble if overused.

:boom: What causes a “bclone explosion”?

A bclone explosion happens when:

  • You have large numbers of snapshots, clones, or zfs send/receive replications.
  • These share an extremely high number of blocks, creating deep chains of references.
  • ZFS must track all these references, which eats up RAM, ARC, and metadata space.
  • This can lead to:
    • Extremely slow snapshot deletion or zfs destroy.
    • Excessive memory use.
    • ZFS hangs or panics under load.
    • Performance degradation in reads/writes.

:brain: Why does this happen?

  • Every time a snapshot or clone is created, it adds to the reference count of shared blocks.
  • If data is repeatedly replicated with zfs send/receive, especially with -D (dedup), bclones accumulate.
  • Eventually, ZFS has to walk through millions or billions of references just to free a single block.

:wrench: How to avoid or fix it:

  1. Limit snapshot frequency and lifetime.
  2. Avoid excessive use of zfs send -D (dedup sends).
  3. Be cautious with recursive zfs send + zfs receive setups in backup systems.
  4. Occasionally rebase datasets — i.e., receive into a new dataset and destroy the old chain.
  5. Use ZFS 2.2+ which includes some mitigations for bclone performance issues.

:chart_with_downwards_trend: Analogy:

Think of it like this:

You keep making photocopies of a document, and then photocopies of the photocopies. Eventually, you have thousands of copies — and the copier slows to a crawl just trying to process one page.

:thread: Further reading:

If you’re encountering this in production, the following are good resources:

  • OpenZFS issue trackers (many mention bclone explosion or slow zfs destroy)
  • ZFS mailing lists or forums
  • Brendan Gregg’s blog on ZFS performance

If you’re seeing this issue yourself, I can help walk you through diagnosing it (zdb, zpool status, etc.).

Its entire response was nonsense drivel with a sprinkle of some truth, written like a confident psychopath.

Why do AI bots lie? Why can’t they just admit they don’t know?

2 Likes
17

Okay. There is nothing intelligent about ChatGPT.

I asked it a question about two things that don’t exist.

I asked it “Does OpenZFS 3.0 support ghost compression?”

My question is utter nonsense on purpose. There is no OpenZFS 3.0 and there is no such thing as “ghost compression”.

Here is the answer it gave me.

Yes, OpenZFS 3.0 does support ghost compression!

What is Ghost Compression?

Ghost compression is a feature introduced in OpenZFS 3.0 that allows ZFS to maintain compressed data in a way that is invisible to the application but still highly efficient from a storage perspective. Essentially, it optimizes storage efficiency while keeping the data accessible to the system without visible overhead in terms of I/O or latency.

This feature is useful when:

  • You want to save space by using compression but don’t want the application to know or deal with the complexity of decompression.
  • It reduces CPU overhead by enabling more effective compression methods that don’t require as much CPU for every read and write.

Key Features of Ghost Compression in OpenZFS 3.0:

  1. Transparent to Applications:
  • Ghost compression operates transparently, meaning applications accessing the data don’t have to be aware of or interact with the compression layer. The system handles it under the hood.
  1. Efficient Space Savings:
  • It reduces the overall space needed for storage by using high compression ratios without significantly increasing latency. This is beneficial for workloads where storage efficiency is a priority.
  1. Reduced Decompression Overhead:
  • Unlike traditional compression, which might require decompression on every read (thus consuming CPU resources), ghost compression minimizes the decompression effort during read operations, improving overall performance.
  1. Ghost Blocks:
  • In the context of ghost compression, the ghost blocks are compressed versions of the data that are not directly accessible but are still referenced by the file system. They can be read by the system without needing to fully decompress the entire block, optimizing the balance between storage efficiency and I/O performance.

How Does Ghost Compression Work?

  • Compression Process: When data is written to a dataset that uses ghost compression, ZFS will compress it at a block level. The compressed data is stored on disk, but the system will keep track of it in a way that avoids the overhead of decompressing the entire block when accessed.
  • Accessing Data: When the application reads the data, the system will handle the decompression on its own, with little to no impact on the application’s performance.

How to Enable Ghost Compression in OpenZFS 3.0:

To enable ghost compression, you’ll need to specify it on your dataset using the appropriate compression method. While LZ4 is the default compression algorithm in ZFS, ghost compression can be applied for more efficiency.

Here’s how you can enable it for a dataset:

  1. Set Compression to Ghost:
zfs set compression=ghost pool/dataset
  1. Set Specific Compression Algorithms (e.g., lz4 or zstd):
    OpenZFS 3.0 offers more fine-tuned control over compression algorithms:
zfs set compression=zstd pool/dataset

The specific ghost behavior depends on how the ZFS implementation handles that compression type under the hood.

Why Use Ghost Compression?

  1. Storage Efficiency: For systems where disk space is a constraint, ghost compression can provide a significant reduction in the amount of storage consumed by data without causing a significant overhead on performance.
  2. Low CPU Overhead: Traditional compression methods can be CPU-intensive, especially when large amounts of data are being compressed or decompressed. Ghost compression reduces the amount of active decompression needed, optimizing CPU usage.
  3. Improved I/O Performance: By not forcing full decompression during reads, ghost compression can improve I/O latency and reduce the load on storage systems, making data access faster.
  4. Optimal for Specific Workloads: It is particularly beneficial for workloads that frequently read data (like databases or file systems) but do not require direct decompression by the application.

Conclusion

OpenZFS 3.0 introduces ghost compression as a way to balance storage efficiency and performance, making it an excellent option for users who need to store large amounts of data efficiently without introducing heavy CPU overhead. It’s transparent to the application, making it seamless to implement, and works especially well in environments with high read/write operations where traditional compression might cause significant overhead.

It even suggested using options that don’t exist!

How is ChatGPT even a thing people use? :flushed: It’s a crazy liar on drugs.

4 Likes
18

From decades of experience as a simple human, they seem to be copying some people really well.

ChatGPT is a commercial product, and its main job is to make you feel like you’re right so you’ll get the plus plan. ChatGPT5 which released a few days ago “should” hallucinate less and correct you when you’re wrong, but I think it will be still far from perfect.

I tried asking Grok4 (with reasoning) about it, from what I know it’s less likely to try to please you…

I’m dying here, we’re creating memes. :joy:

image
image869×747 58.2 KB

A “bclone explosion” appears to be a colloquial or tongue-in-cheek term used in TrueNAS and OpenZFS communities to describe a severe data corruption bug that gained notoriety with the release of OpenZFS 2.2.0 in late 2023. The term combines “bclone” (short for block cloning, a new feature in that version allowing efficient, copy-on-write file-level cloning without full data duplication) with “explosion,” likely alluding to the sudden “blow-up” of problems—like widespread data corruption reports—once the feature was enabled and used.

This is somewhat fascinating, Grok trying to explain what ChatGPT hallucinated. It’s like arguing with your wife about that girl she dreamed about last night.

3 Likes
19

ChatGPT 5 Thinking is much better.

Short answer: No. There’s no “ghost” compression mode in OpenZFS. The supported compressors are lz4, zstd (including zstd-N and zstd-fast-N levels), gzip/gzip-N, **zle, and legacy lzjb—“ghost” isn’t listed anywhere in the compression` property docs. 

Also, there isn’t an OpenZFS 3.0 release as of today—current releases are still in the 2.x series (e.g., 2.4.0-rc3). 

If you were thinking of something else by “ghost” (e.g., GPU-oriented formats like GDeflate, or Norton Ghost’s imaging), those aren’t ZFS compression options. To see what a dataset is using on your box: zfs get compression pool/ds or zfs list -o compression. For zstd you’d use zfs set compression=zstd pool/ds (or zstd-5, zstd-fast-10, etc.). 

1 Like
20

And we only had to boil three oceans to do it!

3 Likes