SOLVED: Bug in Electric Eel RC1 SMB Service with MacOS 15.0?

Good afternoon,
I was using Carbon Copy Cloner (CCC) to conduct a backup of my NAS (see profile below) running RC1 of EE, Normally, this is a routine operation with little to no drama. Instead, CCC reported that multiple files refused to be read, files that had nonsensical names.

The connection between my Mac (Sequoia 15.0) and the NAS is SMB over 10GbE Fiber. Reverting to Dragonfish 24.04.2 restores normalcy, i,e, all files are accessible and I am not getting random errors like this one:

The error makes no sense as these files were previously served up w/o issues under Dragonfish, but in Electric Eel, their files names are allegedly too long. I hope someone else can confirm a similar error with their version of EE or explain to me why the exact same settings in SMB with the exact same dataset causes these sorts of error in EE but not in Dragonfish?

I have filed a report in Jira and submitted my debug file (though it’s of the working DF environment, not the faulty EE one). I am somewhat reluctant to spin up EE again and potentially damage the pool. FWIW, it affected 1 file out of ~28k in my music folder, ~250 out of ~400k images and related files in my pictures dataset. So by no means a very common error.

1 Like

Do you have an example of a file name that errored?

Your wish is my command. This is a random music file.

I miss-spoke. It’s file 36 out of 56.

Thank goodness for rollbacks… I had deleted the file thinking it was a corrupted one. Rolled back to a earlier snapshot… and the file was there as broken as in EE. Located it via the find file in iTunes / Music, it got renamed and re-ordered.

The pictures folder was not so lucky. I will have to look into how these files were renamed, as some of the folders remain mis-spelled. I will have to use an older backup to hunt down and restore older folder names / file names.

That’s a mangled name due to the presence of illegal NTFS characters in it.

1 Like

This how itunes restored it: “36 Is This Our Earth_ (Mixed).M4A”

What is the local path on TrueNAS? The way itunes changes it is irrelevant.

Here is another example: On my remote NAS, the folder name renders correctly, i.e.
Screenshot 2024-10-07 at 16.56.15

(shell)

instead of this
Screenshot 2024-10-07 at 16.55.53

The folder name is mangled on my local NAS but not on the remote NAS that I got my correct folder name from. The two replicate to each other.

I got to the correct folder name by mounting the pool / dataset and using the shell to get to it. From there, I deduced what folder had been mangled and fixed it.

To me, this suggests I should roll back my snapshots until I find one that does not include a mangled file name / folder at that location. Or restore from the remote NAS. Either way, this is not good.

Question mark is an invalid NTFS character. You can disable name mangling in the SMB server, but they you’re breaking non-mac clients potentially (Windows definitely, others possibly).

2 Likes

Why would this work without issues before upgrading NAS to EE?

The remote NAS is reflecting what was on my primary NAS before upgrading to EE.

This very much reminds me of the issues SMB used to have with Mac file names / folders before you guys made it more Mac compatible. Yet, I have not changed any of the Mac compatibility settings in my datasets. It’s as if EE was ignoring some of them.

The most likely reason for this is you aren’t comparing apples to apples (different configuration). The correct Samba behavior with invalid file names is to mangle them (this makes it so that non-Mac clients can actually access the files). If I were to hazard a guess you’ve changed the mangling settings between the two versions.

I haven’t made any such changes. I merely upgraded the OS. Please consider the possibility that EE made some changes on its own, just as Dragonfish was not happy with SMB Aux parameters that CORE happily ignored.

The SMB options for the dataset feature both the AFP Legacy check box and the Apple Character compatibility checkboxes (both unset).
Screenshot 2024-10-07 at 17.22.40

I cannot remember if this dataset at one point was served via AFP or not. Is there a way to determine this?

Also, Apple-Character compatibility cannot be checked/unchecked. Is there a way to change that behavior by TrueNAS? I tried turning off SMB, disabling the SMB Share, etc. all to no avail. Neither TrueNAS 24.04.2 nor 24.10RC1 will allow me to enable Apple-style Character encoding.

@winnielinnie for the win!

  1. Go to Shares
  2. Edit each dataset individually from “default parameters” in Purpose,
  3. Select an alternative to default parameters like “no presets”
  4. Select “Advanced Options”
  5. Re-enable check-box next to “Use Apple-Style Character Encoding”
  6. Acknowledge “I understand”
  7. Hit Save at the bottom of the advanced window.

Do this for all your shares (and I have more than a few) and then this bug should be in your rear-view window. To be fair, for all I know this mangling may have started as part of my CORE-SCALE migration.

Thank you again, WinnieLinnie and @awalkerix!

I do not recommend doing this on existing data. That “I understand” contains relevant information that should not be ignored. This may indicate that your data was originally written with different mangling parameters in play. This is perhaps a solution for your problem, but should not be used for other users.

2 Likes

Thank you for that warning. Let me also re-emphasize the following based on my lived experience:

Please consider the possibility that the transition from DF to EE resets the Purpose / dataset SMB settings that are not explicitly Time Machine and hence inadvertently unsets the Apple-Characterset Compatibility Box.

Perhaps it’s because my Pool was originally built under CORE? I do not know. What I can assure is that I didn’t change any of these settings because Apple-Charset Compatibility is something I insisted on, see the old forum: Why I’m sticking with AFP (2019). Then there is the 2021 advice I gave someone suggesting I made the change to SMB-only sometime in 2020 or so: Using Apple-Charsets Compatibility in SMB (2021)

So these apple-charset boxes were set in CORE, as well as Dragonfish as best as I can tell because I had zero backup issues in July when I last made copies of the NAS content to a DAS using SMB/MacOS/CCC. These errors only popped up after I stopped using Dragonfish and transitioned to Electric Eel. Let me also repeat:

I never modified these charset compatibility boxes since switching to SMB from AFP. They unset themselves and the timing suggests that either EE Beta or EE RC1 were the cause.

1 Like

To close this thread out, iXsystems reports that they investigated and found no errors in their code. The most plausible explanation is user error.

Why I would change the purpose of 10 SMB shares after leaving those settings untouched for four years will remain a mystery to me. I didn’t even know where to look for these settings.

But for anyone else using MacOS with TrueNAS and having a sudden issue with mangled file names and inaccessible files on SMB shares after upgrading to Electric Eel, my suggestion would be as follows:

Mount the relevant dataset. Then use the shell to drop down into the dataset locally to verify that the data is actually good (ie cd /mnt/pool/dataset/directory). Now look for the offending files / folder via ls. Files or directories with illegal-to-NTFS characters like ? or an umlaut will trigger mangling unhappiness.

If all is well and your dataset contains files or folders with illegal-to-NTFS characters and hence likely used to have Apple Character Compatibility turned on, simply go back into the GUI, select Shares, click on the edit pencil associated with each dataset, and then change the “Purpose” of each share from Default to “No presets”. Follow that with click on advanced settings, enable ACC, acknowledge that terrible things can happen if you re-enable ACC, and finally hit save.

You get to do this for each and every SMB share that is not a Time Machine share. Huzzah. And then please let us all know if you too were sleepwalking and inadvertently changing each and every non-TM SMB share setting like I allegedly did.

1 Like

Btw, same problem and same solution but with Ubuntu 24.04 client. No MacOS involved. Using Scale ElectricEel-24.10.2, no migration involved.
Question: Any other setting(s) when creating an smb share that would avoid the issue? (Haven’t tried other Purposes or settings.) Thanks

Excellent question. So far, both of us appear to be edge cases that iXsystems cannot replicate. Hence, they cannot diagnose the chain of events that lead to the issue.

My suspicion is that it has to do with unexpected modification of the purpose of a share that goes sideways as part of the CORE->Scale transition.

Oh well. At least you found a solution to your issue! :grinning:

To add a data point as a Mac user, I did not encounter this issue when migrating from Dragonfish to EE, which I did a couple weeks ago since it was finally the .2 version. Could it be they had fixed said issue with migration?

I was told the code had no issues, ie that the most likely explanation was that I made said changes for 10 shares and then promptly forgot about it.

Such a scenario appears unlikely to me, since I hadn’t mucked about in the shares setups since I transitioned the NAS from AFP to SMB.

But since they could not replicate the issue, and it appears to be a rare issue, they logically closed the ticket. Maybe if more of us report the problem can the chain of events get uncovered.

In the meantime, standard SMB shares with no presets and ACC turned on work well with my MacOS household.