Systemic issues with permissions and SMB

Hello. I have had for the past couple of years an oddly consistent yet intermittent problem with connecting some Linux machines to TrueNAS 13.0.x SMB shares. I’m currently running 13.0-U6.1. This seems like it may be an ACL problem on the TrueNAS side.

The Clients
The Linux machines are always Debian based (Ubuntu, Mint, or Raspberry Pi OS) and packages are always updated to the latest at the time of incident. They have no problem mounting other hosts’ SMB shares. But when there is a problem with accessing one SMB share there is a problem with accessing all of them.

Note that during all of this I have zero problems with Linux and Windows machines accessing around 20 SMB shares continuously.

The Scenario
For a given affected machine, I reproduce the problem like so:

  1. Install the OS with the default packages. Update to latest packages.

Note: Most recently the OSes have been Linux Mint 21.3 xfce and Raspberry Pi OS Bookworm (based on Debian 12 Bookworm).

  1. Add the following line to /etc/fstab:
    //10.0.1.245/pidev /mnt/pidev cifs vers=3.0,cifsacl,credentials=/home/pi/.smbcredentials,rw,uid=1040,gid=1040,noperm 0 0

Note that defining the uid and gid are needed to ensure that the client user’s uid and gid of 1000 are “remapped” to the uid and gid of 1040 on the SMB share. There is no change when removing the uid, gid, and noperm options.

  1. Create .smbcredentials with the correct credentials and execute mount -a, which will return the following:
    mount error(13): Permission denied

What I Have Tried

  • On the client, verify that the credentials are correct, via smbclient --list 10.0.1.245 -U SMBUSER

  • On the client, verify that /mnt/pidev exists and has the correct permissions.

  • On the client, try a simplified entry to fstab:
    //10.0.1.245/pidev /mnt/pidev cifs credentials=/home/pi/.smbcredentials,rw 0 0

  • On the client, try a manual mount:
    mount -t cifs -o user=SMBUSER //10.0.1.245/pidev /mnt/pidev

  • On the client, execute systemctl daemon-reload

  • On the client, try accessing other SMB shares hosted on the TrueNAS machine. I am never able to access any other SMB shares on this TrueNAS machine. I can always access SMB shares hosted on other machines, however.

  • On TrueNAS, create a test SMB share with the filesystem ACLs and share ACLs both allowing all users and groups full control. The dataset has SMB-type ACLs.

  • On TrueNAS, create a SMB share that shares a test dataset with General-type ACLs. Match the filesystem ACLs to the dataset that I’m actually trying to share.

  • On TrueNAS, create a SMB share that shares a test dataset with General-type ACLs with the filesystem ACLs and SMB share ACLs both allowing all users and groups full control.

  • On the client, create a local machine user with the same uid and gid as the dataset owner on TrueNAS and eliminate the uid, gid, and noperm options in the /etc/fstab entry.

  • On the client, wait some time. Sometimes the issue resolves itself in a few days and I can connect and the issue doesn’t come back.

  • Reinstall TrueNAS and restore from backup config file.

  • Reinstall TrueNAS and reconfigure manually.

  • Reinstall the client OS. There is some chance that doing this will completely eliminate the issue even though I follow exactly the same steps as above for setup and don’t install any further packages and do minimal configuration. If an affected system becomes unaffected then I do my best to make sure that the system doesn’t need a OS wipe anytime soon.
    From time to time a machine will get its OS reinstalled and usually the OS is the same distro but a newer version. During this, if the system becomes affected when it was previously not affected, I will go through the client-side attempts listed in this list to verify it’s not operator error, and if it persists then I will wait some days, and then if it again persists I will try a reinstall. Reinstall resolves the issue rarely, and it seems that the longer I wait to reinstall (weeks or months) the more likely it is to resolve the issue, even though I’m installing with the same installation media.

Try mounting without cifsacl (this may cause clients to strip ACLs during chmod operations), but keep noperm.

Thanks for your reply. Here’s the new /etc/fstab entry that I tried:
//10.0.1.245/pidev /mnt/pidev cifs credentials=/home/pi/.smbcredentials,rw,uid=1040,gid=1040,noperm 0 0

The same was returned:
mount error(13): Permission denied

I also tried a manual mount with these options:
mount -t cifs -o user=SMBUSER -o uid=1040 -o gid=1040 -o noperm //10.0.1.245/pidev /mnt/pidev

And again the same was returned:
mount error(13): Permission denied

I have added to my original post one thing that I tried previously: reinstalling TrueNAS.