Unable to use SSH key authentication on newly created user account

TrueNAS General
1

I recently started using TrueNAS, and I want to create separate, non-root account for data, but I’m having some problems with key authentication. Password auth works fine if I turn it on, but there’s some kind of problem with the user account reading the public key. I’ve worked with SSH keys before and I’m familiar with the usual permissions issues, but this is a new one.

The key to log into the root account works just fine, it’s only the newly created user that has problems.

I created the new account using the TrueNAS interface. Shell is set to bash for testing purposes. TrueNAS insists on creating the account only on a subdir of /mnt, so I created a new general home folder within my ZFS pool. I added the key from the UI as well. SSH client shows a permission denied error. Testing it with password, works fine. I tried the same key with the root account, worked. I checked the keyfile permissions and they’re as expected: 700 on .ssh and 600 on authorized_keys. The user owns the folders. I tried recreating the user, switching to an RSA key, no luck.

This is a new pool and I haven’t done anything exotic with file permissions (to my knowledge), ZFS or otherwise. ACL type is set to POSIX and ACL mode is set to discard. I even tried giving the home directory its own dataset. Nothing.

AppArmor is enabled but I don’t see any reference in it to SSH and I haven’t changed any of the settings.

I have two clues that lead me to believe it’s a permissions issue of SOME kind, but not a normal one. First, the contents of the auth log.

tail -f /var/log/auth.log

Feb 19 10:36:07 nas audit[12580]: USER_LOGIN pid=12580 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=login acct="storage" exe="/usr/sbin/sshd" hostname=? addr=192.168.xxx.xxx terminal=sshd res=failed' Feb 19 10:36:07 truenas sshd[12580]: Could not open user 'storage' authorized keys '/mnt/pool/home/storage/.ssh/authorized_keys': Permission denied Feb 19 10:36:07 truenas sshd[12580]: Could not open user 'storage' authorized keys '/mnt/pool/home/storage/.ssh/authorized_keys2': Permission denied Feb 19 10:36:07 truenas sshd[12580]: Connection closed by authenticating user storage 192.168.xxx.xxx port 53164 [preauth] Feb 19 10:36:07 truenas audit[12580]: USER_ERR pid=12580 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=192.168.xxx.xxx addr=192.168.xxx.xxx terminal=ssh res=failed' Feb 19 10:39:50 truenas su[12644]: (to storage) root on pts/1 Feb 19 10:39:50 truenas su[12644]: pam_unix(su:session): session opened for user storage(uid=3000) by root(uid=0) Feb 19 10:41:04 truenas su[12644]: pam_unix(su:session): session closed for user storage

It’s definitely failing to read the authorized_keys file, but I have no idea WHY. The permissions are right. I used su to log into the storage account and I can cat the file fine, so in that sense the permissions are correct… but I did notice something else odd. When I first su to the storage user, I get this error.

bash: /mnt/pool/home/storage/.bashrc: Permission denied

However it’s ONLY for this file, and I can still cat it immediately afterward. I’m wondering if a similar thing is happening with the authorized_keys file and I just can’t see it. But why would it come up with a permissions error on .bashrc and still allow me to access the file immediately afterward?

I’m a bit stumped on this one. I have a sneaking suspicion it’s related to storing the home directory on ZFS, but I really don’t know what difference it would make or how to fix it.

2

Some additional info. I bypassed the UI and went to regular shell commands for account management and created a new user with a home directory on the boot-pool (I know you’re not supposed to do either of these things in TrueNAS, but this was for science).

And… it worked. I was able to log in with the SSH key.

The boot pool is also ZFS, so I don’t think the problem can be with ZFS as such… but could there be some obscure setting or something? I dumped all zfs dataset settings and looked them over. I am not an expert at this level, but I found a few differences seemingly related to permissions.

Restricted ACL settings were Discard on the storage pool, but Restricted on the boot pool (both defaults). I don’t really see how that would make any difference except to inheritance, and the permissions were manually set/verified. Even then, I think this only matters for extended ZFS ACLs, which I’m not using on the storage pool at all and I don’t think are used on the boot pool. I do note that the boot pool setting for acltype is off, while for the data pool it’s set to POSIX (again, both defaults).

This seemed promising… but it didn’t change anything. I made a new dataset on the storage pool with the setting Off rather than POSIX, made a new user, checked file permissions, and tried again. Same result as before. Got the same inability to read the authorized_keys file, same weird .bashrc error when using su to switch to the storage user.

What could possibly be different between the two pools that would cause this?

3

I’m sorry that I can’t provide an answer to your question, but thought I would join the thread and will post any resolution I might find here as I hope you will also.

I’ve been having the same issue trying to use ssh keys to login to my truenas box from a Almalinux 8.10 workstation. I’m started using truenas scale 25.10.1 early this year and recently did the update to 25.10.2.1 and still have the problem. I had resorted to using the password option (not recommended by Truenas) for the user accounts and only periodically have spent too much time trying to figure out how to get the key login to work. I’ve used ssh logins for years on a very large enterprise system and never had this much difficulty. After numerous attempts, it also occurred to me that it may have something to do with the user home folder being on a normal dataset zvol as opposed to the /home folder on the boot pool as you have apparently discovered. On today’s attempt to get it working as expected I tried using the ssh -vvv acctxxx@nas.local to try to get some indication as to where the login was breaking down and stumbled on your post.

Are you also using a Truenas (Goldeye) version?

4

Not sure if this is the issue, but authorized_keys does not need to be restrictive and in fact, can be world-readable because it’s just all public keys (notice the word, public).

5

I’ve seen it said that authorized_keys can have 600, 644, or 640 permission for openssh, but I don’t understand what 644 or 640 provide for. Since the containing .ssh directory is required to have 700 permission for openssh , that alone would effectively prevent any world or group read, write, execute access to the file. I quickly tried changing the authorized_keys file to 644 and 640 and still get the same failure on trying to ssh into the NAS. I think the OP is correct that there is some squirrely truenas permissions issue with having user home directories on a zvol dataset as Truenas suggests. I am able to successfully login with password, but Truenas also recommends not to allow password ssh. So the Truenas recommendations lead to a catch 22 without identifying what the required trick is to get it to work in accordance with their recommendations.

I enabled ssh for the truenas_admin account and copied the user public key to his .ssh directory and the ssh login from any of my network PCs to the NAS works as expected, presumably since the truenas admin home directory is in the boot_pool /home folder.

Everything seems to point to some kind of dataset or zvol permissions setting on the non boot_pool zvol home directory or possibly some ssh daemon configuration setting that OP and I are missing.