Unable to write files to SMB shares

Mugiwara · April 3, 2024, 10:24pm

Hello,

I recently updated to TrueNAS-13.0-U6.1 from 11.3-U5 (I know, long time, big upgrade) a few weeks ago. All seemed well. Then today, for the first time, I tried copying some files from my Windows box to a SMB share, and I am getting a permissions error no matter where I try to copy the file to or from. I can browse, read, and even delete files just fine, but I cannot write anything. From previous forum posts (Unable to connect to domain after upgrade from 11.2.7 to 11.3 | TrueNAS Community) it looks like my AD connection may be the issue, but I cannot seem to figure out what is wrong. I have posted the output of my domain status queries (domain obfuscated with DOMAIN), does anyone have any ideas what I can do to be able to write files from my windows clients again? Thanks.

Samba4 Log output

[2024/04/03 14:40:00.091629, 2] …/…/source3/lib/interface.c:346(add_interface)
added interface em1 ip=192.168.200.153 bcast=192.168.200.255 netmask=255.255.255.0
[2024/04/03 14:40:00.091649, 2] …/…/source3/lib/interface.c:346(add_interface)
added interface vmx0 ip=192.168.200.160 bcast=192.168.200.255 netmask=255.255.255.0
[2024/04/03 14:40:00.091657, 2] …/…/source3/lib/interface.c:346(add_interface)
added interface em0 ip=192.168.200.170 bcast=192.168.200.255 netmask=255.255.255.0
[2024/04/03 14:40:00.093559, 1] …/…/source3/librpc/crypto/gse_krb5.c:418(fill_mem_keytab_from_system_keytab)
…/…/source3/librpc/crypto/gse_krb5.c:417: krb5_kt_start_seq_get failed (No such file or directory)
[2024/04/03 14:40:00.094222, 1] …/…/source3/librpc/crypto/gse_krb5.c:418(fill_mem_keytab_from_system_keytab)
…/…/source3/librpc/crypto/gse_krb5.c:417: krb5_kt_start_seq_get failed (No such file or directory)
[2024/04/03 14:40:00.095588, 1] …/…/source3/librpc/crypto/gse_krb5.c:418(fill_mem_keytab_from_system_keytab)
…/…/source3/librpc/crypto/gse_krb5.c:417: krb5_kt_start_seq_get failed (No such file or directory)
[2024/04/03 14:40:00.101189, 2] …/…/source3/param/loadparm.c:2889(lp_do_section)
Processing section “[nas]”
[2024/04/03 14:40:00.102299, 2] …/…/source3/smbd/service.c:958(make_connection_snum)
192.168.200.175 (ipv4:192.168.200.175:63067) connect to service nas initially as user DOMAINNET\lidarr (uid=21653, gid=21648) (pid 2725)
[2024/04/03 14:40:11.164071, 2] …/…/source3/smbd/service.c:1239(close_cnum)
192.168.200.175 (ipv4:192.168.200.175:63067) closed connection to service nas
[2024/04/03 14:50:54.434366, 2] …/…/source3/lib/tallocmsg.c:84(register_msg_pool_usage)
Registered MSG_REQ_POOL_USAGE
[2024/04/03 14:50:54.438539, 2] …/…/source3/smbd/server.c:816(remove_child_pid)
Could not find child 2882 – ignoring
[2024/04/03 15:00:30.621979, 2] …/…/source3/smbd/open.c:1686(open_file)
DOMAINNET\USER opened file Unsorted/1.png read=Yes write=Yes (numopen=6)
[2024/04/03 15:00:30.628610, 2] …/…/source3/smbd/open.c:1686(open_file)
DOMAINNET\USER opened file Unsorted/1.png read=No write=No (numopen=7)
[2024/04/03 15:00:30.639846, 2] …/…/source3/smbd/open.c:1686(open_file)
DOMAINNET\USER opened file Unsorted/1.png:Zone.Identifier read=Yes write=Yes (numopen=8)
[2024/04/03 15:00:30.642257, 1] …/…/source3/smbd/trans2.c:6633(smb_set_file_dosmode)
smb_set_file_dosmode: file_set_dosmode of Unsorted/1.png failed: Operation not permitted
[2024/04/03 15:00:30.643544, 2] …/…/source3/smbd/close.c:837(close_normal_file)
DOMAINNET\USER closed file Unsorted/1.png:Zone.Identifier (numopen=6) NT_STATUS_OK
[2024/04/03 15:00:30.643579, 2] …/…/source3/smbd/close.c:837(close_normal_file)
DOMAINNET\USER closed file Unsorted/1.png (numopen=5) NT_STATUS_OK
[2024/04/03 15:00:30.644869, 2] …/…/source3/smbd/close.c:837(close_normal_file)
DOMAINNET\USER closed file Unsorted/1.png (numopen=4) NT_STATUS_OK
[2024/04/03 15:00:36.749294, 2] …/…/source3/smbd/open.c:1686(open_file)
DOMAINNET\USER opened file Unsorted/1.png read=Yes write=Yes (numopen=6)
[2024/04/03 15:00:36.758247, 2] …/…/source3/smbd/open.c:1686(open_file)
DOMAINNET\USER opened file Unsorted/1.png read=No write=No (numopen=7)
[2024/04/03 15:00:36.758447, 2] …/…/source3/smbd/open.c:1686(open_file)
DOMAINNET\USER opened file Unsorted/1.png:Zone.Identifier read=Yes write=Yes (numopen=8)
[2024/04/03 15:00:36.765999, 1] …/…/source3/smbd/trans2.c:6633(smb_set_file_dosmode)
smb_set_file_dosmode: file_set_dosmode of Unsorted/1.png failed: Operation not permitted
[2024/04/03 15:00:36.767257, 2] …/…/source3/smbd/close.c:837(close_normal_file)
DOMAINNET\USER closed file Unsorted/1.png:Zone.Identifier (numopen=6) NT_STATUS_OK
[2024/04/03 15:00:36.767296, 2] …/…/source3/smbd/close.c:837(close_normal_file)
DOMAINNET\USER closed file Unsorted/1.png (numopen=5) NT_STATUS_OK
[2024/04/03 15:00:36.768817, 2] …/…/source3/smbd/close.c:837(close_normal_file)
DOMAINNET\USER closed file Unsorted/1.png (numopen=4) NT_STATUS_OK

AD status Log Output

root@nas:~ # net -k -d 5 ads status
WARNING: The option -k|–kerberos is deprecated!
INFO: Current debug levels:
all: 5
tdb: 5
printdrivers: 5
lanman: 5
smb: 5
rpc_parse: 5
rpc_srv: 5
rpc_cli: 5
passdb: 5
sam: 5
auth: 5
winbind: 5
vfs: 5
idmap: 5
quota: 5
acls: 5
locking: 5
msdfs: 5
dmapi: 5
registry: 5
scavenger: 5
dns: 5
ldb: 5
tevent: 5
auth_audit: 5
auth_json_audit: 5
kerberos: 5
drs_repl: 5
smb2: 5
smb2_credits: 5
dsdb_audit: 5
dsdb_json_audit: 5
dsdb_password_audit: 5
dsdb_password_json_audit: 5
dsdb_transaction_audit: 5
dsdb_transaction_json_audit: 5
dsdb_group_audit: 5
dsdb_group_json_audit: 5
lp_load_ex: refreshing parameters
Initialising global parameters
INFO: Current debug levels:
all: 5
tdb: 5
printdrivers: 5
lanman: 5
smb: 5
rpc_parse: 5
rpc_srv: 5
rpc_cli: 5
passdb: 5
sam: 5
auth: 5
winbind: 5
vfs: 5
idmap: 5
quota: 5
acls: 5
locking: 5
msdfs: 5
dmapi: 5
registry: 5
scavenger: 5
dns: 5
ldb: 5
tevent: 5
auth_audit: 5
auth_json_audit: 5
kerberos: 5
drs_repl: 5
smb2: 5
smb2_credits: 5
dsdb_audit: 5
dsdb_json_audit: 5
dsdb_password_audit: 5
dsdb_password_json_audit: 5
dsdb_transaction_audit: 5
dsdb_transaction_json_audit: 5
dsdb_group_audit: 5
dsdb_group_json_audit: 5
Processing section “[global]”
doing parameter dns proxy = No
doing parameter aio max threads = 2
doing parameter max log size = 5120
doing parameter load printers = No
doing parameter printing = bsd
doing parameter disable spoolss = Yes
doing parameter dos filemode = Yes
doing parameter kernel change notify = No
doing parameter directory name cache size = 0
doing parameter server multi channel support = No
doing parameter nsupdate command = /usr/local/bin/samba-nsupdate -g
doing parameter unix charset = UTF-8
doing parameter log level = 2 auth_json_audit:3@/var/log/samba4/auth_audit.log
doing parameter obey pam restrictions = False
doing parameter rpc_daemon:mdssd = disabled
doing parameter rpc_server:mdssvc = disabled
doing parameter enable web service discovery = True
doing parameter logging = file
doing parameter server min protocol = SMB2_02
doing parameter unix extensions = No
doing parameter restrict anonymous = 2
doing parameter server string = Andrew’s Storage Server
doing parameter bind interfaces only = Yes
doing parameter netbios name = nas
doing parameter netbios aliases =
doing parameter server role = member server
doing parameter kerberos method = secrets and keytab
doing parameter workgroup = DOMAINNET
doing parameter realm = DOMAIN.NET
doing parameter security = ADS
doing parameter local master = No
doing parameter domain master = No
doing parameter preferred master = No
doing parameter winbind cache time = 7200
doing parameter winbind max domain connections = 10
doing parameter client ldap sasl wrapping = seal
doing parameter template shell = /bin/sh
doing parameter template homedir = /home/%D/%U
doing parameter ads dns update = Yes
doing parameter allow trusted domains = Yes
doing parameter winbind enum users = Yes
doing parameter winbind enum groups = Yes
doing parameter winbind use default domain = Yes
doing parameter idmap config DOMAINNET: backend = rid
doing parameter idmap config DOMAINNET: range = 20000-90000000
doing parameter idmap config *: backend = tdb
doing parameter idmap config *: range = 90000001-100000000
doing parameter ea support = no
doing parameter store dos attributes = no
doing parameter map archive = no
doing parameter map hidden = no
doing parameter map readonly = no
doing parameter map system = no
doing parameter registry shares = yes
doing parameter include = registry
doing parameter registry shares = yes
process_registry_service: service name global
pm_process() returned Yes
added interface em1 ip=192.168.200.153 bcast=192.168.200.255 netmask=255.255.255.0
added interface vmx0 ip=192.168.200.160 bcast=192.168.200.255 netmask=255.255.255.0
added interface em0 ip=192.168.200.170 bcast=192.168.200.255 netmask=255.255.255.0
Registering messaging pointer for type 2 - private_data=0x0
Registered MSG_REQ_POOL_USAGE
Registering messaging pointer for type 11 - private_data=0x0
Registering messaging pointer for type 12 - private_data=0x0
Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED
Registering messaging pointer for type 1 - private_data=0x0
Registering messaging pointer for type 5 - private_data=0x0
Registering messaging pointer for type 51 - private_data=0x0
added interface em1 ip=192.168.200.153 bcast=192.168.200.255 netmask=255.255.255.0
added interface vmx0 ip=192.168.200.160 bcast=192.168.200.255 netmask=255.255.255.0
added interface em0 ip=192.168.200.170 bcast=192.168.200.255 netmask=255.255.255.0
Opening cache file at /var/run/samba4/gencache.tdb
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
ads_dc_name: domain=DOMAINNET
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
get_sorted_dc_list: attempting lookup for name DOMAIN.NET (sitename Default-First-Site-Name)
saf_fetch: Returning “DOMAINserver.DOMAIN.net” for “DOMAIN.NET” domain
get_dc_list: preferred server list: “DOMAINserver.DOMAIN.net, *”
namecache_fetch: name DOMAIN.NET#1C found.
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
namecache_fetch: name DOMAINserver.DOMAIN.net#20 found.
get_dc_list: returning 2 ip addresses in an ordered list
get_dc_list: 192.168.200.250 192.168.200.252
ads_try_connect: sending CLDAP request to 192.168.200.250 (realm: DOMAIN.NET)
Successfully contacted LDAP server 192.168.200.250
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
saf_fetch: Returning “DOMAINserver.DOMAIN.net” for “DOMAIN.NET” domain
get_dc_list: preferred server list: “DOMAINserver.DOMAIN.net, *”
resolve_ads: Attempting to resolve KDCs for DOMAIN.NET using DNS
dns_rr_srv_fill_done: async DNS AAAA lookup for DOMAINserver.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS AAAA lookup for DOMAINserver.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for DOMAINserver.DOMAIN.net [0] got DOMAINserver.DOMAIN.net → 192.168.200.2 50
dns_rr_srv_fill_done: async DNS A lookup for DOMAINserver.DOMAIN.net [0] got DOMAINserver.DOMAIN.net → 192.168.200.2 50
dns_rr_srv_fill_done: async DNS AAAA lookup for wvm2016-01.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for wvm2016-01.DOMAIN.net [0] got wvm2016-01.DOMAIN.net → 192.168.200.252
dns_rr_srv_fill_done: async DNS A lookup for WVM2016-01.DOMAIN.net [0] got wvm2016-01.DOMAIN.net → 192.168.200.252
dns_rr_srv_fill_done: async DNS AAAA lookup for WVM2016-01.DOMAIN.net returned 0 addresses.
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
namecache_fetch: name DOMAINserver.DOMAIN.net#20 found.
get_dc_list: returning 2 ip addresses in an ordered list
get_dc_list: 192.168.200.250 192.168.200.252
saf_fetch: Returning “DOMAINserver.DOMAIN.net” for “DOMAIN.NET” domain
get_dc_list: preferred server list: “DOMAINserver.DOMAIN.net, *”
resolve_ads: Attempting to resolve KDCs for DOMAIN.NET using DNS
dns_rr_srv_fill_done: async DNS A lookup for wvm2016-01.DOMAIN.net [0] got wvm2016-01.DOMAIN.net → 192.168.200.252
dns_rr_srv_fill_done: async DNS AAAA lookup for wvm2016-01.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for WVM2016-01.DOMAIN.net [0] got WVM2016-01.DOMAIN.net → 192.168.200.252
dns_rr_srv_fill_done: async DNS AAAA lookup for WVM2016-01.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for DOMAINserver.DOMAIN.net [0] got DOMAINserver.DOMAIN.net → 192.168.200.2 50
dns_rr_srv_fill_done: async DNS AAAA lookup for DOMAINserver.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for DOMAINserver.DOMAIN.net [0] got DOMAINserver.DOMAIN.net → 192.168.200.2 50
dns_rr_srv_fill_done: async DNS AAAA lookup for DOMAINserver.DOMAIN.net returned 0 addresses.
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
namecache_fetch: name DOMAINserver.DOMAIN.net#20 found.
get_dc_list: returning 2 ip addresses in an ordered list
get_dc_list: 192.168.200.250 192.168.200.252
create_local_private_krb5_conf_for_domain: wrote file /var/run/samba4/smb_krb5/krb5.conf.DOMAINNET with realm DOMAIN. NET KDC list:
kdc = 192.168.200.250
kdc = 192.168.200.252

ads_dc_name: using server=‘DOMAINSERVER.DOMAIN.NET’ IP=192.168.200.250
ads_try_connect: sending CLDAP request to 192.168.200.250 (realm: DOMAIN.NET)
Successfully contacted LDAP server 192.168.200.250
Connecting to 192.168.200.250 at port 389
Connected to LDAP server DOMAINserver.DOMAIN.net
KDC time offset is 0 seconds
Found SASL mechanism GSS-SPNEGO
ads_sasl_spnego_bind: got OID=1.3.6.1.4.1.311.2.2.30
ads_sasl_spnego_bind: got OID=1.2.840.48018.1.2.2
ads_sasl_spnego_bind: got OID=1.2.840.113554.1.2.2
ads_sasl_spnego_bind: got OID=1.2.840.113554.1.2.2.3
ads_sasl_spnego_bind: got OID=1.3.6.1.4.1.311.2.2.10
GENSEC backend ‘gssapi_spnego’ registered
GENSEC backend ‘gssapi_krb5’ registered
GENSEC backend ‘gssapi_krb5_sasl’ registered
GENSEC backend ‘spnego’ registered
GENSEC backend ‘schannel’ registered
GENSEC backend ‘naclrpc_as_system’ registered
GENSEC backend ‘sasl-EXTERNAL’ registered
GENSEC backend ‘ntlmssp’ registered
GENSEC backend ‘ntlmssp_resume_ccache’ registered
GENSEC backend ‘http_basic’ registered
GENSEC backend ‘http_ntlm’ registered
GENSEC backend ‘http_negotiate’ registered
Starting GENSEC mechanism spnego
Starting GENSEC submechanism gse_krb5
smb_gss_krb5_import_cred ccache[FILE:/tmp/krb5cc_0] failed with [ Miscellaneous failure (see text): unknown mech-code 2 for mech 1 2 840 113554 1 2 2] -the caller may retry after a kinit.
Failed to start GENSEC client mech gse_krb5: NT_STATUS_INTERNAL_ERROR
gensec_spnego_create_negTokenInit_step: Failed to setup SPNEGO negTokenInit request
gensec_update_done: spnego[0x8108cb960]: NT_STATUS_INVALID_PARAMETER
ads_sasl_spnego_gensec_bind(KRB5) failed for ldap/DOMAINserver.DOMAIN.net with user[root] realm=[DOMAIN.NET]: Cannot read password
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
ads_dc_name: domain=DOMAINNET
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
get_sorted_dc_list: attempting lookup for name DOMAIN.NET (sitename Default-First-Site-Name)
saf_fetch: Returning “DOMAINserver.DOMAIN.net” for “DOMAIN.NET” domain
get_dc_list: preferred server list: “DOMAINserver.DOMAIN.net, *”
namecache_fetch: name DOMAIN.NET#1C found.
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
namecache_fetch: name DOMAINserver.DOMAIN.net#20 found.
get_dc_list: returning 2 ip addresses in an ordered list
get_dc_list: 192.168.200.250 192.168.200.252
ads_try_connect: sending CLDAP request to 192.168.200.250 (realm: DOMAIN.NET)
Successfully contacted LDAP server 192.168.200.250
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
saf_fetch: Returning “DOMAINserver.DOMAIN.net” for “DOMAIN.NET” domain
get_dc_list: preferred server list: “DOMAINserver.DOMAIN.net, *”
resolve_ads: Attempting to resolve KDCs for DOMAIN.NET using DNS
dns_rr_srv_fill_done: async DNS A lookup for WVM2016-01.DOMAIN.net [0] got WVM2016-01.DOMAIN.net → 192.168.200.252
dns_rr_srv_fill_done: async DNS AAAA lookup for WVM2016-01.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for DOMAINserver.DOMAIN.net [0] got DOMAINserver.DOMAIN.net → 192.168.200.2 50
dns_rr_srv_fill_done: async DNS AAAA lookup for DOMAINserver.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for DOMAINserver.DOMAIN.net [0] got DOMAINserver.DOMAIN.net → 192.168.200.2 50
dns_rr_srv_fill_done: async DNS AAAA lookup for DOMAINserver.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for wvm2016-01.DOMAIN.net [0] got wvm2016-01.DOMAIN.net → 192.168.200.252
dns_rr_srv_fill_done: async DNS AAAA lookup for wvm2016-01.DOMAIN.net returned 0 addresses.
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
namecache_fetch: name DOMAINserver.DOMAIN.net#20 found.
get_dc_list: returning 2 ip addresses in an ordered list
get_dc_list: 192.168.200.250 192.168.200.252
saf_fetch: Returning “DOMAINserver.DOMAIN.net” for “DOMAIN.NET” domain
get_dc_list: preferred server list: “DOMAINserver.DOMAIN.net, *”
resolve_ads: Attempting to resolve KDCs for DOMAIN.NET using DNS
dns_rr_srv_fill_done: async DNS A lookup for DOMAINserver.DOMAIN.net [0] got DOMAINserver.DOMAIN.net → 192.168.200.2 50
dns_rr_srv_fill_done: async DNS AAAA lookup for DOMAINserver.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for wvm2016-01.DOMAIN.net [0] got wvm2016-01.DOMAIN.net → 192.168.200.252
dns_rr_srv_fill_done: async DNS AAAA lookup for wvm2016-01.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for WVM2016-01.DOMAIN.net [0] got WVM2016-01.DOMAIN.net → 192.168.200.252
dns_rr_srv_fill_done: async DNS AAAA lookup for WVM2016-01.DOMAIN.net returned 0 addresses.
dns_rr_srv_fill_done: async DNS A lookup for DOMAINserver.DOMAIN.net [0] got DOMAINserver.DOMAIN.net → 192.168.200.2 50
dns_rr_srv_fill_done: async DNS AAAA lookup for DOMAINserver.DOMAIN.net returned 0 addresses.
sitename_fetch: Returning sitename for realm ‘DOMAIN.NET’: “Default-First-Site-Name”
namecache_fetch: name DOMAINserver.DOMAIN.net#20 found.
get_dc_list: returning 2 ip addresses in an ordered list
get_dc_list: 192.168.200.250 192.168.200.252
create_local_private_krb5_conf_for_domain: wrote file /var/run/samba4/smb_krb5/krb5.conf.DOMAINNET with realm DOMAIN. NET KDC list:
kdc = 192.168.200.250
kdc = 192.168.200.252

ads_dc_name: using server=‘DOMAINSERVER.DOMAIN.NET’ IP=192.168.200.250
ads_try_connect: sending CLDAP request to 192.168.200.250 (realm: DOMAIN.NET)
Successfully contacted LDAP server 192.168.200.250
Connecting to 192.168.200.250 at port 389
Connected to LDAP server DOMAINserver.DOMAIN.net
KDC time offset is 0 seconds
Found SASL mechanism GSS-SPNEGO
ads_sasl_spnego_bind: got OID=1.3.6.1.4.1.311.2.2.30
ads_sasl_spnego_bind: got OID=1.2.840.48018.1.2.2
ads_sasl_spnego_bind: got OID=1.2.840.113554.1.2.2
ads_sasl_spnego_bind: got OID=1.2.840.113554.1.2.2.3
ads_sasl_spnego_bind: got OID=1.3.6.1.4.1.311.2.2.10
Starting GENSEC mechanism spnego
Starting GENSEC submechanism gse_krb5
smb_gss_krb5_import_cred ccache[FILE:/tmp/krb5cc_0] failed with [ Miscellaneous failure (see text): unknown mech-code 2 for mech 1 2 840 113554 1 2 2] -the caller may retry after a kinit.
Failed to start GENSEC client mech gse_krb5: NT_STATUS_INTERNAL_ERROR
gensec_spnego_create_negTokenInit_step: Failed to setup SPNEGO negTokenInit request
gensec_update_done: spnego[0x8108cb360]: NT_STATUS_INVALID_PARAMETER
ads_sasl_spnego_gensec_bind(KRB5) failed for ldap/DOMAINserver.DOMAIN.net with user[root] realm=[DOMAIN.NET]: Cannot read password
return code = -1

LarsR · April 4, 2024, 6:25am

Which User are you using to connect to the Share? And are you sharing out the root dataset or a child dataset? With truenas 12.something there was a change so that root can no longer access shares and the permissions of the root Dldataset cant be changed and can no longer be used for shares.

Davvo · April 4, 2024, 7:14am

Can you share with us your ACL?

Mugiwara · April 6, 2024, 5:42pm

Thank you both for the response, and I think what LarsR is saying may be what I am running into. If thats the case, then, are there any easy answers here other than rolling back? For some reason all of my automated processes still appear to be writing to their shares just fine, its only user interactive processes (as far as I can tell) that are failing, which is why it took me a few weeks to even notice.

As far as how I manage the SMB share, I grant permissions on TrueNAS to “everyone” within the domain, and then I use folder permissions to restrict the folders to specific domain users. Im not sure how to get the SMB ACLs from the CLI, so I have attached screenshots of the configuration. Any suggestions on how to get out of this without having to reconfigure share paths on dozens of devices but sharing a non-root dataset would be appreciated.

Pool and Share info

zfs list | grep -i wdc
wdc8TBRed 48.5T 33.2T 48.5T /mnt/wdc8TBRed
wdc8TBRed/.system 532M 33.2T 1.26M legacy
wdc8TBRed/.system/configs-66311c036e824820af44b2dbf4c55f10 365M 33.2T 365M legacy
wdc8TBRed/.system/cores 188K 1024M 188K legacy
wdc8TBRed/.system/rrd-66311c036e824820af44b2dbf4c55f10 141M 33.2T 141M legacy
wdc8TBRed/.system/samba4 5.92M 33.2T 1.25M legacy
wdc8TBRed/.system/services 205K 33.2T 205K legacy
wdc8TBRed/.system/syslog-66311c036e824820af44b2dbf4c55f10 18.1M 33.2T 18.1M legacy
wdc8TBRed/.system/webui 188K 33.2T 188K legacy

root@nas:/mnt # sharesec --view-all
[nas]
REVISION:1
CONTROL:SR|DP
OWNER:
GROUP:
ACL:S-1-1-0:ALLOWED/0x0/FULL
ACL:S-1-5-21-23653053-2197752985-2071920098-500:ALLOWED/0x0/FULL
ACL:S-1-5-21-23653053-2197752985-2071920098-512:ALLOWED/0x0/FULL

Mugiwara · April 6, 2024, 5:43pm

ACLs since I could only attach one image per post

LarsR · April 6, 2024, 5:46pm

You can create a child dataset and move over your Data from the Root Dataset to the Child, but that should be done via cli or you rollback to the previous version, create the Child Datast and move everything over via network share.
After moving the Data you have to edit the permissions so your users have access rights to the child Dataset and adjust your jail mountpoints, if you’re using jails and mount data to them.

Mugiwara · April 6, 2024, 6:01pm

Thanks for the info! This particular share does not house any jails (there’s a separate pool for those). Would moving the data from to a child dataset reset the permissions on the folders? This is a large dataset (over 100TB), with hundreds of custom permissioned folders (windows based AD ACLs), so I really wouldn’t want to do anything that might reset all of those permissions. Im assuming I could create the child dataset and set the share mountpoint (/nas in my case) on the child which should mean my share paths on my remote hosts should stay the same, but I want to make sure permissions would too. Also, if I do the mv via CLI, it wouldn’t have to actually move all the data, it would just change the pointer, right (it would take days to weeks to actually do a full move to a new location over my 1GBe network)?

winnielinnie · April 6, 2024, 6:05pm

Did you actually save files/folders directly in the root dataset?

If you’re simply “sharing out” the root dataset, as a matter of convenience, then you can instead create separate shares for each separate child dataset.

What does the pool’s layout look like?

zfs list -r -t filesystem wdc8TBRed

Mugiwara · April 6, 2024, 6:18pm

It is the root unfortunately. I never intended to use it for anything by a very large shared pool on the network so when I created it (like 12 years ago now), I was pretty naïve to storage and didnt anticipate needing to divide it up.

Essentially, the root dataset is shared as “nas”, so clients use smb to connect to
smb:nas.domain.net/nas/folders, where “nas” is the actual share off the dataset

zfs list

root@nas:/mnt # zfs list -r -t filesystem wdc8TBRed
NAME USED AVAIL REFER MOUNTPOINT
wdc8TBRed 48.5T 33.2T 48.5T /mnt/wdc8TBRed
wdc8TBRed/.system 532M 33.2T 1.26M legacy
wdc8TBRed/.system/configs-66311c036e824820af44b2dbf4c55f10 365M 33.2T 365M legacy
wdc8TBRed/.system/cores 188K 1024M 188K legacy
wdc8TBRed/.system/rrd-66311c036e824820af44b2dbf4c55f10 141M 33.2T 141M legacy
wdc8TBRed/.system/samba4 5.92M 33.2T 1.25M legacy
wdc8TBRed/.system/services 205K 33.2T 205K legacy
wdc8TBRed/.system/syslog-66311c036e824820af44b2dbf4c55f10 18.2M 33.2T 18.2M legacy
wdc8TBRed/.system/webui 188K 33.2T 188K legacy

Client Access Example

PS Microsoft.PowerShell.Core\FileSystem::\nas\nas> ls

Directory: \\nas\nas

Mode LastWriteTime Length Name

d----- 3/23/2024 23:13 Backups
d----- 3/3/2024 21:09 Books
d----- 12/18/2017 14:31 Bookss
d----- 1/6/2024 19:14 Downloads
d----- 1/28/2024 20:39 Media
d----- 7/28/2023 18:04 Pictures
d----- 9/16/2018 11:08 Raw
d----- 5/12/2020 16:00 Edited
d----- 10/9/2023 09:41 Working
d----- 8/25/2022 13:49 scans
d----- 3/20/2019 18:16 Software
d----- 4/3/2024 15:00 Unsorted
------ 12/12/2018 18:16 3513223 20181212_175645.jpg
------ 12/9/2020 15:56 3927454 20201209_155603.jpg
------ 12/24/2020 07:06 3156940 20201224_070628.jpg
------ 1/25/2021 15:31 4428236 20210125_153155.jpg
------ 2/2/2021 14:04 3647340 20210202_140453.jpg
------ 3/17/2021 17:51 3305880 20210317_175131.jpg
------ 8/2/2023 09:36 63488 Thumbs.db

PS Microsoft.PowerShell.Core\FileSystem::\nas\nas> pwd

Path

Microsoft.PowerShell.Core\FileSystem::\nas\nas

winnielinnie · April 6, 2024, 6:27pm

That’s not good.

Your 2 choices are:

Move ~50 TiB of data into a newly created child dataset. (Hopefully it doesn’t get interrupted or result in lost data.)
Use the command-line to override the root dataset permissions (which is discouraged, and could introduce problems with the middleware later on).
Not possible, since it’s the root dataset instead of a child: Use the zfs rename feature to shuffle the dataset to a new location within the pool.

Unless you have a spare 50+ TiB pool as a backup that you can use to temporarily hold the data as you create a new pool?

(Do you have such a backup for all this data in the first place?)

Mugiwara · April 6, 2024, 6:37pm

While I backup about 4TB of data in important folders, the rest, while it would be a bummer, I don’t care about losing enough to build another redundant 100TB storage system.

This is a built on a zfs2, 2 8-drive pools consisting of a total of 16 8TB drives. I do not have enough slack storage to be able to copy it all somewhere. I am anticipating upgrading to a new setup within the next few years (mainly to reduce the number of drives and power consumption), but thats far enough away that it won’t help me.

I guess I could just revert back to 11.3 for now (it was working fine, which is why I hadn’t upgraded in so long), and run on that for the next couple of years if the my only alternates are to somehow find a place to stage a copy over the data. Alternatively, if I am OK running on this revision for a while and wouldn’t upgrade until I’m ready to move to a new setup.

zpools

root@nas:/mnt # zpool status
pool: Jails
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using ‘zpool upgrade’. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:42 with 0 errors on Sun Mar 10 00:00:42 2024
config:

    NAME                                          STATE     READ WRITE CKSUM
    Jails                                         ONLINE       0     0     0
      gptid/5867ebc4-e08c-11e7-91af-000c290e9eb1  ONLINE       0     0     0

errors: No known data errors

pool: cloud
state: ONLINE
scan: scrub repaired 0B in 00:01:24 with 0 errors on Sun Mar 10 00:01:24 2024
config:

    NAME                                          STATE     READ WRITE CKSUM
    cloud                                         ONLINE       0     0     0
      gptid/0ea2caf9-c076-11ee-b2eb-000c290e9ebb  ONLINE       0     0     0

errors: No known data errors

pool: freenas-boot
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using ‘zpool upgrade’. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:26 with 0 errors on Sun Mar 31 03:45:26 2024
config:

    NAME          STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      da0p2       ONLINE       0     0     0

errors: No known data errors

pool: wdc8TBRed
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using ‘zpool upgrade’. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 11:53:30 with 0 errors on Sun Mar 10 11:53:30 2024
config:

    NAME                                            STATE     READ WRITE CKSUM
    wdc8TBRed                                       ONLINE       0     0     0
      raidz2-0                                      ONLINE       0     0     0
        gptid/da236a18-53c1-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/da9b66da-53c1-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/db17e55d-53c1-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/dba37408-53c1-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/dc215693-53c1-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/6e7f797a-84e7-11ea-b875-000c290e9ebb  ONLINE       0     0     0
        gptid/dd2839bb-53c1-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/ddaddddb-53c1-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
      raidz2-1                                      ONLINE       0     0     0
        gptid/08fa490b-53d2-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/4f5b8a5d-6a23-11ee-a6fa-000c290e9ebb  ONLINE       0     0     0
        gptid/0a1ef707-53d2-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/0aaecd43-53d2-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/0b3d956e-53d2-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/0bd92415-53d2-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/0c6f4c81-53d2-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0
        gptid/0d10b861-53d2-11e7-9ee5-000c290e9eb1  ONLINE       0     0     0

errors: No known data errors

Stux · April 6, 2024, 6:43pm

Well. You can mv the files in the shell. You can use tmux to make it more tolerant of disconnections.

You could do it in batches.

And if you erased the snapshots then you’d recover the space as the moves completed.

Mugiwara · April 6, 2024, 6:53pm

I’m willing to looking into figuring out a way to move everything. What I am worried about is all my (Windows, active directory) configured folder shares within the current SMB share. I could potentially go back and recreate them all, but that would be an exercise in finding broken things for the next few months as automated writes start failing due to me missing a share group or user on one of the sub folders.

This is for home use, but I also use the server as a backup location for a number of family members, and write a lot of data from different automation systems that all have different service accounts configured depending on what they are doing or writing from. Probably needlessly complex, but its how the system works for now, so I would like to (hopefully) not break all the automation that is currently in place.

If there is any documentation on overriding the root dataset perms, maybe I could also just take a look at that and see how it goes?

Mugiwara · April 13, 2024, 2:07am

Just as an update, I wasn’t able to figure out how to mess with the permissions on the mount to get it to work, so I booted to the previous environment and that seemed to resolve the issue for now. I guess I will run on this old version until I can figure out what to do next with move the data. Thanks all for the hep troubleshooting and the suggestions.