This one had me running in circles for two weeks, finally solved 2 nights ago. Confirmed that my fix works and they are still connected and working.
When I was trying to join a truenas install to the domain, I got BLASTED with this MEGAERROR (did my best to purge identifying info)
[EFAULT] [EFAULT] Current debug levels: all: 5 tdb: 5 printdrivers: 5 lanman: 5 smb: 5 rpc_parse: 5 rpc_srv: 5 rpc_cli: 5 passdb: 5 sam: 5 auth: 5 winbind: 5 vfs: 5 idmap: 5 quota: 5 acls: 5 locking: 5 msdfs: 5 dmapi: 5 registry: 5 scavenger: 5 dns: 5 ldb: 5 tevent: 5 auth_audit: 5 auth_json_audit: 5 kerberos: 5 drs_repl: 5 smb2: 5 smb2_credits: 5 dsdb_audit: 5 dsdb_json_audit: 5 dsdb_password_audit: 5 dsdb_password_json_audit: 5 dsdb_transaction_audit: 5 dsdb_transaction_json_audit: 5 dsdb_group_audit: 5 dsdb_group_json_audit: 5 ldapsrv: 5 lp_load_ex: refreshing parameters Initialising global parameters rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) INFO: Current debug levels: all: 5 tdb: 5 printdrivers: 5 lanman: 5 smb: 5 rpc_parse: 5 rpc_srv: 5 rpc_cli: 5 passdb: 5 sam: 5 auth: 5 winbind: 5 vfs: 5 idmap: 5 quota: 5 acls: 5 locking: 5 msdfs: 5 dmapi: 5 registry: 5 scavenger: 5 dns: 5 ldb: 5 tevent: 5 auth_audit: 5 auth_json_audit: 5 kerberos: 5 drs_repl: 5 smb2: 5 smb2_credits: 5 dsdb_audit: 5 dsdb_json_audit: 5 dsdb_password_audit: 5 dsdb_password_json_audit: 5 dsdb_transaction_audit: 5 dsdb_transaction_json_audit: 5 dsdb_group_audit: 5 dsdb_group_json_audit: 5 ldapsrv: 5 Processing section “[global]” doing parameter disable spoolss = True doing parameter dns proxy = False doing parameter load printers = False doing parameter max log size = 5120 doing parameter printcap = /dev/null doing parameter bind interfaces only = True doing parameter fruit:nfs_aces = False doing parameter fruit:zero_file_id = False doing parameter rpc_daemon:mdssd = disabled doing parameter rpc_server:mdssvc = disabled doing parameter restrict anonymous = 2 doing parameter winbind request timeout = 60 doing parameter passdb backend = tdbsam:/var/run/samba-cache/private/passdb.tdb doing parameter workgroup = AD doing parameter netbios name = doing parameter netbios aliases = doing parameter guest account = nobody doing parameter obey pam restrictions = False doing parameter create mask = 0664 doing parameter directory mask = 0775 doing parameter ntlm auth = False doing parameter server multichannel support = False doing parameter unix charset = UTF-8 doing parameter local master = False doing parameter server string = Backup file server doing parameter log level = 1 doing parameter logging = file doing parameter server smb encrypt = required doing parameter idmap config * : backend = tdb doing parameter idmap config * : range = 90000001 - 100000000 doing parameter idmap config * : read only = True doing parameter smb3 directory leases = no doing parameter server role = member server doing parameter kerberos method = secrets only doing parameter sync machine password to keytab = /etc/samba/kerberos/krb5.keytab0:account_name:sync_kvno:machine_password /etc/samba/kerberos/krb5.keytab1:sync_spns:sync_kvno:machine_password /etc/samba/kerberos/krb5.keytab2:spn_prefixes=nfs:sync_kvno:machine_password doing parameter security = ADS doing parameter domain master = False doing parameter preferred master = False doing parameter winbind cache time = 7200 doing parameter winbind max domain connections = 10 doing parameter winbind use default domain = False doing parameter client ldap sasl wrapping = seal doing parameter template shell = /bin/sh doing parameter allow trusted domains = False doing parameter realm = doing parameter template homedir = /var/empty doing parameter winbind enum users = True doing parameter winbind enum groups = True doing parameter machine password timeout = 0 doing parameter create krb5 conf = False doing parameter idmap config AD : backend = rid doing parameter idmap config AD : range = 100000001 - 200000000 doing parameter zfs_core:zfs_integrity_streams = False doing parameter zfs_core:zfs_block_cloning = False doing parameter registry shares = True doing parameter include = registry doing parameter registry shares = yes process_registry_service: service name global pm_process() returned Yes added interface enp6s18 ip= bcast= netmask= added interface enp6s18 ip= bcast= netmask= added interface enp6s18 ip= bcast= netmask= added interface enp6s18 ip= bcast= netmask= Registering messaging pointer for type 2 - private_data=(nil) register_msg_pool_usage: Registered MSG_REQ_POOL_USAGE Registering messaging pointer for type 11 - private_data=(nil) Registering messaging pointer for type 12 - private_data=(nil) Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED Registering messaging pointer for type 1 - private_data=(nil) Registering messaging pointer for type 5 - private_data=(nil) Registering messaging pointer for type 51 - private_data=(nil) added interface enp6s18 ip= bcast= netmask= added interface enp6s18 ip= bcast= netmask= added interface enp6s18 ip= bcast= netmask= added interface enp6s18 ip= bcast= netmask= libnet_Join: libnet_JoinCtx: struct libnet_JoinCtx in: struct libnet_JoinCtx dc_name : ‘’ machine_name : ‘’ domain_name : * domain_name : ‘’ domain_name_type : JoinDomNameTypeDNS (1) account_ou : NULL admin_credentials : * passed_machine_password : machine_password : join_flags : 0x00000023 (35) 0: WKSSVC_JOIN_FLAGS_IGNORE_UNSUPPORTED_FLAGS 0: WKSSVC_JOIN_FLAGS_JOIN_WITH_NEW_NAME 0: WKSSVC_JOIN_FLAGS_JOIN_DC_ACCOUNT 0: WKSSVC_JOIN_FLAGS_DEFER_SPN 0: WKSSVC_JOIN_FLAGS_MACHINE_PWD_PASSED 0: WKSSVC_JOIN_FLAGS_JOIN_UNSECURE 1: WKSSVC_JOIN_FLAGS_DOMAIN_JOIN_IF_JOINED 0: WKSSVC_JOIN_FLAGS_WIN9X_UPGRADE 0: WKSSVC_JOIN_FLAGS_ACCOUNT_DELETE 1: WKSSVC_JOIN_FLAGS_ACCOUNT_CREATE 1: WKSSVC_JOIN_FLAGS_JOIN_TYPE os_version : NULL os_name : NULL os_servicepack : NULL create_upn : 0x00 (0) upn : NULL dnshostname : ‘.’ modify_config : 0x00 (0) ads : NULL debug : 0x01 (1) secure_channel_type : SEC_CHAN_WKSTA (2) desired_encryption_types : 0x0000001c (28) provision_computer_account_only: 0x00 (0) odj_provision_data : NULL request_offline_join : 0x00 (0) Opening cache file at /var/run/samba-lock/gencache.tdb sitename_fetch: Returning sitename for realm ‘’: “” namecache_fetch: no entry for .#20 found. resolve_hosts: Attempting host lookup for name .<0x20> namecache_store: storing 5 addresses for .#20: [],[],[],[], sitename_fetch: Returning sitename for realm ‘’: “” namecache_fetch: name .#20 found. Connecting to at port 445 socket options: SO_KEEPALIVE=0, SO_REUSEADDR=0, SO_BROADCAST=0, TCP_NODELAY=1, TCP_KEEPCNT=9, TCP_KEEPIDLE=7200, TCP_KEEPINTVL=75, IPTOS_LOWDELAY=0, IPTOS_THROUGHPUT=0, SO_REUSEPORT=0, SO_SNDBUF=87040, SO_RCVBUF=131072, SO_SNDLOWAT=1, SO_RCVLOWAT=1, SO_SNDTIMEO=0, SO_RCVTIMEO=0, TCP_QUICKACK=1, TCP_DEFER_ACCEPT=0, TCP_USER_TIMEOUT=0 cli_session_setup_spnego_send: Connect to . as @ using SPNEGO GENSEC backend ‘gssapi_spnego’ registered GENSEC backend ‘gssapi_krb5’ registered GENSEC backend ‘gssapi_krb5_sasl’ registered GENSEC backend ‘spnego’ registered GENSEC backend ‘schannel’ registered GENSEC backend ‘ncalrpc_as_system’ registered GENSEC backend ‘sasl-EXTERNAL’ registered GENSEC backend ‘ntlmssp’ registered GENSEC backend ‘ntlmssp_resume_ccache’ registered GENSEC backend ‘http_basic’ registered GENSEC backend ‘http_ntlm’ registered GENSEC backend ‘http_negotiate’ registered Starting GENSEC mechanism spnego Starting GENSEC submechanism gse_krb5 gensec_gse_client_prepare_ccache: No kinit required for @ to access cifs/., KEYRING:persistent:0:krb_ccache_slwxWDf signed SMB2 message (sign_algo_id=2) signed SMB2 message (sign_algo_id=2) Bind RPC Pipe: host . auth_type 0, auth_level 1 rpc_api_pipe: host . signed SMB2 message (sign_algo_id=2) rpc_read_send: data_to_read: 76 check_bind_response: accepted! rpc_api_pipe: host . signed SMB2 message (sign_algo_id=2) rpc_read_send: data_to_read: 32 rpc_api_pipe: host . signed SMB2 message (sign_algo_id=2) rpc_read_send: data_to_read: 220 rpc_api_pipe: host . signed SMB2 message (sign_algo_id=2) rpc_read_send: data_to_read: 32 signed SMB2 message (sign_algo_id=2) sitename_fetch: Returning sitename for realm ‘’: “” namecache_fetch: name .#20 found. ads_try_connect: ads_try_connect: sending CLDAP request to (realm: ) Successfully contacted LDAP server Connecting to at port 389 Connected to LDAP server . ads_current_time: server time offset is 0 seconds ads_current_time: server time offset is 0 seconds Starting GENSEC mechanism spnego Starting GENSEC submechanism gse_krb5 gensec_gse_client_prepare_ccache: No kinit required for @ to access ldap/., KEYRING:persistent:0:krb_ccache_slwxWDf ads_gen_add: AD LDAP: Adding cn=, libnet_join_precreate_machine_acct: Machine account successfully created ads_gen_mod: AD LDAP: Modifying CN=, ads_domain_func_level: 7 sitename_fetch: Returning sitename for realm ‘’: “” namecache_fetch: name .#20 found. ads_try_connect: ads_try_connect: sending CLDAP request to (realm: ) Successfully contacted LDAP server Connecting to at port 389 Connected to LDAP server . ads_current_time: server time offset is 0 seconds ads_current_time: server time offset is 0 seconds Starting GENSEC mechanism spnego Starting GENSEC submechanism gse_krb5 gensec_gse_client_prepare_ccache: Doing kinit for $@ to access ldap/. into MEMORY:4cuLN1j gensec_gse_client_prepare_ccache: Kinit for $@ to access ldap/. failed: Cannot find KDC for requested realm: NT_STATUS_NO_LOGON_SERVERS Failed to start GENSEC client mech gse_krb5: NT_STATUS_INVALID_PARAMETER Starting GENSEC submechanism ntlmssp ads_sasl_spnego_bind: ads_sasl_spnego_gensec_bind() failed for ldap/. with user[$@]: Invalid credentials libnet_Join: libnet_JoinCtx: struct libnet_JoinCtx out: struct libnet_JoinCtx odj_provision_data : NULL account_name : ‘$’ netbios_domain_name : ‘AD’ dns_domain_name : ‘’ forest_name : ‘’ dn : ‘CN=,’ domain_guid : domain_sid : * domain_sid : modified_config : 0x00 (0) error_string : ‘failed to connect to AD: Invalid credentials’ domain_is_ad : 0x01 (1) set_encryption_types : 0x00000000 (0) krb5_salt : NULL dcinfo : * dcinfo: struct netr_DsRGetDCNameInfo dc_unc : * dc_unc : ‘\.’ dc_address : * dc_address : ‘\’ dc_address_type : DS_ADDRESS_TYPE_INET (1) domain_guid : domain_name : * domain_name : ‘’ forest_name : * forest_name : ‘’ dc_flags : 0xe003f1fc (3758354940) 0: NBT_SERVER_PDC 1: NBT_SERVER_GC 1: NBT_SERVER_LDAP 1: NBT_SERVER_DS 1: NBT_SERVER_KDC 1: NBT_SERVER_TIMESERV 1: NBT_SERVER_CLOSEST 1: NBT_SERVER_WRITABLE 0: NBT_SERVER_GOOD_TIMESERV 0: NBT_SERVER_NDNC 0: NBT_SERVER_SELECT_SECRET_DOMAIN_6 1: NBT_SERVER_FULL_SECRET_DOMAIN_6 1: NBT_SERVER_ADS_WEB_SERVICE 1: NBT_SERVER_DS_8 1: NBT_SERVER_DS_9 1: NBT_SERVER_DS_10 1: NBT_SERVER_HAS_DNS_NAME 1: NBT_SERVER_IS_DEFAULT_NC 1: NBT_SERVER_FOREST_ROOT dc_site_name : * dc_site_name : ‘’ client_site_name : * client_site_name : ‘’ account_rid : 0x00009ca7 (40103) result : WERR_GEN_FAILURE return code = -1 Freeing parametrics:
It took a while to digest what was going on. I thought it was a kerberos error. I found logs and desynced DCs that wasn’t showing up with dcdiag with verbose and full testing parameters. I even found an IP overlap in ipv6, oops!
After spinning up new DCs and poking at them till they managed to sync from the old ones, the old ones were decomissioned. Tried joining again, NO DICE!
And then those two started desyncing. Rolled up another two which stayed in sync.
STILL COULDNT JOIN!
And the other truenas fileserver started saying its kerberos ticket expired, and when I tried to fix that, it blasted me with the error too!
I had tried 25.04.2.6 and 25.10.1.
What happened then was going down a huge rabbit hole of truenas logs, event logs, etc. Everything was pointing to some kind of malfunction in kerberos… except NTLM logs.
I have NTLM auditing on, even though all NTLM is blocked, both in domain and to the DCs themselves. Well, the NTLM logs had complaints about the DC trying to auth against itself, every minute. This was confusing.
However, years ago I learned that when NTLM is disabled, other microsoft services can malfunction too. DFS Namespace management was one of those.
So I dive into GPOs, reenable NTLM to test, use repadmin /syncall and gpupdate on each domain controller, then test join. Success! Can join to the domain though it makes no sense because we’re supposed to be using kerberos (???)
I tinker with the GPOs for a while and determine which one it is. There are mainly two GPOs to apply to domain controllers:
Network security: Restrict NTLM: Incoming NTLM traffic (the culprit)
Network security: Restrict NTLM: NTLM authentication in this domain
The first one blocks any attempts to use NTLM authentication to the domain controller itself, like trying to log on. When it was set to DENY ALL, joining with truenas failed. Set to DENY DOMAIN ACCOUNTS, it works for some reason???
The second, NTLM auth in this domain, controls if someone can use NTLM authentication with accounts or to services on machines OTHER THAN the domain controllers. This is still set to DENY ALL.
This used to not be an issue in prior versions of truenas. The last known working was truenas 24.–. This also could be a side effect of some windows server 2022 patch microsoft released. I don’t know why this is a problem now. What I do know is that I did this test multiple times with different settings, it was always Network security: Restrict NTLM: Incoming NTLM traffic set to DENY ALL on domain controllers that broke truenas joining.
Hopefully this helps anyone who is having problems.