LDAP setup help, possible bug: says it can't connect to KDC, but I know that it actually can

I’ve set up a Kerberos server so that clients can connect securely to NFS shares, and now I’m trying to add LDAP to the mix so that I can have UID mapping instead of all users being treated as “nobody”. I’m using OpenLDAP and I populated the directory manually, so from a TrueNAS perspective this is a plain LDAP setup, not AD or IPA. The LDAP server is working fine with other clients, but when I try to connect to it using the “configure directory services” button in TrueNAS, it fails with the error:

[KRB5_KDC_UNREACH] Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM'

(I’ve replaced my actual realm name with EXAMPLE.COM here.)

The thing is, I know that TrueNAS actually is able to communicate with the KDC, and that the credentials in the keytab are correct. Clients are able to successfully mount NFS shares using sec=krb5p, and from a shell on TrueNAS, I can successfully kinit as my host/ principal using the keytab. If I copy the keytab to another machine that has ldapsearch installed (since TrueNAS doesn’t), I can successfully query LDAP with it. So I don’t think this is a connectivity or a credentials problem.

However, when I try to enable directory services on TrueNAS and it gives me that error, /var/log/middlewared.log says:

(WARNING) middlewared.process_message():325 - Private method 'kerberos.keytab.kerberos_principal_choices' called on a connection without private_methods enabled

…followed by a message with a Python stack trace that’s the source of the KRB5_KDC_UNREACH error:

Stack trace
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 527, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 574, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 670, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 667, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 116, in wrapped
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/directoryservices_/datastore.py", line 577, in update
    verrors.check()
  File "/usr/lib/python3/dist-packages/middlewared/service_exception.py", line 71, in check
    raise self
middlewared.service_exception.ValidationErrors: [EINVAL] directoryservices.update.credential.principal: [KRB5_KDC_UNREACH] Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2529639068): Cannot contact any KDC for realm 'EXAMPLE.COM'

(This is embedded in a JSON object in the log; so all the linebreaks are escaped as \n there; I’ve decoded it for readability here.)

Running tcpdump on the KDC host, I see that there’s no communication at all from the TrueNAS host when this error occurs. It seems like TrueNAS isn’t even attempting the connection, because of (apparently) an internal problem that prevents the middleware from reading the keytab for this purpose even though other things are using it successfully.

This is my first time using directory services in TrueNAS, though, so I’m not sure where to go from here. Is there a good next step for troubleshooting or debugging, or a setting somewhere that could maybe resolve this? Or does it look like a pure bug and I just have to wait for a release with a fix?

I’m using TrueNAS 25.10.6 (and first encountered this in 25.10.5 before I installed the recent update). Here are all my settings on the Directory Services page (again with the domain changed to example.com):

Kerberos Realms

Realm: EXAMPLE.COM
Primary KDC: kerberos.example.com
KDC: kerberos.example.com
Admin Servers: kerberos.example.com
Password Servers: kerberos.example.com

(I also have the standard DNS SRV records for automatically finding the KDC from the realm name, but I’ve specified the hostnames here instead of relying on that.)

Kerberos Keytab

Name: nas.example.com

For the file, I used the ktadd command in kadmin to export the keys for the host/nas.example.com and nfs/nas.example.com principals.

Directory Services Configuration
  • Configuration Type: LDAP
  • Enable Service: yes
  • Enable Account Cache: yes
  • Enable DNS Updates: no (I don’t need this, and my DNS isn’t based on LDAP anyway)
  • Timeout (seconds): 10 (note that this is for DNS lookups, and the error I get is immediate, not after a timeout)
  • Kerberos Realm: EXAMPLE.COM

Credential Configuration

  • Credential Type: Kerberos Principal
  • Kerberos Principal: host/nas.example.com@EXAMPLE.COM (from a dropdown that has the host/ and nfs/ principals from the keytab as options)

LDAP Configuration

  • Server URLs: ldap://kerberos.example.com (OpenLDAP is on the same host as the KDC)
  • Base DN: dc=example,dc=com
  • Start TLS: no (I’ll enable TLS later, but it’s not important right now)
  • Validate Certificates: no (not applicable anyway since TLS is off)
  • Schema: RFC2307bis

Auxiliary Parameters

  • Use Standard Auxiliary Parameters: yes

Search Bases

  • Use Standard Search Bases: yes

Attribute Maps:

  • Use Standard Attribute Maps: yes

After some debugging, this turned out to be a combination of two things:

First, it was using a krb5.conf that didn’t have a kdc = line. The “real” krb5.conf, the one used for normal operation, includes the hostnames that I’ve configured for the Kerberos realm (in the advanced settings), and that version is the one that existed when I tested with kinit in a shell. But for the TrueNAS middleware testing whether the host’s Kerberos credentials are valid, it uses a temporary krb5.conf that’s generated in a different way, and there’s code that figures out what KDC to use for AD and IPA domains, but for plain LDAP it does nothing for that part, so the temporary krb5.conf file doesn’t specify a KDC hostname.

Second, my DNS SRV record for the Kerberos realm was incorrect: I had set the value to just the hostname, but it needs to be the FQDN. This shouldn’t have mattered since I’d explicitly provided all the hostnames in the TrueNAS Kerberos realm settings, but since the temporary krb5.conf file doesn’t use that information, Kerberos has to get it from DNS SRV records instead, so the wrong data there meant that it couldn’t find the KDC.

The message about private_methods being disabled was a red herring: that’s just a warning message, and not the cause of the problem.

1 Like