Use FreeIPA/RedHat IdM for Auth in TrueNAS with Kerberos & LDAP for NFSv4

I am working on a proof of concept for integrating TrueNAS with FreeIPA user and host management. The goal is to get Kerberos Auth working with NFSv4. For that, LDAP is needed to get the FreeIPA users and Kerberos for ticket authentication. After extensively researching this, I was unable to find any forum post or blog entry here or anywhere on the internet on how to fully configure this and get it working. So, first of all, is this currently even possible with TrueNAS SCALE 24.10?

Since I am not allowed to post links or images, please excuse the obfuscated urls.
General Info about my setup (all hostnames have been changed):
Kerberos Realm: MYREALM (dot) EXAMPLE (dot) COM
TrueNAS SCALE Server: truenas (dot) srv (dot) example (dot) com
FreeIPA Server: ipa (dot) srv (dot) example (dot) com

My configuration steps:

Create a new Host on the FreeIPA Server and create the corresponding keytab file for TrueNAS:

sudo ipa host-add truenas.srv.example.com
sudo ipa-getkeytab -p host/truenas.srv.example.com -k truenas.keytab

Then I uploaded that keytab file after configuring the other required Kerberos Settings:

imgur (dot) com/u1qmAFa

Some information I found online said to add a couple of lines to the “Libdefaults Auxiliary Parameters” which is how I ended up with the following config:

default_realm = MYREALM.EXAMPLE.COM
rdns = no
dns_lookup_kdc = true
dns_lookup_realm = false

Now to the issue I am facing: the LDAP config. On the FreeIPA Server I ran the command ldapmodify -x -D 'cn=Directory Manager' -W to create the system LDAP user according to the official documentation (www (dot) freeipa (dot) org/page/HowTo/LDAP). This command is for entering the LDAP cli and will ask for the (LDAP) admin password set during FreeIPA installation. After entering it and pressing enter, write the lines below. Make sure to use a new password for that user. Then press enter twice until the “adding new entry” line is printed. After that, exit that cli by pressing Ctrl+D. The full terminal log should look like this:

~$ sudo ldapmodify -x -D 'cn=Directory Manager' -W
Enter LDAP Password:
dn: uid=system,cn=sysaccounts,cn=etc,dc=myrealm,dc=example,dc=com
changetype: add
objectclass: account
objectclass: simplesecurityobject
uid: system
userPassword: <new_password_here>
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0

adding new entry "uid=system,cn=sysaccounts,cn=etc,dc=myrealm,dc=example,dc=com"

^D
~$

To my understanding, the advanced LDAP Settings in TrueNAS are there to use Kerberos to authenticate with the FreeIPA LDAP Server. Is that correct? If so, there appears to be an issue with this. The following is my configuration and the error I get when saving the configuration. For simplicity, I am using LDAP unencrypted, but in production it should be LDAPS.

imgur (dot) com/gzfCADe

Either there is an issue with TrueNAS when retrieving the keytab file or something similar. Or TrueNAS doesn’t correctly use the Keytab file to authenticate with FreeIPA.

The following is the error output when saving the LDAP config in TrueNAS:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kerberos.py", line 417, in do_kinit
    gss_acquire_cred_principal(
  File "/usr/lib/python3/dist-packages/middlewared/utils/directoryservices/krb5.py", line 251, in gss_acquire_cred_principal
    cr = gssapi.Credentials(
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/gssapi/creds.py", line 77, in __new__
    res = cls.acquire(name, lifetime, mechs, usage,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/gssapi/creds.py", line 173, in acquire
    res = rcred_cred_store.acquire_cred_from(b_store, name,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "gssapi/raw/ext_cred_store.pyx", line 161, in gssapi.raw.ext_cred_store.acquire_cred_from
gssapi.raw.exceptions.MissingCredentialsError: Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2529639068): Cannot contact any KDC for realm 'MYREALM.EXAMPLE.COM'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 488, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 533, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 671, in do_update
    await self.ldap_validate(old, new, verrors)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 499, in ldap_validate
    await self.validate_credentials(data)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 756, in validate_credentials
    await self.kinit(ldap_config)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 745, in kinit
    await self.middleware.call('kerberos.do_kinit', {'krb5_cred': cred})
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1626, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1468, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1361, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kerberos.py", line 430, in do_kinit
    raise KRB5Error(
middlewared.utils.directoryservices.krb5_error.KRB5Error: [KRB5_KDC_UNREACH] Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2529639068): Cannot contact any KDC for realm 'MYREALM.EXAMPLE.COM'

The first Traceback is in my eye the most relevant one since that is the first error that occurred. It states gssapi.raw.exceptions.MissingCredentialsError: Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2529639068): Cannot contact any KDC for realm 'MYREALM.EXAMPLE.COM'. I don’t understand why, because the correct LDAP user and password are provided, as well as the required keytab.

Here is the generated kerberos config /etc/krb5.conf from TrueNAS.

[libdefaults]
        default_realm = MYREALM.EXAMPLE.COM
        dns_lookup_realm = false
        forwardable = true
        default_ccache_name = KEYRING:persistent:%{uid}
        dns_lookup_kdc = false
        dns_canonicalize_hostname = false
        udp_preference_limit = 0

[appdefaults]

[realms]
        MYREALM.EXAMPLE.COM = {
                default_domain = MYREALM.EXAMPLE.COM
                kdc = ipa.srv.example.com:389
        }

[domain_realms]
        myrealm.example.com = MYREALM.EXAMPLE.COM
        .myrealm.example.com = MYREALM.EXAMPLE.COM
        MYREALM.EXAMPLE.COM = MYREALM.EXAMPLE.COM
        .MYREALM.EXAMPLE.COM = MYREALM.EXAMPLE.COM

What I am confused about is, why does the kbc value under [realms] has the port 389? That is the LDAP port. Why is this set in the kerberos config? Shouldn’t it be port 88? The only setting where a port was configured is in the LDAP screen and if I remove the port and only specify the FreeIPA hostname, the same error as shown above occurs, but the /etc/krb5.conf file stays the same.

When I manually edit the /etc/krb5.conf file to port 88 or remove the port, I get the following error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 911, in __start
    await self.ipa_kinit(ipa_config, ldap['bindpw'])
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 876, in ipa_kinit
    await self.middleware.call('kerberos.do_kinit', {
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1626, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1468, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1361, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kerberos.py", line 447, in do_kinit
    raise CallError('Password is required')
middlewared.service_exception.CallError: [EFAULT] Password is required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 488, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 533, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 49, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 179, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 681, in do_update
    await self.__start(job, ds_type)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/ldap.py", line 932, in __start
    if not err.err_msg.startswith('[KRB5_REALM_UNKNOWN]'):
           ^^^^^^^^^^^
AttributeError: 'CallError' object has no attribute 'err_msg'

The error is middlewared.service_exception.CallError: [EFAULT] Password is required which is also confusing for the same reason as the other error above. Why is a password required when I have the keyfile? Either way, this is no permanent fix, since the file is regularly re-written.

I would very much appreciate some help with this!

We are also struggling with this. Any progress?