[KRB5KDC_ERR_PREAUTH_FAILED] Errors on AD quite often

TrueNAS General
24

thanks for the post, i just hit this on my machine, this was helpful as i failed to rejoin using command line or mdictl

one change to @GlendonKuhns instructions it is remove the machine account from the field ‘kerberos principle’ that enables the rejoin UI elements, there is no field called keytab

I also did a kdestroy for good measure at the command line before rejoining

possible causes?

the only thing i can think of is the kerberos keys are expiring and not being renewed

I also changed my administrator password about 7 days ago - not sure if thats cached to renew trhe ticket

co-incidentally i was learning how to make a debian VM join a domain so it could update DNS - one of things i had to do was make a systemd unit.service to run kinit to refresh the kerberos tokens every 4 hours… i wonder how truenas does it and that is where this is all failing - the tickets are not being renewed, expired, then we are borked…?

this was the error i was getting when i just toggled the service

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kerberos.py", line 418, in do_kinit
    gss_acquire_cred_principal(
  File "/usr/lib/python3/dist-packages/middlewared/utils/directoryservices/krb5.py", line 261, in gss_acquire_cred_principal
    cr = gssapi.Credentials(
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/gssapi/creds.py", line 77, in __new__
    res = cls.acquire(name, lifetime, mechs, usage,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/gssapi/creds.py", line 173, in acquire
    res = rcred_cred_store.acquire_cred_from(b_store, name,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "gssapi/raw/ext_cred_store.pyx", line 161, in gssapi.raw.ext_cred_store.acquire_cred_from
gssapi.raw.exceptions.MissingCredentialsError: Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2529638936): Preauthentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/activedirectory.py", line 465, in do_update
    await self.validate_credentials(new, domain_info['KDC server'])
  File "/usr/lib/python3/dist-packages/middlewared/plugins/activedirectory.py", line 691, in validate_credentials
    await self.middleware.call('kerberos.do_kinit', {
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1000, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 726, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 619, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 178, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/kerberos.py", line 431, in do_kinit
    raise KRB5Error(
middlewared.utils.directoryservices.krb5_error.KRB5Error: [KRB5KDC_ERR_PREAUTH_FAILED] Major (458752): No credentials were supplied, or the credentials were unavailable or inaccessible, Minor (2529638936): Preauthentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 515, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 560, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 174, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 48, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/activedirectory.py", line 540, in do_update
    raise ValidationError(key, msg)
middlewared.service_exception.ValidationError: [EINVAL] activedirectory.kerberos_principal: Kerberos principal credentials are no longer valid. Rejoining active directory may be required.

IMO there is 100% a bug here in keeping keys refreshed and upto date… and if this is because the admin crednetials are being used to kinit - that is the wrong way to have designed this, admin creds should be used at first join to get initial tickets, do kinit for the machine account, thereafter kinit can be called to renew the machine account certs without needing the AD admin credentials ever again