Broken HA state: 25.10.2.1 > 25.10.3.1

TrueNAS General
1

TLDR: Anyone able to assist of why Controller B cannot talk to Controller A after 25.10.2.1>25.10.3.1 update?

I have an X20 dual controller unit that was running in HA mode rather beautifully on 25.10.2.1 other than some occasional active directory sync issues I was trying to resolve. The update channel was giving me a warning that 25.10.2.1 was an early adopter release and an update to 25.10.3.1 is a recommended action.

Controller B was the active controller (with HA enabled and healthy) when update was initiated via the web interface. Everything seemed to download well and pass pre-checks and while pinging both controllers, controller A stopped pinging during the update installation. After some minutes, it began pinging again. 20+ minutes later, it seems controller A is missing according to the web interface.

I SSH’d into controller A and it seemed as though things may have been working from the CLI perspective of things but my only obvious action seemed to be to try a good ol $sudo reboot.

Controller A rebooted and still ping responsive and still able to ssh into it but the web interface of Controller B claims it eventually timed out waiting for Controller A to update and reboot, controller A appears offline to controller B.

Controller A ssh interface: I entered “cli” from the linux shell > menu > then navigated to network configurations to be sure everything is looking normal among my fairly simple adapter configurations. It is. It is also showing that it is running 25.10.3.1, a seemingly successful OS upgrade.

I figured it’d be worth a shot to simply roll controller A back to 25.10.2.1 via the CLI. When trying to do that via command

system bootenv

I get: Namespace bootenv not found

Therefore, I am unable to change me boot environment back to the previous version. In linux cli rummaging around zfs commands, I can see my boot pool volumes for 25.10.2.1 do still exist.

Why such a failure? Anyone able to assist of why Controller B cannot talk to Controller A after 25.10.2.1 to 25.10.3.1 update?

When attempting to manually “sync to peer and reboot peer” in HA settings, I get this:

`Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/plugins/failover_/remote.py”, line 332, in call_remote
return self.CLIENT.call(method, *args, **opts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/failover_/remote.py”, line 197, in call
raise CallError(‘Remote connection unavailable’, errno.ECONNREFUSED)
middlewared.service_exception.CallError: [ECONNREFUSED] Remote connection unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py”, line 361, in process_method_call
result = await method.call(app, id_, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py”, line 57, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 956, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 784, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 667, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 116, in wrapped
result = func(*args)
^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/failover.py”, line 377, in sync_to_peer
self.middleware.call_sync(‘failover.datastore.send’)
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1083, in call_sync
return self.run_coroutine(self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1115, in run_coroutine
return fut.result()
^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/_base.py”, line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/_base.py”, line 401, in __get_result
raise self.exception
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 667, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/failover/datastore.py", line 87, in send
token = self.middleware.call_sync(‘failover.call_remote’, ‘auth.generate_token’, [
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1083, in call_sync
return self.run_coroutine(self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1115, in run_coroutine
return fut.result()
^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/_base.py”, line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/_base.py”, line 401, in __get_result
raise self.exception
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 667, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/failover/remote.py", line 336, in call_remote
raise CallError(str(e), e.errno)
middlewared.service_exception.CallError: [ECONNREFUSED] [ECONNREFUSED] Remote connection unavailable`

2

Is this a TrueNAS Enterprise system with support? HA is only supported in the Enterprise edition

3

No support contract, it has a valid enterprise HA license active. Reaching out to the community for help.

4

Serial’d into controller A in x86 mode to see the bootloader. Switched to 25.10.2.1.

Boots up, pings, Controller B web interface acts like A is not there. Otherwise running all other services well. Still cannot sync to peer/anything HA.

Web interface of controller A allows me to attempt login (so does booting 25.10.3.1) but gives me:

image
image1748×478 23.6 KB

5

Hi Steve, can you please share the full output of the command hactl? Run the command from serial on both controllers please.

6

Obviously, we’d prefer to support you under a contract if you need HA. If it’s something simple, we’ll let you know, but no guarantees. Our customer support/sales team can recommend a refresh/renewal if there is a budget.

Primary observation is that HA depends very much on networking being very solid…i’d suggest starting with outlining the network set-up. Has it changed? Do you have solid connectivity between controllers…

If HA is too complex to maintain well, the HA systems can be configured back to a single controller with a convenient cold spare.

7

Thanks for chiming in. I didn’t see this response to see what the output of hactl is before I took other actions.

Poking around in the Truenas CLI prompt of the limbo A controller, I was able to get HA status claiming “no pong” from peer, etc.

On the controller B healthily still on 25.10.2.1 (pending 25.10.3.1 hasn’t been rebooted yet) this morning, web interface complained of controller A being unable to sync, peer is missing.

I SSH’d back into controller A and toyed around. Was double-checking network/nic configurations and everything looked absolutely perfect. I tried simply clearing out and re-applying network configurations of NICs but the CLI network menu complained of HA being enabled. - HA was currently disabled on controller B, I figure it’s probably a good idea to get them both in the same sate.

Digging, I found command: sudo midclt call failover.update ‘{“disabled”: true}’
from old truenas community post

Tried that.

Controller A, the one that’s been stuck in limbo (still manually override booted back to 25.10.2.1) suddenly became primary controller as controller B is rebooting. This obviously means the controllers ARE able to talk to one another. This also means that controller B is going to now boot up into the new pending 25.10.3.1 OS. Meanwhile, SMB and NFS services and data was perfectly accessible via controller A. That’s a good sign. Web interface was allowing me to sign in and see what’s going on, as well. It stated that it was waiting on controller B, pool, datasets, disks all looked healthy.

Upon B booting up, everything was fine other than they’re now running different OS releases, interface wasn’t complaining about that, though. I knew that was wrong so, as everything seemed healthy, I then told controller A to reboot to revert back to its primary 25.10.3.1 OS. B took primary during this process.

While B took primary, I lost access to SMB and NFS shares. Web interface showed no pool, no datasets. It showed all disks as healthy and my pool as exported. I had no option to import the pool as there was no existing pool to import the “exported pool” disks into.

I waited for controller A to fire up. I then SSH’d into A and in linux CLI, I could see that the pool and datasets are missing on that controller too, boot pool is fine, obviously. Failed over to A to jump into its web interface, web interface still showed no options to import the exported pool, still no access to any data. I took no action of pool/ZFS functions. I didn’t want to risk having to move/restore a ton of data.

I instead decided to simply import my exported config file that I exported just prior to beginning the update to 25.10.3.1.

**Both controllers then immediately rebooted. Both controllers booted back up into 25.10.3.1 and boom, pool and datasets are back and everything appears to be healthy. Tested interface invoked failovers back and forth, tested cable-pulling failovers, all working good. Rebooted both controllers several times. Everything seems fine.
**
Why this update was goofy? Don’t know. Previous release updates went fine.

1 Like