Unagle to recover from fangtooh to EE

due to performance issues i took my primary server from fangtooh to EE. I removed everything from ground zero. trying to restore the previous config results in the following error when i try to connect EE to FT.

Access denied to replication.list_datasets

Source is required

I have tried manually setting up the ssh credentials manually and using the semi-automatic way…and pulling from ee to ft results in that error., pushing from ft to ee also fails int he same way. ideas?

going from the ft system to the ee gets me this. the username is root for both the ssh and the webadmin.

Error: Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/plugins/keychain.py”, line 563, in remote_ssh_semiautomatic_setup
client = Client(os.path.join(re.sub(“^http”, “ws”, data[“url”]), “websocket”),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/truenas_api_client/init.py”, line 94, in init
self.__client = client_class(uri, reserved_ports, py_exceptions, log_py_exceptions, call_timeout, verify_ssl)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/truenas_api_client/legacy.py”, line 210, in init
self._ws.connect()
File “/usr/lib/python3/dist-packages/truenas_api_client/legacy.py”, line 59, in connect
self.socket = connect(self.url, sockopt, proxy_info(), None)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/websocket/_http.py”, line 145, in connect
sock = _open_socket(addrinfo_list, options.sockopt, options.timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/websocket/_http.py”, line 232, in _open_socket
raise err
File “/usr/lib/python3/dist-packages/websocket/_http.py”, line 209, in _open_socket
sock.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py”, line 323, in process_method_call
result = await method.call(app, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py”, line 40, in call
result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 883, in call_with_audit
result = await self._call(method, serviceobj, methodobj, params, app=app,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 692, in call
return await methodobj(*prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 88, in wrapped
result = await func(*args)
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/keychain
/ssh_connections.py", line 74, in setup_ssh_connection
resp = await self.middleware.call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 977, in call
return await self._call(
^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 703, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 596, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.11/concurrent/futures/thread.py”, line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py”, line 96, in wrapped
result = func(*args)
^^^^^^^^^^^
File “/usr/lib/python3/dist-packages/middlewared/plugins/keychain.py”, line 568, in remote_ssh_semiautomatic_setup
raise CallError(f"Unable to connect to remote system: {e}")
middlewared.service_exception.CallError: [EFAULT] Unable to connect to remote system: [Errno 111] Connection refused

ok i have the primary back to core 13.3. now i can get them to to connect via ssh…but now naming schema is the issue. core machine pool is named Data and the backups dataset is named Backup. Now trying to get the naming schema right so i can get my data back onto the core machine.

ok i can replicate everything except my dataset claled stuff which is at /mnt/backup/stuff as it says there’s no snapshots to replicate…but there’s 1.7TB of data i need to send back…how can i get this dataset to get sent over?

finally had to create a replication tasdk from the fangtooth mchine to the core machine jsut for the stuff dataset…and that one is running now alongside the pull from the fangtooth to core. once i get everything back to core i will be redoing the secondary mahcine back to core…:slight_smile: