Can not do test replication from Core to Scale

TrueNAS Replication Fails Due to Dataset Unmounting Issue

ElectricEel-24.10.1
TrueNAS-13.3-U1

Hello TrueNAS Community,

I have just built a new nas, now i have a core and scale. The Scale is all new. I have just spent 2 days trying to get the dam ssh connection via key-pair to work, had to be massively tinkered with manually but i have got an actual connection, but now i cant send the most simple of replications !!!

I’m currently struggling with a replication setup between my TrueNAS CORE and SCALE systems. Despite days of troubleshooting, I keep encountering some error’s that I can’t resolve:

  • Error:
    • Passive side: cannot send 'NAS-main/mandie-home': I/O error
    • Active side: cannot unmount '/mnt/basepool/mandie-home': permission denied.

Setup Details:

  1. Source System: TrueNAS CORE

    • Dataset: NAS-main/mandie-home
    • Replication Task: Configured to send NAS-main/mandie-home to basepool/mandie-home on SCALE.
  2. Destination System: TrueNAS SCALE

    • Pool: basepool
    • I initially tried replicating directly to basepool without creating a sub-dataset (basepool/mandie-home), but a different error occurred: the replication task tried to overwrite the entire pool, - Passive side: cannot send 'NAS-main/mandie-home': I/O error Active side: cannot unmount '/var/db/system/nfs': permission denied. which was not the intended behavior.
    • Then, I manually created the dataset basepool/mandie-home to specify a target sub-dataset explicitly.
  3. Replication Task Configuration:

    • Direction: PUSH
    • Transport: SSH+NETCAT
    • Destination Dataset: basepool/mandie-home
    • Replication from Scratch: Enabled
    • Include Dataset Properties: Enabled
    • Snapshot Retention Policy: Same as Source
  4. System Dataset Placement:

    • The .system dataset is on basepool because it is my only pool on SCALE.

Problem:

The replication task attempts to unmount the destination dataset (basepool/mandie-home) on SCALE during the process, but it fails with a “permission denied” error.

* Replication "NAS-main/mandie-home - basepool" failed: Passive side: cannot
send 'NAS-main/mandie-home': I/O error Active side: cannot unmount
'/mnt/basepool/mandie-home': permission denied..

I’ve tried the following without success:

  1. Ensured truenas_admin (replication user) has passwordless sudo permissions.
  2. Manually unmounted the dataset (sudo zfs unmount basepool/mandie-home), but it remounts automatically.
  3. Set canmount=off for basepool/mandie-home, yet the same error persists.
  4. Excluded .system from replication by ensuring only mandie-home is included as the source.

What I’ve Tried to Achieve:

I want the contents of NAS-main/mandie-home on CORE to replicate directly into basepool/mandie-home on SCALE. I do not want an additional nested structure like basepool/mandie-home/mandie-home.

Questions:

  1. Why is the replication process trying to unmount the destination dataset (basepool/mandie-home)?
  2. Is there a way to configure TrueNAS SCALE to allow replication without unmounting the destination dataset?
  3. Should I relocate the .system dataset on SCALE to prevent interference, and if so, how should I proceed given I only have one pool?

Additional Information:

I’ve attached screenshots of my replication task and configuration:

Any help or guidance would be greatly appreciated. I’ve spent several days on this issue, and I’m hoping someone can shed some light on what I might be missing.

Thank you in advance for your time and assistance!

Best regards,
Jack

Append:

I can do it with the same user via ssh without error.

jack@TruenasCore ~ $ sudo zfs send NAS-main/mandie-home@NAS-main-12hours-auto-2024-12-29_00-30 | ssh -i ~/.ssh/rep_key truenas_admin@192.168.1.117 sudo zfs receive -F basepool/mandie-home
Password:
jack@TruenasCore ~ $ 
truenas_admin@TruenasScale ~ $ sudo zfs list -t snap basepool/mandie-home
NAME                                                          USED  AVAIL  REFER  MOUNTPOINT
basepool/mandie-home@NAS-main-12hours-auto-2024-12-29_00-30     0B      -  1.66G  -
truenas_admin@TruenasScale ~ $ ls -al /mnt/basepool/mandie-home
total 358
drwxrwxrwx 6 1001 root    15 Nov  1 00:19  .
drwxr-xr-x 3 root root     3 Dec 29 11:50  ..
drwxr-xr-x 4 1001 root     4 May  2  2024  .Trash-1000
-rwxrwxrwx 1 1001 1002   962 Jan 16  2023  .cshrc
-rwxrwxrwx 1 1001 1002   323 Jan 16  2023  .login
-rwxrwxrwx 1 1001 1002    91 Jan 16  2023  .login_conf
-rwxrwxrwx 1 1001 1002   301 Jan 16  2023  .mail_aliases
-rwxrwxrwx 1 1001 1002   267 Jan 16  2023  .mailrc
-rwxrwxrwx 1 1001 1002   978 Jan 16  2023  .profile
-rwxrwxrwx 1 1001 1002   695 Jan 16  2023  .shrc
drwxr-xr-x 3 1001 root     6 Nov  9  2023  Epim
drwxr-xr-x 4 1001 root     6 Dec 28 18:17  Epim2
drwxr-xr-x 5 1001 root   388 Nov  1 17:30  Music
-rw-r--r-- 1 1001 root  6631 Jun 11  2022  Space.gif
-rw-r--r-- 1 1001 root 91378 Jan 16  2023 's-l500 (2).jpg'
truenas_admin@TruenasScale ~ $ 

No problem there. so what have i got wrong in the UI?

I tried to pull it from the scale system but that failed to??

[EFAULT] Passive side: cannot unmount '/mnt/.ix-apps/docker': pool or dataset is busy Active side: cannot send 'NAS-main/mandie-home': I/O error.

Iv got to send my old nas data over to the new scale, im starting to wonder how im actually going to do that?

I seen mention of push and pull but how do i push from core was that not what i already tried above?