Can not do test replication from Core to Scale

TrueNAS General
1

TrueNAS Replication Fails Due to Dataset Unmounting Issue

ElectricEel-24.10.1
TrueNAS-13.3-U1

Hello TrueNAS Community,

I have just built a new nas, now i have a core and scale. The Scale is all new. I have just spent 2 days trying to get the dam ssh connection via key-pair to work, had to be massively tinkered with manually but i have got an actual connection, but now i cant send the most simple of replications !!!

I’m currently struggling with a replication setup between my TrueNAS CORE and SCALE systems. Despite days of troubleshooting, I keep encountering some error’s that I can’t resolve:

  • Error:
    • Passive side: cannot send 'NAS-main/mandie-home': I/O error
    • Active side: cannot unmount '/mnt/basepool/mandie-home': permission denied.

Setup Details:

  1. Source System: TrueNAS CORE

    • Dataset: NAS-main/mandie-home
    • Replication Task: Configured to send NAS-main/mandie-home to basepool/mandie-home on SCALE.

  2. Destination System: TrueNAS SCALE

    • Pool: basepool
    • I initially tried replicating directly to basepool without creating a sub-dataset (basepool/mandie-home), but a different error occurred: the replication task tried to overwrite the entire pool, - Passive side: cannot send 'NAS-main/mandie-home': I/O error Active side: cannot unmount '/var/db/system/nfs': permission denied. which was not the intended behavior.
    • Then, I manually created the dataset basepool/mandie-home to specify a target sub-dataset explicitly.

  3. Replication Task Configuration:

    • Direction: PUSH
    • Transport: SSH+NETCAT
    • Destination Dataset: basepool/mandie-home
    • Replication from Scratch: Enabled
    • Include Dataset Properties: Enabled
    • Snapshot Retention Policy: Same as Source

  4. System Dataset Placement:

    • The .system dataset is on basepool because it is my only pool on SCALE.

Problem:

The replication task attempts to unmount the destination dataset (basepool/mandie-home) on SCALE during the process, but it fails with a “permission denied” error.

* Replication "NAS-main/mandie-home - basepool" failed: Passive side: cannot
send 'NAS-main/mandie-home': I/O error Active side: cannot unmount
'/mnt/basepool/mandie-home': permission denied..

I’ve tried the following without success:

  1. Ensured truenas_admin (replication user) has passwordless sudo permissions.
  2. Manually unmounted the dataset (sudo zfs unmount basepool/mandie-home), but it remounts automatically.
  3. Set canmount=off for basepool/mandie-home, yet the same error persists.
  4. Excluded .system from replication by ensuring only mandie-home is included as the source.

What I’ve Tried to Achieve:

I want the contents of NAS-main/mandie-home on CORE to replicate directly into basepool/mandie-home on SCALE. I do not want an additional nested structure like basepool/mandie-home/mandie-home.

Questions:

  1. Why is the replication process trying to unmount the destination dataset (basepool/mandie-home)?
  2. Is there a way to configure TrueNAS SCALE to allow replication without unmounting the destination dataset?
  3. Should I relocate the .system dataset on SCALE to prevent interference, and if so, how should I proceed given I only have one pool?

Additional Information:

I’ve attached screenshots of my replication task and configuration:

Any help or guidance would be greatly appreciated. I’ve spent several days on this issue, and I’m hoping someone can shed some light on what I might be missing.

Thank you in advance for your time and assistance!

Best regards,
Jack

Screenshot_20241229_115651
Screenshot_20241229_1156511053×1441 132 KB

Append:

I can do it with the same user via ssh without error.

jack@TruenasCore ~ $ sudo zfs send NAS-main/mandie-home@NAS-main-12hours-auto-2024-12-29_00-30 | ssh -i ~/.ssh/rep_key truenas_admin@192.168.1.117 sudo zfs receive -F basepool/mandie-home
Password:
jack@TruenasCore ~ $ 

truenas_admin@TruenasScale ~ $ sudo zfs list -t snap basepool/mandie-home
NAME                                                          USED  AVAIL  REFER  MOUNTPOINT
basepool/mandie-home@NAS-main-12hours-auto-2024-12-29_00-30     0B      -  1.66G  -
truenas_admin@TruenasScale ~ $ ls -al /mnt/basepool/mandie-home
total 358
drwxrwxrwx 6 1001 root    15 Nov  1 00:19  .
drwxr-xr-x 3 root root     3 Dec 29 11:50  ..
drwxr-xr-x 4 1001 root     4 May  2  2024  .Trash-1000
-rwxrwxrwx 1 1001 1002   962 Jan 16  2023  .cshrc
-rwxrwxrwx 1 1001 1002   323 Jan 16  2023  .login
-rwxrwxrwx 1 1001 1002    91 Jan 16  2023  .login_conf
-rwxrwxrwx 1 1001 1002   301 Jan 16  2023  .mail_aliases
-rwxrwxrwx 1 1001 1002   267 Jan 16  2023  .mailrc
-rwxrwxrwx 1 1001 1002   978 Jan 16  2023  .profile
-rwxrwxrwx 1 1001 1002   695 Jan 16  2023  .shrc
drwxr-xr-x 3 1001 root     6 Nov  9  2023  Epim
drwxr-xr-x 4 1001 root     6 Dec 28 18:17  Epim2
drwxr-xr-x 5 1001 root   388 Nov  1 17:30  Music
-rw-r--r-- 1 1001 root  6631 Jun 11  2022  Space.gif
-rw-r--r-- 1 1001 root 91378 Jan 16  2023 's-l500 (2).jpg'
truenas_admin@TruenasScale ~ $

No problem there. so what have i got wrong in the UI?

2

I tried to pull it from the scale system but that failed to??

image
image793×1943 145 KB

[EFAULT] Passive side: cannot unmount '/mnt/.ix-apps/docker': pool or dataset is busy Active side: cannot send 'NAS-main/mandie-home': I/O error.

Iv got to send my old nas data over to the new scale, im starting to wonder how im actually going to do that?

I seen mention of push and pull but how do i push from core was that not what i already tried above?

3

I too struggle with a vanilla install of Scale giving me these exact errors, when doing a push replication from Core. I’m still working towards a solution myself and most likely will create yet another post to the forums with this problem. Frustration with Scale is the name of the game. Core to Core works like it should. I’m about ready to ditch Scale, even in the light of Core going away. I just need this stuff to work - so I can get back to work! Let me know if you ever resolved this @jackdinn Thanks :wink:

4

Yep its a PITA, but it cant be that important to the dev’s.

No it still does the same but i use Scale as my main now, I do still use Core as a secondary backup of the Scale NAS (even though im still not 100% sure that i can even send the data from Core to Scale still). As you must know Scale to Core is fine, going back the other way is …well…err… hmm.

But thats for another day, i hope (touch wood).

Sorry i cant help. I’m in the same boat still.

5

Man, I jumped the gun in that previous post. After studying the account I was using, the truenas_admin builtin, I switched to using the root account for test. Just needed to add the pubkey. So, there are TWO accounts with privilege that are created in a vanilla install of Scale at present. Use the root account, and core to scale replication now works. No more problems experienced. No more permissions and un-mounting troubles, etc. Go figure ha!

I see so many posts, but nobody bothers to put their solutions in them? Yourself excluded as you never resolved it… thanks for following up with me my friend! I hope this helps someone along the same path… that worked for me anyway.

1 Like
6

Ok cool, if/when (hopefully never) my Scale system completely borks, i wont have to look to far for the solution.

Cheers for posting.