Pls add bsdtar / libarchive to SCALE to ease/improve CORE migration

Feature Requests
1

Problem/Justification

I am currently migrating data from an old CORE host to a new SCALE host.

I only migrating data, no hardware.

Thanks my network topology, the most efficient way of doing this is via

tar | zstd | netcat -> netcat | zstd | tar

CORE includes only bsdtar (presumably from libarchive), whereas SCALE includes only GNUtar.

Impact

These two tar implementations do not like each other’s tar format and in my scenario GNUtar will spew many warnings. While I can suppress/ignore them, I strongly prefer to have the assurance that the transfer really went flawlessly and that I see any proper errors / warnings that happen during large multi-hour transfers.

Tools that reduce migration friction and increase migration-related peace of mind are important. Since all CORE users are being encouraged to migrate to SCALE, some, like me, will take this as an opportunity to upgrade hardware.

User Story

See above.

While ssh’ing into a truenas box to complete tasks isn’t encouraged, manually installing packages is actively discouraged. Reasonably so. But that leaves me with no easy way to add bsdtar to SCALE.

Given this, and that there’s also reasonably no toolchain installed that would let me buid bsdtar, one might argue that this is a CORE ↔ SCALE data portability issue.

The cost of adding bsdtar / libarchive seems like a very small one to pay given the upsides.

Thanks for considering.

2

I think you need to explain why you aren’t using more, conventional methods of data migration between ZFS systems. Rsync, ZFS snapshots and replication, SMB…

2 Likes
3

Why not just import the pool?

You said this.

If you can’t import the pool in place because you want to copy everything to a new pool, @SmallBarky listed a few common ways to do it.

How do you think everyone else is able to switch to SCALE?

1 Like
4

Would be zfs replication… surely?

1 Like
5

Based on my measurements, rsync isn’t as performant given that I’m moving large files, and I’m not planning to do this more than once. rsync is great for ongoing incremental updates but not the most efficient otherwise.

I suppose I could SMB mount the target on the source; that’s a fair point. I’ll run a performance comparison to see how well it does in terms of throughput given that I’m pushing several TB of data around and the source host is gigabit-only.

Unless I’m misunderstanding how they work, using snapshots and replication implies that I want to keep all my per-pool or per-dataset organization intact. I don’t. This is a great opportunity for me to reorganize.

I’m absolutely fascinated by the amount of You’re Holding It Wrong commentary the FR getting. At no point did I suggest that it’s impossible to do via other means, and for folks who can do the migration that way, their needs are met.

Given that the incompatibility between GNUtar and bsdtar is long-standing and well-known, it’s also surprising to me that this hasn’t bitten anyone yet.

I’ve used TrueNAS for well over a decaide precisely because it’s not Synology or any of the other turnkey options that significantly constrain the solution space for a given problem. Wwhen I need to do something in a way that’s familiar to me, I can pop into the shell and solve the issue quickly.

6

For a one time migration? How so?

How is it less efficient than piping through bsdtar?

This is a one time migration?

After the replication finishes, you can reorganize your files and folders however you like on the new server.

7

Because you are truly asking a wrong question, or not disclosing all pertinent details.

The fastest way to transfer data to/from zfs is by sending entire fielsystem, never file by file.

Explain why this does not work for you.

Fut

What prevents you from copying/installing/getting whatever tools you need wherever you need them? On core you can install whatever you want with pkg

You woudl need to enable remote repository, and then disable it back, and make sure that you don't update existing packages and don't install anythign that will mess up dependencies of existing packages -- otherwise the world will break. 
sed -i .orig 's/enabled: yes/enabled: no/' /usr/local/etc/pkg/repos/local.conf
sed -i .orig 's/enabled: no/enabled: yes/' /usr/local/etc/pkg/repos/FreeBSD.conf

Better yet – install what you need in jail.

Either way, it’s a moot point. Do zfs send | zfs receive. see man zfs-send for details.

8

Well, when TrueNAS includes more than one way to transfer the data, but you don’t want to use any of them and instead come up with your own, which then doesn’t work acceptably for entirely foreseeable[1] reasons, and you suggest the product should change to accommodate your one-off custom solution, yes, you’re going to get that. You’re free to develop your own solutions, of course, but if they don’t work (or work as desired) with the products as they are, they aren’t very good solutions, are they?

  1. Yes, CORE uses bsdtar, because it’s FreeBSD. SCALE/CE uses GNU tar, because it’s GNU/Linux. All of this is exactly as should have been expected. It’s unfortunate that GNU tar and BSD tar have diverged, but it’s been well-known for quite some time, as also with grep and several other tools. ↩︎