Problem migrating data from QNAP QuTS host

TrueNAS General
,
1

Hi,

I’m trying to migrate a bunch of data from a QNAP NAS that is running QuTS Hero (based on ZFS) to TrueNAS SCALE (running TrueNAS SCALE 24.04.2.2). I have done a zfs send on snapshots from the QNAP with:

zfs send -v zpool1/zfs1@migrate3 > zfs1.snapshot
zfs send -v zpool1/zfs2@migrate3 > zfs2.snapshot

When I try and receive the first snapshot into TrueNAS (using a test pool with just one device for now) I get:

# zfs receive -dv temp-pool/restore < zfs1.snapshot
receiving full stream of zpool1/zfs1@migrate3 into temp-pool/restore/zfs1@migrate3
cannot open 'temp-pool/restore/zfs1': dataset does not exist
cannot receive new filesystem stream: dataset does not exist

Maybe I have misunderstood the -d parameter here, but I expected that to create the dataset for me? No problem I thought, so I did:

# zfs create temp-pool/restore/zfs1
# zfs receive -d temp-pool/restore < zfs1.snapshot
cannot receive new filesystem stream: destination 'temp-pool/restore/zfs1' exists
must specify -F to overwrite it

Hmmm…

# zfs receive -Fd temp-pool/restore < zfs1.snapshot
cannot receive new filesystem stream: dataset does not exist

OK, maybe there’s an issue with that snapshot. Let’s try the 2nd one:

# zfs receive -d temp-pool/restore < zfs2.snapshot
cannot receive new filesystem stream: invalid backup stream

I’ve tried using zstreamdump on both files and they seems OK:

# zstreamdump zfs1.snapshot
BEGIN record
        hdrtype = 1
        features = 4
        magic = 2f5bacbac
        creation_time = 671143b5
        type = 2
        flags = 0x4
        toguid = d514a22751804d40
        fromguid = 0
        toname = zpool1/zfs1@migrate3
        payloadlen = 0
END checksum = 1b596bb27ecc14d/b67978c902fbec9c/9c38850172e55d54/45db71b2243a1710
SUMMARY:
        Total DRR_BEGIN records = 1 (0 bytes)
        Total DRR_END records = 1 (0 bytes)
        Total DRR_OBJECT records = 1589 (268944 bytes)
        Total DRR_FREEOBJECTS records = 327 (0 bytes)
        Total DRR_WRITE records = 3764 (322710016 bytes)
        Total DRR_WRITE_BYREF records = 0 (0 bytes)
        Total DRR_WRITE_EMBEDDED records = 1 (0 bytes)
        Total DRR_FREE records = 1825 (0 bytes)
        Total DRR_SPILL records = 1 (4096 bytes)
        Total records = 7509
        Total payload size = 322983056 (0x13405490)
        Total header overhead = 2342808 (0x23bf98)
        Total stream length = 325325864 (0x13641428)
# zstreamdump zfs2.snapshot
BEGIN record
        hdrtype = 1
        features = 4
        magic = 2f5bacbac
        creation_time = 671143b5
        type = 2
        flags = 0x0
        toguid = 3907ba4a0ea7cfce
        fromguid = 0
        toname = zpool1/zfs2@migrate3
        payloadlen = 0
END checksum = 2c97cd94a9b83c/b9929659c7ac52f6/8ba3ff2da7907548/f12af225a109af94
SUMMARY:
        Total DRR_BEGIN records = 1 (0 bytes)
        Total DRR_END records = 1 (0 bytes)
        Total DRR_OBJECT records = 251 (41704 bytes)
        Total DRR_FREEOBJECTS records = 51 (0 bytes)
        Total DRR_WRITE records = 487 (38205952 bytes)
        Total DRR_WRITE_BYREF records = 0 (0 bytes)
        Total DRR_WRITE_EMBEDDED records = 0 (0 bytes)
        Total DRR_FREE records = 305 (0 bytes)
        Total DRR_SPILL records = 0 (0 bytes)
        Total records = 1096
        Total payload size = 38247656 (0x2479ce8)
        Total header overhead = 341952 (0x537c0)
        Total stream length = 38589608 (0x24cd4a8)

My initial suspicion was that QNAP’s ZFS implementation had some changes that made it incompatible, but I think the zstreamdump output suggests otherwise?

I don’t have a huge amount of experience with ZFS, so would appreciate any pointers on other things to try! I still have access to the original data so can always regenerate the snapshot stream if necessary. If the worst comes to the worst I can always do an rsync instead, but I’m hoping to avoid that if I can.

Thanks,
Neil

2

QNAP did indeed do quite a few changes from whatever version of ZFS they forked (best I can guess is v0.7.x) but that would usually manifest as an error of unsupported feature flag if something was enabled that isn’t supported.

Are you able to try a direct system-to-system send, instead of the go-between of piping snapshots into files?

Failing that, an rsync or other file-by-file replication (we’ve got a pretty good one already, and it’s going to get some extra polish in 24.10) will copy your files over.

1 Like
3

Unfortunately yes it’s still the same odd errors:

# zfs send -v zpool1/zfs1@migrate3 | ssh root@truenas "zfs receive -v temp-pool/recv/zfs1"
send from @ to zpool1/zfs1@migrate3 estimated size is 312M
total estimated size is 312M
TIME        SENT   SNAPSHOT
receiving full stream of zpool1/zfs1@migrate3 into temp-pool/recv/zfs1@migrate3
cannot open 'temp-pool/recv/zfs1': dataset does not exist
cannot receive new filesystem stream: dataset does not exist

# zfs send -v zpool1/zfs2@migrate3 | ssh root@truenas "zfs receive -v temp-pool/recv/zfs2"
send from @ to zpool1/zfs2@migrate3 estimated size is 37.5M
total estimated size is 37.5M
TIME        SENT   SNAPSHOT
receiving full stream of zpool1/zfs2@migrate3 into temp-pool/recv/zfs2@migrate3
cannot receive new filesystem stream: invalid backup stream

I guess I will be doing an rsync :slight_smile: (well actually 2 rsyncs as I want to eventually move everything back onto the QNAP hardware after installing TrueNAS on it)

Thanks,
Neil

4

For the benefit of anybody who comes across this post in the future, I had a look through the published GPL/CDDL source code for QuTS Hero (available on SourceForge) looking at QNAP’s zfs send and receive code. I can see they have definitely modified the stream format in ways that have likely caused the invalid backup stream error I hit above.

The most obvious is that they have added new types of record to the stream that are incompatible with types in OpenZFS (presumably added after their fork). For example, their DRR_VERSION record type is incorrectly interpreted by OpenZFS as a DRR_WRITE_EMBEDDED record.

Here’s a snippet from QNAP’s code that shows the above additions, helpfully delineated by #ifdefs:

Code from zfs_ioctl.h 
typedef struct dmu_replay_record {
	enum {
		DRR_BEGIN, DRR_OBJECT, DRR_FREEOBJECTS,
		DRR_WRITE, DRR_FREE, DRR_END, DRR_WRITE_BYREF,
		DRR_SPILL,
#ifdef QNAP_SNAPSYNC_VERSION
		DRR_VERSION,
#endif /* QNAP_SNAPSYNC_VERSION */
#ifdef QNAP_LARGE_DIR
		DRR_SPILL_SPILLDN,
#endif
		DRR_NUMTYPES
	} drr_type;
	uint32_t drr_payloadlen;
	union {
		struct drr_begin drr_begin;
		struct drr_end drr_end;
		struct drr_object drr_object;
		struct drr_freeobjects drr_freeobjects;
		struct drr_write drr_write;
		struct drr_free drr_free;
		struct drr_write_byref drr_write_byref;
		struct drr_spill drr_spill;
#ifdef QNAP_SNAPSYNC_VERSION
		struct drr_version drr_version;
#endif /* QNAP_SNAPSYNC_VERSION */
#ifdef QNAP_LARGE_DIR
		struct drr_spill_spilldn drr_spill_spilldn;
#endif
	} drr_u;
} dmu_replay_record_t;

I’ve not dug much further as it’s obvious that doing a zfs send | zfs receive from a QNAP QuTS hero system to any other ZFS based system is not going to work, at least not without a probably large amount of effort to strip their modifications out. And who knows what else they’ve modified, so I wouldn’t trust the results anyway! A shame, but not too surprising.

Thanks,
Neil

1 Like