The ZFS "copies" features is useless: Change my mind

General Discussion
21

Well, my thought, (untested as of now), is to make a zVol on my desktop, and then make a ZFS pool in it. That pool would then have “copies=2”, (or 3), and I would copy stuff in it. My blog entries, and backups of misc. stuff. Plus a bunch of writings, including un-published works.

Then export that virtual pool. Last, perhaps snapshot the zVol and use that to burn the M-Disc.

Testing afterwards should be as simple as trying to import the pool, R/O, via the optical drive device path. All the files should be accessible and available as normal. Though I don’t know if you can “scrub” a R/O pool. However, if you get zero errors in “zpool status” of this pool, any file copied off should be perfect.

But, I have not yet gotten a round tuit. I had planned to take the copy to my office as a disaster recovery location… But, I was sent home late February 2020, (if I remember correctly). Have only been back to the office a few times, And not in the last year or 2. I am now officially a member of the Hermits United Club of America.

2 Likes
22

That’s very interesting, but it seems like one of those use-cases where alternatives actually shine better than ZFS.

I can think of two alternatives, which both allow instant file access on the disc (like browsing a folder), without the need to do any extra steps, such as transferring or importing, nor relying on zfs or zpool actions.

Option 1: PAR2

Put all your important files into a folder.

Create PAR2 recovery blocks, with you preferred percentage of data-to-recovery. (You can base this on how much will fit onto a disc.)

par2create -R -r15 MyFolder MyFolder

Burn MyFolder and all MyFolder.*par2 files directly to the disc.

Result: All files and folders are immediately accessible on the disc, no different than browsing a USB stick. You have enough recovery blocks that can take as much damage (or bitrot) to the media (anywhere on its surface), up to the amount you specified when you created the recovery files.

Caveat: This does not safeguard against filesystem damage (such as UDF or metadata.)

Option 2: dvdisaster

Put all your important files into a folder.

Create an ISO out of this folder (preferably UDF format.)

Use dvdisaster to create an “augmented” ISO, with recovery bits baked into the entire ISO itself.

Burn this ISO as a disc.

Result: All files and folders are immediately accessible on the disc, no different than browsing a USB stick. You have enough recovery information that can take as much damage (or bitrot) to the media (anywhere on its surface), based on the total capacity minus the data’s consumed amount. (E.g, if the M-Disc is 25 GiB, and the total data is 15 GiB, than you have 10 GiB worth of parity protection.)

Bonus: This protects against any type of damage, even filesystem or metadata damage.

I can vouch for Option 2. I’ve actually tested it myself, and it works, even when I took a knife and made a scratch across the disc.

The recovery did take a while, but it restored the ISO to 100% original condition.

It’s a two-part process.

  1. You tell dvdisaster to try to read the entire disc, in which it is saved as an ISO to your computer. Naturally, this ISO will not be the original, since the disc has physical damage.

  2. You tell dvdisaster to try to restore the ISO to its original state, using the baked-in augmented recovery info. (This is all automatic. Nothing required on your part, not even foreknowledge of the ISO.) This part takes a while (CPU intensive, Reed-Solomon algorithm), but in the end, it worked! :sunglasses:

4 Likes
23

This thread, basically:

zfstrix

24

I was thinking to burn a filesystem image, rather than a snapshot stream. I don’t think a snapshot stream would preserve the copies?

zpool create -O copies=3 burntank /path/to/file/burntank.img
Copy files to it, export it, burn the img file to a DVD.

I wonder if it would be better to create a DVD-sized file first and to loopmount it. ZFS is supposed to spread copies across the LBA range, so it might be good to size the file.

You should be able to import a ZFS filesystem from most any seekable device.

(Ewww, PAR2. What is this, Usenet?)
(I wasn’t familiar with dvdisaster. Neat. More space efficient.)

26

One slight advantage of a ZFS pool on an optical disk, is that it is, (in my opinion), more generic. The PAR2 or DVDisaster method requires those 2 programs to both be installed and functional. Since I use OpenZFS on all my computers, (desktop, media server, 2 x laptops and NAS), that would not be a problem, for me.

I have looked at DVDisaster, it is even loaded on my desktop right now. But, I have never used it. So it may be better or possibly more widely available, as there does seem to be versions for MS-Windows, FreeBSD and MacOS.

2 Likes
27

Here is a use case. You have FreeBSD installed, say, on a laptop only with space for a single drive. copies=2 may save your data from bitrot. It will not protect against drive failure, but as Allan Jude says, ZFS, even on a single drive, and even with non-ECC RAM, will do a better job than any of it’s contemporaries.
What’s the alternative? Install linux? Run UFS? No, it doesn’t really help in most cases, however, there are cases where it could be useful…

28

Isn’t copies=2/3 supposed to spread blocks as far as possible in a pool with multiple vdevs? If one of said vdevs croaks completely due to happenstance (e.g. three disks giving a salute in a wide RAID-Z2), you could still recover copies=2/3 data. Same reasoning why it’s apparently applied to metadata out of the box.

1 Like
29

Interesting point. Yes, it’s supposed to “try”. How that translates into the real world with a wide RAIDZ I’m not really sure.

Any ZFS Guru’s out there know the answer?

1 Like
30

Something about your username makes me suspect some bias.

:unamused:

Can’t quite put my finger on it…

2 Likes
31

Maybe, maybe not. My name has a dash instead of an equals sign. :grin:

3 Likes
32

Doing it to a single wide RAID-Z probably won’t help, with what it striping blocks as wide as possible, the chances that all ditto blocks are broken are nearly guaranteed. The idea is to spread it across multiple vdevs (i.e. multiple RAID-Z/mirrors).

2 Likes
33

As a bonus fun fact:
ZFS Metadata is always copied at least 2 times. Why not have an option to copy the rest of your data 2 times :slight_smile:

Also, Jim Salters has a good writeup on this topic

Testing copies=n resiliency – JRS Systems: the blog (jrs-s.net)

Without copies=2 set, we lost 93 files instead of 2. So copies=n was definitely better than nothing for our test of writing 0.1% of the filesystem as bad blocks… but it wasn’t fabulous, either, and it fell flat on its face with 1% or 10% of the filesystem corrupted. By comparison, a truly redundant pool - one with two disks in it, in a mirror vdev - would have survived the same test (corrupting ANY number of blocks on a single disk) with flying colors.

zfs: copies=n is not a substitute for device redundancy! – JRS Systems: the blog (jrs-s.net)

But yeah, it’s really got niche usecases :stuck_out_tongue:

2 Likes
34

I wouldnt be using copies=2 because I am wanting to save money, but I do use it the device I am using only supports one storage device.

The drive I am using in this scenario is ridiculously over spec’d, it will routinely have in excess of 95% free on its capacity, and the device I am putting it in only supports one storage device, so yes copies=2 is useful in this scenario and probably the prime use case for it.

If you using a redundant setup such as ZFS mirror, then copies=1 does the job.

35

Also, to add (sorry it seems I cant edit) if its having to fix things on the drive, it is kind of an early warning system, something isnt right with the hardware as well. As I wouldnt expect errors on hardware that is working properly.

36

I use it (copies=2) for companies data stored on my laptop (only one nvme). laptop isn’t the only one place that these data exist but I need them read correctly being anywhere, even without access to NAS. If ssd fails, ok, shit happens but I have more chance to get them when any sector blows up :slight_smile:
and… it could happen:
233 Media_Wearout_Indicator 0x0032 085 085 000 Old_age Always - 0
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 43
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 2
:slight_smile:

4 Likes
37

Pity. Another user I have to add to my block list. :pensive:

1 Like
38

You must not be very good at chess if that is a checkmate :slight_smile:

A mirror is better but a single copies=2 is not useless

  1. If you only have one SATA port left in a NAS box but want large drive to store onsite backups (still have a 2nd copy of the data)
  2. When drives fail (Mechanical) they typically give warning in the form of back sectors they do not go up in smoke the majority of the time. Usually time to replace that drive (not as good as a mirror but better than ufs solo drive
  3. Price two 8TB enterprise drives often cost more than one 16TB enterprise SATA
  4. Heat and Power - A solo HD produces less heat and uses less power
  5. Space - One drive takes less physical space, uese one less power and SATA cable and uses one less SATA port.

Check and Mate!

1 Like