@jcizzo - Okay, I understand more clearly.
Their are 2 parts to Grub for booting. (And a 3rd for setting up or changing booting whence the OS is booted.)
First, it’s from the BIOS where the BIOS selects the boot device and launches the boot code. This is the part BEFORE you get the Grub menu, which allows launching the Grub menu. I install boot block code on both my boot devices, either because it’s MD-RAID Mirrored or manually.
grub-install --target=i386-pc /dev/sda
grub-install --target=i386-pc /dev/sdb
or
grub-install --efi-directory=/boot/efi --target=x86_64-efi --no-nvram
Whence you get the Grub menu, if the target is Mirrored, (or even ZFS RAID-Zx), as long as Grub understands the file system, it can then continue with booting the OS. Technically, it's the Linux initial RAM disk that has the file system handling code that allows booting to RAIDed devices, (Mirror or other RAID).
Their are lots of gotchas. Here are the ones I can think of right now, in order:
- Bad block in the boot code on the first boot device. BIOSes, and probably EFI, may not be able to recover and select next boot device. But, user can manually select next boot device IF there is one. The problem is that BIOSes and probably EFI don’t understand Mirrors or RAID.
- Bad block in Grub / EFI code on first Mirror of “/boot” or “/boot/efi”. Again, user can select 2nd Mirror manually. Again, problem is that the boot block code does not understand Mirrors or RAID.
- Bad block in “/boot” for Linux kernel, initial RAM disk or other critical support file. Again, user can select 2nd Mirror manually. Again, problem is that Grub likely does not understand Mirrors or RAID.
Now one way some people get around this, is to use hardware RAID-1, (aka Mirroring), for “/boot”, (and “/boot/efi”). This theoretically prevents a failed Mirror from preventing booting. But, their is a problem. Most cheaper hardware RAID controllers don’t support both scrubbing AND automatically fixing bad blocks like ZFS does.
So, some people think using 3 devices for booting solves that problem. Use 2 way hardware RAID-1 Mirroring, and use ZFS Mirroring with that and a 3rd device. Thus, ZFS can both detect corruption in either the hardware RAID-1 Mirror or it’s other device. ZFS only sees it as a 2 way Mirror.
In a perfect world, we would have BIOSes that have:
- A clear, precise boot order that does not change when adding new devices
- Understand Mirrors, perhaps even have a clear setup in the boot order saying:
Mirror of “sda” “sdb”
Recovery “sdc”
Then have the boot code loaded above understand Mirroring too, so that reading a Mirrored “/boot” and or “/boot/efi” works regardless of bad blocks on either device, (as long as it’s not the same block on both devices that is needed).
Complicated but fixable.
