For some reason, I am now getting a failure on each boot. The problem appears to be with ZFS trying to mount ca-certificates in a /root//var/ folder, which is odd.
Once I make it continue to boot (with exit), everything gets back to normal. With zfs list -t I can see ca-certificates mounted properly:boot-pool/ROOT/24.04.1.1/var/ca-certificates 96K 4.80G 96K /var/local/ca-certificates
The only other ca-certificate entry is for 24.04:boot-pool/ROOT/24.04.0/var/ca-certificates 96K 4.80G 96K /var/local/ca-certificates
Any idea what is going on here? I think the extra /root// prefix is some leftover from my recent upgrade to 24.04.1.1 maybe?
Did you do any manual operations to the boot pool?
Changing properties or destroying datasets?
EDIT: can you paste “zpool history boot-pool”
william:
zpool history boot-pool
2024-06-02.01:42:42 zpool import -N -f boot-pool
2024-06-02.03:45:03 py-libzfs: zpool scrub boot-pool
2024-06-02.16:28:21 zpool set autoexpand=on boot-pool
2024-06-02.16:28:42 zpool online -e boot-pool vda3
2024-06-02.16:40:05 zpool online -e boot-pool vda3
2024-06-02.16:44:28 zpool online -e boot-pool /dev/vda3
2024-06-02.16:44:52 zfs create -o mountpoint=legacy -o truenas:kernel_version=6.6.29-production+truenas -o zectl:keep=False boot-pool/ROOT/24.04.1.1
2024-06-02.16:44:52 zfs create -u -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o exec=off -o acltype=off -o aclmode=discard -o atime=off boot-pool/ROOT/24.04.1.1/audit
2024-06-02.16:44:52 zfs create -u -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o exec=off -o acltype=off -o aclmode=discard boot-pool/ROOT/24.04.1.1/conf
2024-06-02.16:44:52 zfs snapshot boot-pool/ROOT/24.04.0/data@install-2024-06-02-14-44-52
2024-06-02.16:44:52 zfs clone -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o exec=off -o acltype=off -o aclmode=discard -o atime=off boot-pool/ROOT/24.04.0/data@install-2024-06-02-14-44-52 boot-pool/ROOT/24.04.1.1/data
2024-06-02.16:44:52 zfs create -u -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o exec=off -o acltype=off -o aclmode=discard -o atime=off boot-pool/ROOT/24.04.1.1/mnt
2024-06-02.16:44:52 zfs create -u -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o exec=off -o acltype=off -o aclmode=discard boot-pool/ROOT/24.04.1.1/etc
2024-06-02.16:44:52 zfs snapshot boot-pool/ROOT/24.04.0/home@install-2024-06-02-14-44-52
2024-06-02.16:44:52 zfs clone -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o exec=off -o acltype=off -o aclmode=discard boot-pool/ROOT/24.04.0/home@install-2024-06-02-14-44-52 boot-pool/ROOT/24.04.1.1/home
2024-06-02.16:44:52 zfs create -u -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o acltype=off -o aclmode=discard boot-pool/ROOT/24.04.1.1/opt
2024-06-02.16:44:52 zfs snapshot boot-pool/ROOT/24.04.0/root@install-2024-06-02-14-44-52
2024-06-02.16:44:52 zfs clone -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o acltype=off -o aclmode=discard boot-pool/ROOT/24.04.0/root@install-2024-06-02-14-44-52 boot-pool/ROOT/24.04.1.1/root
2024-06-02.16:44:52 zfs create -u -o mountpoint=legacy -o canmount=noauto -o acltype=off -o aclmode=discard -o atime=off boot-pool/ROOT/24.04.1.1/usr
2024-06-02.16:44:52 zfs create -u -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o acltype=off -o aclmode=discard -o atime=off boot-pool/ROOT/24.04.1.1/var
2024-06-02.16:44:52 zfs create -u -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o exec=off -o acltype=off -o aclmode=discard boot-pool/ROOT/24.04.1.1/var/ca-certificates
2024-06-02.16:44:52 zfs snapshot boot-pool/ROOT/24.04.0/var/log@install-2024-06-02-14-44-52
2024-06-02.16:44:53 zfs clone -o mountpoint=legacy -o canmount=noauto -o setuid=off -o devices=off -o exec=off -o atime=off boot-pool/ROOT/24.04.0/var/log@install-2024-06-02-14-44-52 boot-pool/ROOT/24.04.1.1/var/log
2024-06-02.16:45:44 zpool set bootfs=boot-pool/ROOT/24.04.1.1 boot-pool
2024-06-02.17:58:24 zfs set readonly=off boot-pool/ROOT/24.04.1.1/usr
2024-06-02.17:58:24 zfs set readonly=on boot-pool/ROOT/24.04.1.1/usr
2024-06-02.17:58:24 zfs set readonly=off boot-pool/ROOT/24.04.1.1/audit
2024-06-02.17:58:24 zfs set mountpoint=/audit boot-pool/ROOT/24.04.1.1/audit
2024-06-02.17:58:24 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/audit
2024-06-02.17:58:24 zfs set readonly=on boot-pool/ROOT/24.04.1.1/conf
2024-06-02.17:58:24 zfs snapshot boot-pool/ROOT/24.04.1.1/conf@pristine
2024-06-02.17:58:25 zfs set mountpoint=/conf boot-pool/ROOT/24.04.1.1/conf
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/conf
2024-06-02.17:58:25 zfs set readonly=off boot-pool/ROOT/24.04.1.1/data
2024-06-02.17:58:25 zfs set mountpoint=/data boot-pool/ROOT/24.04.1.1/data
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/data
2024-06-02.17:58:25 zfs set readonly=off boot-pool/ROOT/24.04.1.1/mnt
2024-06-02.17:58:25 zfs set mountpoint=/mnt boot-pool/ROOT/24.04.1.1/mnt
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/mnt
2024-06-02.17:58:25 zfs set readonly=off boot-pool/ROOT/24.04.1.1/etc
2024-06-02.17:58:25 zfs snapshot boot-pool/ROOT/24.04.1.1/etc@pristine
2024-06-02.17:58:25 zfs set mountpoint=/etc boot-pool/ROOT/24.04.1.1/etc
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/etc
2024-06-02.17:58:25 zfs set readonly=off boot-pool/ROOT/24.04.1.1/home
2024-06-02.17:58:25 zfs set mountpoint=/home boot-pool/ROOT/24.04.1.1/home
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/home
2024-06-02.17:58:25 zfs set readonly=on boot-pool/ROOT/24.04.1.1/opt
2024-06-02.17:58:25 zfs snapshot boot-pool/ROOT/24.04.1.1/opt@pristine
2024-06-02.17:58:25 zfs set mountpoint=/opt boot-pool/ROOT/24.04.1.1/opt
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/opt
2024-06-02.17:58:25 zfs set readonly=off boot-pool/ROOT/24.04.1.1/root
2024-06-02.17:58:25 zfs set mountpoint=/root boot-pool/ROOT/24.04.1.1/root
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/root
2024-06-02.17:58:25 zfs set readonly=on boot-pool/ROOT/24.04.1.1/usr
2024-06-02.17:58:25 zfs snapshot boot-pool/ROOT/24.04.1.1/usr@pristine
2024-06-02.17:58:25 zfs set mountpoint=/usr boot-pool/ROOT/24.04.1.1/usr
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/usr
2024-06-02.17:58:25 zfs set readonly=off boot-pool/ROOT/24.04.1.1/var
2024-06-02.17:58:25 zfs snapshot boot-pool/ROOT/24.04.1.1/var@pristine
2024-06-02.17:58:25 zfs set mountpoint=/var boot-pool/ROOT/24.04.1.1/var
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/var
2024-06-02.17:58:25 zfs set readonly=off boot-pool/ROOT/24.04.1.1/var/ca-certificates
2024-06-02.17:58:25 zfs set mountpoint=/var/local/ca-certificates boot-pool/ROOT/24.04.1.1/var/ca-certificates
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/var/ca-certificates
2024-06-02.17:58:25 zfs set readonly=off boot-pool/ROOT/24.04.1.1/var/log
2024-06-02.17:58:25 zfs set mountpoint=/var/log boot-pool/ROOT/24.04.1.1/var/log
2024-06-02.17:58:25 zfs set org.zectl:bootloader="" boot-pool/ROOT/24.04.1.1/var/log
2024-06-02.17:58:25 zfs set readonly=on boot-pool/ROOT/24.04.1.1
2024-06-02.17:58:25 zfs snapshot boot-pool/ROOT/24.04.1.1@pristine
2024-06-02.17:58:25 zfs snapshot boot-pool/.system/samba4@update--2024-06-02-15-58--SCALE-24.04.0
2024-06-02.17:59:41 zpool import -N -f boot-pool
2024-06-02.17:59:56 py-libzfs: zfs promote boot-pool/ROOT/24.04.1.1/root
2024-06-02.17:59:56 py-libzfs: zfs promote boot-pool/ROOT/24.04.1.1/var/log
2024-06-02.17:59:56 py-libzfs: zfs promote boot-pool/ROOT/24.04.1.1/data
2024-06-02.17:59:56 py-libzfs: zfs promote boot-pool/ROOT/24.04.1.1/home
2024-06-02.18:04:46 zfs set readonly=off boot-pool/ROOT/24.04.1.1/usr
2024-06-02.18:05:27 zfs set readonly=on boot-pool/ROOT/24.04.1.1/usr
2024-06-02.19:16:01 zpool import -N -f boot-pool
2024-06-02.19:47:00 zpool import -N -f boot-pool
2024-06-02.19:55:36 zpool import -N -f boot-pool
2024-06-02.20:04:07 zpool import -N -f boot-pool
2024-06-02.20:15:19 zpool import -N -f boot-pool
2024-06-02.20:16:57 zpool import -N -f boot-pool
2024-06-02.20:18:54 zpool import -N -f boot-pool
2024-06-03.13:43:49 zpool import -N -f boot-pool
2024-06-03.19:02:44 zpool import -N -f boot-pool
2024-06-04.22:00:30 zpool import -N -f boot-pool
2024-06-05.00:54:39 zpool import -N -f boot-pool
2024-06-05.00:58:20 zpool import -N -f boot-pool
2024-06-05.01:05:38 zpool import -N -f boot-pool
2024-06-05.01:12:22 zpool import -N -f boot-pool
2024-06-05.15:13:00 zpool import -N -f boot-pool
2024-06-06.20:39:58 zpool import -N -f boot-pool
2024-06-06.20:43:05 zpool import -N -f boot-pool
2024-06-06.20:48:24 zpool import -N -f boot-pool
2024-06-06.23:27:25 zpool import -N -f boot-pool
2024-06-06.23:44:42 zpool import -N -f boot-pool
2024-06-07.11:00:19 zpool import -N -f boot-pool
2024-06-07.11:59:32 zpool import -N -f boot-pool
2024-06-07.12:19:28 zpool import -N -f boot-pool
The only manual change I made was a temporary readonly=off on boot-pool/ROOT/24.04.1.1/usr to install the OneDrive SDK.
Can you confirm whats “zfs get mountpoint boot-pool/ROOT/24.04.1.1/var/ca-certificates” after it boots?
root@truenas:~# zfs get mountpoint boot-pool/ROOT/24.04.1.1/var/ca-certificates
NAME PROPERTY VALUE SOURCE
boot-pool/ROOT/24.04.1.1/var/ca-certificates mountpoint /var/local/ca-certificates local
@William any other ideas on how to troubleshoot this?
@wrobelda no ideas, but since you did change the root filesystem its impossible to tell what has changed.
Please let us know if you can reproduce on a pristine install.
I know exactly what I changed, basically cp -ar src/onedrivesdk /usr/lib/python3/dist-packages. It’s hard to imagine this would have any effect at all, especially since /usr/ is a separate dataset from /var
What I am asking is for ideas on where could this mount point be even coming from, there must be a reason why it is being attempted at boot time but not listed by zfs list. I am a software engineer very familiar with Linux, so happy to do some debugging, but need some hints first.
I did investigate a bit and apparently at boot time, everything is mounted into /root first:
So I checked the folder it complains about and indeed it does not exist:
.
Creating that folder manually allows mount to carry on without any issue:
So correct me if I am wrong, but the issue here is that boot-pool/ROOT/24.04.1.1./var/local dataset is missing the ca-certificate subfolder.
My best guess here is that changing /usr might have messed the ordering of listed dataset, exposing a bug in the zfs initramfs responsible for mounting the datasets.
zfs/contrib/initramfs/scripts/zfs at truenas/zfs-2.3-release · truenas/zfs · GitHub This is the script ran on boot.
