Newly upgraded MINI-X keeps rebooting

TrueNAS General
1

I just updated from CORE to SCALE ( TrueNAS-SCALE-23.10.2 TRUENAS-MINI-3.0-X) and it keeps rebooting every few minutes. It is quite functional in between reboots but the cycle of reboots every 5 minutes gets annoying.

See nothing in the logs. Attaching a snippet with last lines of one of the runs

journalctl 
Apr 09 01:01:11 freenas kernel: IPVS: rr: UDP 172.17.0.10:53 - no destination available
Apr 09 01:01:11 freenas k3s[5878]: I0409 01:01:11.663398    5878 csi_plugin.go:99] kubernetes.io/csi: Trying to validate a new CSI Driver with name: zfs.csi.openebs.io endpoint: /var/lib/kubelet/plugins
/zfs-localpv/csi.sock versions: 1.0.0
Apr 09 01:01:11 freenas k3s[5878]: I0409 01:01:11.663454    5878 csi_plugin.go:112] kubernetes.io/csi: Register new plugin with name: zfs.csi.openebs.io at endpoint: /var/lib/kubelet/plugins/zfs-localpv
/csi.sock
Apr 09 01:01:11 freenas k3s[5878]: I0409 01:01:11.910477    5878 csi_plugin.go:178] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin zfs.csi.openebs.io
Apr 09 01:01:12 freenas k3s[5878]: I0409 01:01:12.150888    5878 controller.go:615] quota admission added evaluator for: zfsnodes.zfs.openebs.io
Apr 09 01:01:12 freenas k3s[5878]: I0409 01:01:12.303412    5878 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="kube-system/openebs-zfs-controller-0" podStartSLOduration=18.303
335122 pod.CreationTimestamp="2024-04-09 01:00:54 -0500 CDT" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-04-09 01:01:11.677471543 -0500 CDT m=+40.077098213" watchObservedRunningTime="2024-04-09 01:01:12.303335122 -0500 CDT m=+40.702961784"
Apr 09 01:01:12 freenas k3s[5878]: I0409 01:01:12.913840    5878 csi_plugin.go:99] kubernetes.io/csi: Trying to validate a new CSI Driver with name: zfs.csi.openebs.io endpoint: /var/lib/kubelet/plugins/zfs-localpv/csi.sock versions: 1.0.0
Apr 09 01:01:12 freenas k3s[5878]: I0409 01:01:12.913905    5878 csi_plugin.go:112] kubernetes.io/csi: Register new plugin with name: zfs.csi.openebs.io at endpoint: /var/lib/kubelet/plugins/zfs-localpv/csi.sock
Apr 09 01:01:16 freenas chronyd[3410]: Selected source 50.205.57.38 (0.debian.pool.ntp.org)
Apr 09 01:01:44 freenas k3s[5878]: I0409 01:01:44.658801    5878 scope.go:115] "RemoveContainer" containerID="be3b8a620a982ebe5532ed544d24d5cea5c17c5a2cb9ca86c2d2620042750dd3"
Apr 09 01:01:44 freenas k3s[5878]: I0409 01:01:44.703360    5878 scope.go:115] "RemoveContainer" containerID="a59331a64516e553036f4f017ab343c0805b1357d1f588e409ec2fb142ac7f04"
Apr 09 01:01:44 freenas k3s[5878]: I0409 01:01:44.737841    5878 scope.go:115] "RemoveContainer" containerID="90709c67975d00eb7d80220bb7b26494df8d45fa85213611a4514295ab6b1716"
Apr 09 01:01:44 freenas k3s[5878]: I0409 01:01:44.774160    5878 scope.go:115] "RemoveContainer" containerID="e6ccce2065e16eb35362a59514e7e938af6cb43645538f138e3c8d9756c46411"
Apr 09 01:01:44 freenas k3s[5878]: I0409 01:01:44.813163    5878 scope.go:115] "RemoveContainer" containerID="3f46a8508d526241f957d7aafaa3a0c9530f96dcf30743b37054fa5cb0afc5fa"
Apr 09 01:01:44 freenas k3s[5878]: I0409 01:01:44.872597    5878 scope.go:115] "RemoveContainer" containerID="3939e02caf9f55e83f1e0246b51193618c5fe154a16e9d5b3eea928a7a791ff2"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.019411    5878 scope.go:115] "RemoveContainer" containerID="d0ba38d90cee9629287c38bce909d021d4631972be041bcf4da89855509c5ff0"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.078617    5878 scope.go:115] "RemoveContainer" containerID="60f0bdacaacf544e4168104f413ea03c07361ec9eeb09e004d028f4cdf026c8c"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.104076    5878 scope.go:115] "RemoveContainer" containerID="d70cd004880e692a8bc087fd937eda86c119c521251b0414d4c7e870e52a1a96"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.127240    5878 scope.go:115] "RemoveContainer" containerID="89abfa89328530666c5e43caf66af9853f760bf4a27e804bd917817537576bc1"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.151705    5878 scope.go:115] "RemoveContainer" containerID="a02918e5851eb4b0e99ea725f398ccad0224cdba1f8bfce56a22b3d7f11b1192"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.179046    5878 scope.go:115] "RemoveContainer" containerID="21c346888d3ad03beafebca2c3c5df2b972e231f7eec1740e0c7c997dfb312ec"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.208566    5878 scope.go:115] "RemoveContainer" containerID="41586996af0ea2bf4c26b300128884f4a628b884b822883e20fb424285364bcd"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.249037    5878 scope.go:115] "RemoveContainer" containerID="117563a7ef300bec5885431b25010fa3b62bf99e9efff5f230143fe96d382058"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.316877    5878 scope.go:115] "RemoveContainer" containerID="fbf197f63a4054eadeddf49d5220618f1ea7177785b0282b7d40fbf93417c1a6"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.373505    5878 scope.go:115] "RemoveContainer" containerID="81a84a3c7f4afc7a732574161a79e524997f7bf8490da192d2406fd9049115b5"
Apr 09 01:01:45 freenas k3s[5878]: I0409 01:01:45.432916    5878 scope.go:115] "RemoveContainer" containerID="26d5c9f454e79f8c53a680712e0eb3334f0cc942630fc148b9e770f5198193f6"
Apr 09 01:01:51 freenas k3s[5878]: {"level":"warn","ts":"2024-04-09T01:01:51.576-0500","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000fb8380/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Apr 09 01:01:59 freenas systemd[1]: run-containerd-runc-k8s.io-bfd6927ff68119fd92a2599fc1faf7e5e6dbd37b61ea459850a32cd6a2f24098-runc.0HeMjl.mount: Deactivated successfully.
-- Boot 4dd210f78c074efa9eb671571f4e9bb6 --
Apr 09 01:05:11 freenas kernel: microcode: microcode updated early to revision 0x38, date = 2021-12-02
Apr 09 01:05:11 freenas kernel: Linux version 6.1.74-production+truenas (root@tnsbuilds01.tn.ixsystems.net) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #2 SMP PREEMPT_DYNAMIC Wed Feb 21 20:30:38 UTC 2024
Apr 09 01:05:11 freenas kernel: Command line: BOOT_IMAGE=/ROOT/23.10.2@/boot/vmlinuz-6.1.74-production+truenas root=ZFS=freenas-boot/ROOT/23.10.2 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
1 Like
2

First: Did you save a copy of your TrueNAS configuration?
Second: Did you upgrade the ZFS Feature Flags (I Hope Not)?

If you have not upgraded the ZFS feature flags, DON’T!

In the GUI, can you select the previous Boot Environment? If you have not updated the feature flags, try to roll back to your previous CORE boot environment.

These steps will get you back to your data. Migrating from CORE to SCALE is not as cut and dry as most would expect. Some go well, some not at all.

3

If you cannot roll back, Disable any jails (which no longer exist in SCALE), VM’s, everything. I think the log it telling you a Docker is causing the reboot, I I’m not experienced enough with Docker failures to know for certain.

If you find the culprit, leave it off until you can figure out a way to move forward.

Best of luck.

4

Hard reboots like that on a Mini could be indicative of something else going on… You may want to file a bug ticket with a debug attached so we can look closer, see if there is a kernel panic or something else.

5

My mailbox is being spammed by True NAS reboots every 6:30 so it is very stable. It seems to me the reason for these reboots is a confused watchdog.

I also find this message in error log

Apr  9 06:46:03 freenas kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000003e2e0000-0x000000003e2fffff], contact BIOS vendor for fixes
Apr  9 06:52:29 freenas kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000003e2e0000-0x000000003e2fffff], contact BIOS vendor for fixes
Apr  9 06:58:56 freenas kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000003e2e0000-0x000000003e2fffff], contact BIOS vendor for fixes
Apr  9 07:05:22 freenas kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000003e2e0000-0x000000003e2fffff], contact BIOS vendor for fixes
Apr  9 07:11:46 freenas kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000003e2e0000-0x000000003e2fffff], contact BIOS vendor for fixes
Apr  9 07:18:12 freenas kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x000000003e2e0000-0x000000003e2fffff], contact BIOS vendor for fixes
6

Thanks, that is some helpful information. You may want to contact iX support, make sure we can validate you are on latest BIOS and have them give it a check.

7

They will not support me since I have no contract. That’s the reason for switching to SCALE. I wonder if MINI was tested with SCALE.

8

On every reboot MINI talks to www.google-analytics.com. At least somebody cares about my problems.

9

Angular gotta angular :slight_smile:

I’d suggest filing a bug ticket with debug file attached then. We’ll take a look and see if there’s something we can address. You may want to re-test with 24.10-RC.1 in the meantime, since that’s where our own testing is focused at the moment.

10

Case 00054333 filed. The support is sending me back to the forums.

11

Bug tickets get filed here under the TrueNAS project.

https://ixsystems.atlassian.net/

12

A reboot every five minutes could be the watchdog timer not behaving.

Can you try sudo echo 0 > /proc/sys/kernel/nmi_watchdog to see if that disables it?

13

Didn’t help. ;(

14

However disabling Watchdog Timer in BIOS did.

uptime 11 minutes, for the very first time since yesterday

2 Likes