Any advise? My system keeps crapping out and going into zombie mode and I need to reboot it. The logs are FULLL of gpu errors. The whole time the system is running k3s is complaining about the GPU. In the meantime I am going to try to disable the GPU to verify if this solves the crashing.
Jun 24 18:37:08 truenas k3s[11257]: I0624 18:37:08.545040 11257 scope.go:115] “RemoveContainer” containerID=“58d183ff47f78fbe4231de08adca9b4eee27c7dfd8c811304546ac1a2b739240”
Jun 24 18:37:13 truenas kernel: NVRM: GPU 0000:08:00.0: RmInitAdapter failed! (0x23:0x65:1426)
Jun 24 18:37:13 truenas kernel: NVRM: GPU 0000:08:00.0: rm_init_adapter failed, device minor number 0
Jun 24 18:37:15 truenas k3s[11257]: {“level”:“warn”,“ts”:“2024-06-24T18:37:15.753-0400”,“logger”:“etcd-client”,“caller”:“v3@v3.5.7-k3s1/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,>
Jun 24 18:37:17 truenas kernel: NVRM: GPU 0000:08:00.0: RmInitAdapter failed! (0x23:0x65:1426)
Jun 24 18:37:17 truenas kernel: NVRM: GPU 0000:08:00.0: rm_init_adapter failed, device minor number 0
Jun 24 18:37:18 truenas k3s[11257]: I0624 18:37:18.246498 11257 scope.go:115] “RemoveContainer” containerID=“58d183ff47f78fbe4231de08adca9b4eee27c7dfd8c811304546ac1a2b739240”
Jun 24 18:37:21 truenas kernel: NVRM: GPU 0000:08:00.0: RmInitAdapter failed! (0x23:0x65:1426)
Jun 24 18:37:21 truenas kernel: NVRM: GPU 0000:08:00.0: rm_init_adapter failed, device minor number 0
Jun 24 18:37:25 truenas kernel: NVRM: GPU 0000:08:00.0: RmInitAdapter failed! (0x23:0x65:1426)
Jun 24 18:37:25 truenas kernel: NVRM: GPU 0000:08:00.0: rm_init_adapter failed, device minor number 0
Jun 24 18:37:29 truenas kernel: NVRM: GPU 0000:08:00.0: RmInitAdapter failed! (0x23:0x65:1426)
Jun 24 18:37:29 truenas kernel: NVRM: GPU 0000:08:00.0: rm_init_adapter failed, device minor number 0
Jun 24 18:37:32 truenas k3s[11257]: W0624 18:37:32.593994 11257 manager.go:1174] Failed to process watch event {EventType:0 Name:/system.slice/kubepods-besteffort-pod9aac1ba3_4f42_4cbd_a54b_dfa2f5a66a>
Jun 24 18:37:33 truenas kernel: NVRM: GPU 0000:08:00.0: RmInitAdapter failed! (0x23:0x65:1426)
Jun 24 18:37:33 truenas kernel: NVRM: GPU 0000:08:00.0: rm_init_adapter failed, device minor number 0
Jun 24 18:37:46 truenas k3s[11257]: {“level”:“warn”,“ts”:“2024-06-24T18:37:46.914-0400”,“logger”:"etcd-cli
ul 16 09:06:31 truenas k3s[11254]: time=“2024-07-16T09:06:31-04:00” level=info msg=“COMPACT deleted 332 rows from 339 revisions in 25.241735ms - compacted to 3273169/3274169”
Jul 16 09:06:38 truenas k3s[11254]: I0716 09:06:38.161433 11254 scope.go:115] “RemoveContainer” containerID=“c49c4e5addf1388633df311e3349642163dc86a4572f54de37c8ef03bd40735e”
Jul 16 09:06:38 truenas k3s[11254]: E0716 09:06:38.161965 11254 pod_workers.go:965] “Error syncing pod, skipping” err=“failed to "StartContainer" for "nvidia-device-plugin-ctr" with CrashLoopBackOff: "back-off 5m0s restarting failed container=nvidia-device-plugin-ctr pod=nvidia-device-plugin-daemonset-msvff_kube-system(c3cecb55-b447-4bf1-a00b-a5bfe8fb0318)"” pod=“kube-system/nvidia-device-plugin-daemonset-msvff” podUID=c3cecb55-b447-4bf1-a00b-a5bfe8fb0318
Jul 16 09:06:42 truenas k3s[11254]: {“level”:“warn”,“ts”:“2024-07-16T09:06:42.260-0400”,“logger”:“etcd-client”,“caller”:“v3@v3.5.7-k3s1/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc000a4e380/kine.sock”,“attempt”:0,“error”:“rpc error: code = Unknown desc = no such table: dbstat”}
Jul 16 09:06:52 truenas k3s[11254]: I0716 09:06:52.170424 11254 scope.go:115] “RemoveContainer” containerID=“c49c4e5addf1388633df311e3349642163dc86a4572f54de37c8ef03bd40735e”
Jul 16 09:06:52 truenas k3s[11254]: E0716 09:06:52.170961 11254 pod_workers.go:965] “Error syncing pod, skipping” err=“failed to "StartContainer" for "nvidia-device-plugin-ctr" with CrashLoopBackOff: "back-off 5m0s restarting failed container=nvidia-device-plugin-ctr pod=nvidia-device-plugin-daemonset-msvff_kube-system(c3cecb55-b447-4bf1-a00b-a5bfe8fb0318)"” pod=“kube-system/nvidia-device-plugin-daemonset-msvff” podUID=c3cecb55-b447-4bf1-a00b-a5bfe8fb0318
Jul 16 09:07:05 truenas k3s[11254]: I0716 09:07:05.160538 11254 scope.go:115] “RemoveContainer” containerID=“c49c4e5addf1388633df311e3349642163dc86a4572f54de37c8ef03bd40735e”
Jul 16 09:07:05 truenas k3s[11254]: E0716 09:07:05.161089 11254 pod_workers.go:965] “Error syncing pod, skipping” err=“failed to "StartContainer" for "nvidia-device-plugin-ctr" with CrashLoopBackOff: "back-off 5m0s restarting failed container=nvidia-device-plugin-ctr pod=nvidia-device-plugin-daemonset-msvff_kube-system(c3cecb55-b447-4bf1-a00b-a5bfe8fb0318)"” pod=“kube-system/nvidia-device-plugin-daemonset-msvff” podUID=c3cecb55-b447-4bf1-a00b-a5bfe8fb0318
Jul 16 09:07:17 truenas k3s[11254]: {“level”:“warn”,“ts”:“2024-07-16T09:07:17.742-0400”,“logger”:“etcd-client”,“caller”:“v3@v3.5.7-k3s1/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc000a4e380/kine.sock”,“attempt”:0,“error”:“rpc error: code = Unknown desc = no such table: dbstat”}
Jul 16 09:07:20 truenas k3s[11254]: I0716 09:07:20.161284 11254 scope.go:115] “RemoveContainer” containerID=“c49c4e5addf1388633df311e3349642163dc86a4572f54de37c8ef03bd40735e”
Jul 16 09:07:20 truenas k3s[11254]: E0716 09:07:20.161798 11254 pod_workers.go:965] “Error syncing pod, skipping” err=“failed to "StartContainer" for "nvidia-device-plugin-ctr" with CrashLoopBackOff: "back-off 5m0s restarting failed container=nvidia-device-plugin-ctr pod=nvidia-device-plugin-daemonset-msvff_kube-system(c3cecb55-b447-4bf1-a00b-a5bfe8fb0318)"” pod=“kube-system/nvidia-device-plugin-daemonset-msvff” podUID=c3cecb55-b447-4bf1-a00b-a5bfe8fb0318
Jul 16 09:07:31 truenas k3s[11254]: I0716 09:07:31.161281 11254 scope.go:115] “RemoveContainer” containerID=“c49c4e5addf1388633df311e3349642163dc86a4572f54de37c8ef03bd40735e”
Jul 16 09:07:31 truenas k3s[11254]: E0716 09:07:31.161780 11254 pod_workers.go:965] “Error syncing pod, skipping” err=“failed to "StartContainer" for "nvidia-device-plugin-ctr" with CrashLoopBackOff: "back-off 5m0s restarting failed container=nvidia-device-plugin-ctr pod=nvidia-device-plugin-daemonset-msvff_kube-system(c3cecb55-b447-4bf1-a00b-a5bfe8fb0318)"” pod=“kube-system/nvidia-device-plugin-daemonset-msvff” podUID=c3cecb55-b447-4bf1-a00b-a5bfe8fb0318
Jul 16 09:07:45 truenas k3s[11254]: I0716 09:07:45.169555 11254 scope.go:115] “RemoveContainer” containerID=“c49c4e5addf1388633df311e3349642163dc86a4572f54de37c8ef03bd40735e”
Jul 16 09:07:45 truenas k3s[11254]: E0716 09:07:45.170032 11254 pod_workers.go:965] “Error syncing pod, skipping” err=“failed to "StartContainer" for "nvidia-device-plugin-ctr" with CrashLoopBackOff: "back-off 5m0s restarting failed container=nvidia-device-plugin-ctr pod=nvidia-device-plugin-daemonset-msvff_kube-system(c3cecb55-b447-4bf1-a00b-a5bfe8fb0318)"” pod=“kube-system/nvidia-device-plugin-daemonset-msvff” podUID=c3cecb55-b447-4bf1-a00b-a5bfe8fb0318
Jul 16 09:16:22 truenas k3s[11254]: {“level”:“warn”,“ts”:“2024-07-16T09:16:22.948-0400”,“logger”:“etcd-client”,“caller”:“v3@v3.5.7-k3s1/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc000a4e380/kine.sock”,“attempt”:0,“error”:“rpc error: code = Unknown desc = no such table: dbstat”}