Issue description
- VMs will run fine then randomly go from showing status of running in Truenas Instance Menu to a blank status with no status showing at all and when I attempt to remote into them with Remote Desktop I am unable to connect and when I go to shut them down in the GUI I get this message and must forcefully shut down the VM in order for it to actually stop running and restart. Once restarted normal operation resumes until the next time the issue reoccurs (typically within a 24 to 48 hour time period) pictured below are some screenshots from event viewer which I strongly believe to be related to or the direct culprit of these issues.
Error message screenshot
Error message text
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 515, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 560, in __run_body
rv = await self.method(*args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 88, in wrapped
result = await func(*args)
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/virt/instance.py", line 563, in stop
await incus_call_and_wait(f'1.0/instances/{id}/state', 'put', {'json': {
File "/usr/lib/python3/dist-packages/middlewared/plugins/virt/utils.py", line 98, in incus_call_and_wait
return await incus_wait(result, running_cb, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/virt/utils.py", line 83, in incus_wait
await asyncio.wait_for(task, timeout)
File "/usr/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/middlewared/plugins/virt/websocket.py", line 108, in wait
raise CallError(data)
middlewared.service_exception.CallError: [EFAULT] The instance cannot be cleanly shutdown as in Error status
TrueNAS bare metal hardware information
OS Version:25.04-RC.1
Product:Z690 AORUS ULTRA
Model:Intel(R) Core™ i7-14700K
Memory:126 GiB
Virtual machine information
Windows Server 2022 for the purposes of running an Ark Ascended Server
As seen from the above images I also have a general stability issue that I haven’t been able to pin down, it seems its related to kernel power as this also happens in my other VM running Windows Server 2025 that is only used for extremely light workloads and produces the same behavior regarding the system kernel power event viewer messages and the inability to restart the VM in the GUI of TrueNAS Scale due to it being in “Error status”
Im sure there is more information that I could provide to make diagnosing this easier however that is out of the purview of my current knowledge,
Thank you for your time.
Can you post the output of these commands?
grep qemu /var/log/messages
grep qemu /var/log/error
Output for grep qemu /var/log/messages
Mar 22 20:10:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45285 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45289 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45284 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45290 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45286 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45288 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:39 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45631 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:39 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45636 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:39 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45630 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:39 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45632 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:39 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45635 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:39 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45634 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:10:39 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/45633 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:13:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/46919 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:13:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/46916 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:13:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/46917 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:13:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/46918 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:13:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/46915 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:13:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/46920 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:13:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/46914 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:13:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/46908 took a split_lock trap at address: 0xfffff8079d6bd79c
Mar 22 20:18:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/51706 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:18:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/51708 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:18:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/51707 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:18:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/51710 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:18:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/51709 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:18:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/51711 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:18:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/51712 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:24:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/57035 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:24:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/57039 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:24:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/57037 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:24:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/57040 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:24:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/57036 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:24:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/57038 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:24:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/57034 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:25:07 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/58456 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:25:07 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/58458 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:25:07 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/58455 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:25:07 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/58459 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:31:59 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/65006 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:31:59 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/65005 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:31:59 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/65003 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:31:59 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/65007 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:31:59 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/65004 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:31:59 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/65002 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:36:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/70395 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:36:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/70394 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:36:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/70396 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:36:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/70393 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:36:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/70392 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:36:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/70390 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:46:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/7084 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:46:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/7091 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:46:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/7082 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:46:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/7087 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:46:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/7085 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:46:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/7089 took a split_lock trap at address: 0x7fe7d050
Mar 22 20:46:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/7088 took a split_lock trap at address: 0x7fe7d050
Mar 22 21:00:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/6975 took a split_lock trap at address: 0x7fe7d050
Mar 22 21:00:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/6973 took a split_lock trap at address: 0x7fe7d050
Mar 22 21:00:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/6972 took a split_lock trap at address: 0x7fe7d050
Mar 22 21:00:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/6970 took a split_lock trap at address: 0x7fe7d050
Mar 22 21:00:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/6974 took a split_lock trap at address: 0x7fe7d050
Mar 22 21:00:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/6978 took a split_lock trap at address: 0x7fe7d050
Mar 22 21:00:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/6977 took a split_lock trap at address: 0x7fe7d050
Mar 22 21:22:21 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/6949 took a split_lock trap at address: 0xfffff806b5ebd79c
Mar 23 06:38:09 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/209683 took a split_lock trap at address: 0x7fe7d050
Mar 23 06:38:09 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/209681 took a split_lock trap at address: 0x7fe7d050
Mar 23 06:38:09 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/209679 took a split_lock trap at address: 0x7fe7d050
Mar 23 06:38:09 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/209678 took a split_lock trap at address: 0x7fe7d050
Mar 23 06:38:09 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/209684 took a split_lock trap at address: 0x7fe7d050
Mar 23 06:38:09 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/209682 took a split_lock trap at address: 0x7fe7d050
Mar 23 06:38:09 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/209680 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:22:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/376299 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:22:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/376300 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:22:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/376298 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:22:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/376296 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:22:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/376297 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:22:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/376295 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:22:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/376294 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:24:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/376289 took a split_lock trap at address: 0xfffff802a60bd79c
Mar 23 12:31:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/380489 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:31:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/380492 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:31:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/380493 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:31:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/380491 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:31:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/380488 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:31:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/380487 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:32:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/381210 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:32:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/381214 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:32:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/381213 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:32:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/381208 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:32:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/381209 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:32:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/381212 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:32:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/381211 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383735 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383740 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383726 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383741 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383743 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383744 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383742 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383737 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383739 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:08 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/383738 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384132 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384133 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384150 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384144 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384148 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384147 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384146 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384137 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384138 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384134 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:37:35 truenas kernel: qemu-system-x86 invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Mar 23 12:37:35 truenas kernel: CPU: 7 UID: 65534 PID: 384134 Comm: qemu-system-x86 Tainted: P OE 6.12.15-production+truenas #1
Mar 23 12:37:35 truenas kernel: [ 381192] 65534 381192 2696668 2121552 67222 362 2053968 18190336 0 0 qemu-system-x86
Mar 23 12:37:35 truenas kernel: [ 384123] 65534 384123 17359994 14790167 16194 557 14773416 119947264 0 0 qemu-system-x86
Mar 23 12:37:35 truenas kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=incus.service,mems_allowed=0,global_oom,task_memcg=/system.slice/incus.service,task=qemu-system-x86,pid=384123,uid=65534
Mar 23 12:38:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384971 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:38:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384969 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:38:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384972 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:38:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384973 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:38:26 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/384970 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385538 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385550 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385548 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385546 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385553 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385552 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385537 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385541 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385536 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:39:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385547 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:41:20 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/385531 took a split_lock trap at address: 0xfffff80453495876
Mar 23 12:44:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/388571 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:44:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/388572 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:44:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/388568 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:44:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/388569 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:44:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/388570 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:44:17 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/388573 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389539 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389541 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389535 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389524 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389542 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389533 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389526 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389529 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389530 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:45:33 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389532 took a split_lock trap at address: 0x7fe7d050
Mar 23 12:46:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/389518 took a split_lock trap at address: 0x77772cf5
Mar 23 12:50:40 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392286 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:40 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392291 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:40 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392287 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:40 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392289 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:40 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392290 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:40 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392288 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392379 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392380 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392383 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:50:41 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392384 took a split_lock trap at address: 0x7ff1e050
Mar 23 12:51:27 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/392285 took a split_lock trap at address: 0x77d72cf5
Mar 23 15:07:43 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8060 took a split_lock trap at address: 0x7ff1e050
Mar 23 15:07:43 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8069 took a split_lock trap at address: 0x7ff1e050
Mar 23 15:07:43 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8073 took a split_lock trap at address: 0x7ff1e050
Mar 23 15:07:44 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8954 took a split_lock trap at address: 0x7ff1e050
Mar 23 15:07:44 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8960 took a split_lock trap at address: 0x7ff1e050
Mar 23 15:07:44 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8955 took a split_lock trap at address: 0x7ff1e050
Mar 23 15:07:44 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8959 took a split_lock trap at address: 0x7ff1e050
Mar 23 15:08:18 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8055 took a split_lock trap at address: 0x775a2cf5
Mar 23 18:43:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/133742 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:43:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/133743 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:43:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/133738 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:43:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/133741 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:43:00 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/133737 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:43:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/134233 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:43:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/134230 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:43:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/134232 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:43:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/134228 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:55 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139175 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:55 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139180 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:55 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139178 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:55 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139177 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:55 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139176 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:56 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139266 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:56 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139270 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:56 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139271 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:56 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139268 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:49:56 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139267 took a split_lock trap at address: 0x7ff1e050
Mar 23 18:50:29 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/139174 took a split_lock trap at address: 0x76fc2cf5
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156805 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156796 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156804 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156807 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156808 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156811 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156813 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156810 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156797 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:20:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/156800 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158079 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158081 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158085 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158095 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158096 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158089 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158094 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158092 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158080 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:22:12 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158091 took a split_lock trap at address: 0x7fe7d050
Mar 23 19:30:16 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/158073 took a split_lock trap at address: 0xfffff804cd6aafac
Mar 24 08:55:53 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/640661 took a split_lock trap at address: 0x7ff1e050
Mar 24 08:55:53 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/640667 took a split_lock trap at address: 0x7ff1e050
Mar 24 08:55:53 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/640665 took a split_lock trap at address: 0x7ff1e050
Mar 24 08:55:53 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/640664 took a split_lock trap at address: 0x7ff1e050
Mar 24 08:56:39 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/640660 took a split_lock trap at address: 0x771b2cf5
Mar 24 09:00:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/644645 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:00:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/644642 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:00:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/644644 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:00:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/644647 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:00:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/644648 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:00:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/644643 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:00:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/644646 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:04:36 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8619 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:04:36 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8623 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:04:36 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8620 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:04:36 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8625 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:04:36 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8627 took a split_lock trap at address: 0x7ff1e050
Mar 24 09:04:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/9380 took a split_lock trap at address: 0x7fe7d050
Mar 24 09:04:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/9381 took a split_lock trap at address: 0x7fe7d050
Mar 24 09:04:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/9363 took a split_lock trap at address: 0x7fe7d050
Mar 24 09:04:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/9374 took a split_lock trap at address: 0x7fe7d050
Mar 24 09:04:38 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/9375 took a split_lock trap at address: 0x7fe7d050
Mar 24 09:05:19 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/8618 took a split_lock trap at address: 0x77db2cf5
Mar 24 10:45:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/63023 took a split_lock trap at address: 0x7ff1e050
Mar 24 10:45:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/63025 took a split_lock trap at address: 0x7ff1e050
Mar 24 10:45:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/63024 took a split_lock trap at address: 0x7ff1e050
Mar 24 10:45:48 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/63027 took a split_lock trap at address: 0x7ff1e050
Mar 24 10:47:29 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/63020 took a split_lock trap at address: 0x77cf2cf5
Mar 24 15:05:24 truenas kernel: qemu-system-x86[9354]: segfault at 2110 ip 0000555927848e40 sp 00007f72803f4f70 error 4 in qemu-system-x86_64[51ce40,555927732000+597000] likely on CPU 3 (core 4, socket 0)
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229433 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229432 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229436 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229445 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229443 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229444 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229434 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229437 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229427 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:43:51 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229428 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:48:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/229420 took a split_lock trap at address: 0xfffff800d92aafac
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234395 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234400 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234393 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234408 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234410 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234409 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234405 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234402 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234397 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:13 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234406 took a split_lock trap at address: 0x7fe7d050
Mar 24 15:51:59 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/234388 took a split_lock trap at address: 0x772d2cf5
Mar 26 18:44:54 truenas kernel: CPU: 3 UID: 65534 PID: 229420 Comm: qemu-system-x86 Tainted: P OE 6.12.15-production+truenas #1
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997944 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997943 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997939 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997940 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997956 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997957 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997955 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997941 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997947 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:05 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/3997945 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:13:34 truenas kernel: qemu-system-x86 invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Mar 26 19:13:34 truenas kernel: CPU: 7 UID: 65534 PID: 234385 Comm: qemu-system-x86 Tainted: P W OE 6.12.15-production+truenas #1
Mar 26 19:13:35 truenas kernel: [ 234385] 65534 234385 18380856 16835860 60459 153 16775248 136400896 0 0 qemu-system-x86
Mar 26 19:13:35 truenas kernel: [3997927] 65534 3997927 17571036 13045547 24315 0 13021232 106840064 0 0 qemu-system-x86
Mar 26 19:13:35 truenas kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=incus.service,mems_allowed=0,global_oom,task_memcg=/system.slice/incus.service,task=qemu-system-x86,pid=234385,uid=65534
Mar 26 19:13:36 truenas kernel: oom_reaper: reaped process 234385 (qemu-system-x86), now anon-rss:0kB, file-rss:612kB, shmem-rss:61722096kB
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005325 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005316 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005318 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005317 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005327 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005326 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005319 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005323 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005320 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:02 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005321 took a split_lock trap at address: 0x7fe7d050
Mar 26 19:16:49 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005311 took a split_lock trap at address: 0x776d2cf5
Mar 26 19:22:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/4005505 took a split_lock trap at address: 0xfffff807cf4aafac
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718524 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718512 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718528 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718526 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718513 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718525 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718514 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718520 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718511 took a split_lock trap at address: 0x7fe7d050
Mar 27 15:25:28 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718517 took a split_lock trap at address: 0x7fe7d050
Mar 27 18:06:57 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2718495 took a split_lock trap at address: 0xfffff8041085590f
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336885 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336874 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336873 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336884 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336875 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336878 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336880 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336886 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336890 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:57:24 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336882 took a split_lock trap at address: 0x7fe7d050
Mar 28 15:58:11 truenas kernel: x86/split lock detection: #AC: qemu-system-x86/2336866 took a split_lock trap at address: 0x77c62cf5
Output for grep qemu /var/log/error
Mar 23 12:37:35 truenas kernel: Out of memory: Killed process 384123 (qemu-system-x86) total-vm:69439976kB, anon-rss:64776kB, file-rss:2228kB, shmem-rss:59093664kB, UID:65534 pgtables:117136kB oom_score_adj:0
Mar 24 15:06:49 truenas systemd-coredump[208093]: Cannot store coredump of 9354 (qemu-system-x86): Disk quota exceeded
Mar 24 15:06:49 truenas systemd-coredump[208093]: Process 9354 (qemu-system-x86) of user 65534 terminated abnormally without generating a coredump.
Mar 26 19:13:35 truenas kernel: Out of memory: Killed process 234385 (qemu-system-x86) total-vm:73523424kB, anon-rss:241836kB, file-rss:612kB, shmem-rss:67100992kB, UID:65534 pgtables:133204kB oom_score_adj:0
It seems you are running out of memory which is causing the crash, as seen in the /var/log/error output. The kernel killed the VM so as to not crash the whole server. I see you allocated 64GB to this VM and have 128GB of RAM total. This is alot of RAM allocation. I would start by reducing to 48G or 32G.
Do you have any other VMs, Apps, Sharing services running on the NAS? I suspect you may just be over-allocating memory.
This also might be the reason you are getting the split lock detection messages, but I am not sure. I don’t have a 14th gen CPU to test with. This is a very new CPU and it’s possible that it may also be contributing to some issues, but I don’t think it has anything to do with the crashes.
I have some other services running but in total they only consume around 20gb of ram at the absolute most. However I will reduce to 32gb and observe stability to see if that resolves the issue.
Ran memtest86 as I had plenty of RAM left over and was definitely not over utilizing my RAM but was still experiencing instability issues. memtest86 ran fine over night with no issues and passed however when I enabled xmp again (like I had been running the machine the entire time) I ran into immediate errors, disabled xmp and now the VM instability, random pool data degradation notifications and overall instability is now gone. I believe this is due to the fact that I am now running with 4 DIMM slots full and the memory controller on my motherboard doesn’t jive well with xmp profile being enabled and all that RAM.
