Quota exceeded on dataset <...>/.system/cores

Karmalakas · December 30, 2024, 10:03am

I remember after updating to one of the later versions I got flooded with alerts (a few emails an hour):

Quota exceeded on dataset <…>/.system/cores. Used 97.86% (1002.13 MiB of 1 GiB)

So I reverted back to the version I was using and waited for another update and alerts disappeared. I was using 24.04 and then upgraded to 24.10.0 and there were no such alerts, but after updating to 24.10.1 these alerts came back and I’m getting them multiple times a week.

It seems I can’t find anything on the topic

Anything I should look into? Should I change some settings?

NugentS · December 30, 2024, 10:58am

Delete the cores

Karmalakas · December 30, 2024, 11:03am

So one of the errors is:

Quota exceeded on dataset Data/.system/cores

~~How do I delete cores? There’s no such folder .system inside Data Do I need to enable something to see it?~~

$ cd /mnt/Data/.system
cd: no such file or directory: /mnt/Data/.system

Update:
Found it with

$ /usr/sbin/zfs list

NAME                                                                       USED  AVAIL  REFER  MOUNTPOINT
Data/.system                                                              2.09G  7.20T  1.47G  legacy
Data/.system/configs-ae32c386e13840b2bf9c0083275e7941                     16.0M  7.20T  16.0M  legacy
Data/.system/cores                                                         127M   897M   127M  legacy
Data/.system/netdata-ae32c386e13840b2bf9c0083275e7941                      483M  7.20T   483M  legacy
Data/.system/nfs                                                           149K  7.20T   149K  legacy
Data/.system/samba4                                                       2.51M  7.20T   687K  legacy

Should I delete the whole .system maybe? My other data pool doesn’t have it at all and mountpoint says legacy

neofusion · December 30, 2024, 11:57am

I wouldn’t advice removing the whole system dataset unless you really had to.

I believe the cores dataset is mounted here:
/var/db/system/cores

Karmalakas · December 30, 2024, 4:14pm

Would it be safe to use this?

$ /usr/sbin/zfs destroy Data/.system/cores

NugentS · December 30, 2024, 6:25pm

No. Its a folder not a dataset

winnielinnie · December 30, 2024, 6:53pm

Aren’t those core dumps? How often do core files get created? I’ve only ever had one core file in my entire time using FreeNAS and TrueNAS Core.

Maybe it’s apt for a bug report?

Alister · May 20, 2025, 12:03am

How?

I can’t cd to it as .system doesn’t exist on the datasat

winnielinnie · May 20, 2025, 12:05am

Because there is no folder named “.system”. That’s the hidden dataset’s name. Datasets can be mounted to any path.

The cores are found under /var/db/system/cores/

Alister · May 20, 2025, 1:00am

There’s none at presnt

TRUENAS-SCALE% pwd
/var/db/system/cores
TRUENAS-SCALE% sudo ls -la
total 33
drwxr-xr-x 2 root root 2 May 20 01:34 .
drwxr-xr-x 9 root root 9 Nov  3  2024 ..
TRUENAS-SCALE%

I managed to see some brieftly before it disappeared earlier due to perodic job


TRUENAS-SCALE% cd /var/db/system/cores/
TRUENAS-SCALE% ls -la
total 384338
drwxr-xr-x 2 root root         3 May 20 01:09 .
drwxr-xr-x 9 root root         9 Nov  3  2024 ..
-rw-r----- 1 root root 393219960 May 20 01:09 core.python.0.e95a7a4259574f679dc5f1e0831d4832.4066500.1747699778000000.zst
TRUENAS-SCALE% ls -la
total 187422
drwxr-xr-x 2 root root         3 May 20 01:10 .
drwxr-xr-x 9 root root         9 Nov  3  2024 ..
-rw-r----- 1 root root 191726646 May 20 01:10 core.python.0.e95a7a4259574f679dc5f1e0831d4832.4067678.1747699804000000.zst
TRUENAS-SCALE% ls -la

is there any way to increase the quota? so I don’t keep getting the alerts - it was fixed but having it set at 1GiB seems to small

awalkerix · May 20, 2025, 1:10am

Generally it’s fine. Corefiles are very compressible and you shouldn’t be generating them. Usually their presence in a stable release indicates:

broken custom application
failing boot device
unexpected OS changes

caramel-juni · July 4, 2025, 1:24am

I’m also getting this issue periodically,
Quota exceeded on dataset tank/.system/cores. Used 98.02% (1003.77 MiB of 1 GiB) - yet not on a pool that manages booting or anything.

For me it seems to be qbit (?) which seems to stay relatively stable… although am monitoring whether it grows, or whether it was another app that’s now been cleared.

juni-nas@truenas /var/db/system/cores
 % ls -lahs
total 70M
33K drwxr-xr-x 2 root root   3 Jul  4 08:59 .
11K drwxr-xr-x 8 root root   8 Jan 28 21:29 ..
70M -rw-r----- 1 root root 70M Jul  4 08:59 core.qbittorrent-nox.568.c852a959f53d41948751b3d70a3bc621.1287972.1751585243000000.zst

Did anyone find a way to increase core size, or troubleshoot what this means/what is generating them? I’m using mainly trueNAS-store apps, so not a heap of custom ones (e.g. the one showing up, qbit, is a TrueNAS store app), so I thought they’d play nice with the system cores given they’re purpose-built for it… has anyone had any success in why apps are generating/behaving like this?

So, if it’s not the app (which i feel it may be…), would be a tad worried about the other two options you mentioned ( 2. failing boot device, 3. unexpected OS changes) and thought I’d ask if you had any advice on where to go next to troubleshoot/diagnose these as they’re both quite vague. I haven’t had boot issues or really any signs of system-level issues - my disk checks are all good, routine scrubbing for boot drive, data drives, etc…

Am booting off a single NVMe 476GB SSD… but the dataset that is throwing the space concerns, tank of tank/.system/cores, is referring to my 1 x RAIDZ1 | 3 wide | 21.68 TiB HDD pool which I use for data storage, so I don’t see how it would be a boot issue?

Just a bit confuzzled.

Prelude · July 25, 2025, 3:32am

Did you ever figure out what was causing this issues? i only have 2 apps on my TrueNAS - Immich and JellyFin and i have been getting this alert non-stop

Karmalakas · July 25, 2025, 10:24am

For me it’s been a while since I had this error, but I still get them occasionally

Prelude · July 25, 2025, 2:03pm

i noticed that i got the alerts when i installed immich and tried to use CUDA machine learning. my video card in my server is a GTX980 i think it maybe be under powered or too old for Machine learning so i did a re-install not using the CUDA machine learning and changed the temp file location for machine learning to a separate Dataset. So far so good. i am still really new at Unix and TrueNAS.