Quota exceeded on dataset <...>/.system/cores

TrueNAS General
,
1

I remember after updating to one of the later versions I got flooded with alerts (a few emails an hour):

Quota exceeded on dataset <…>/.system/cores. Used 97.86% (1002.13 MiB of 1 GiB)

So I reverted back to the version I was using and waited for another update and alerts disappeared. I was using 24.04 and then upgraded to 24.10.0 and there were no such alerts, but after updating to 24.10.1 these alerts came back and I’m getting them multiple times a week.

It seems I can’t find anything on the topic :confused:

Anything I should look into? Should I change some settings?

1 Like
2

Delete the cores

3

So one of the errors is:

Quota exceeded on dataset Data/.system/cores

How do I delete cores? There’s no such folder .system inside Data :confused: Do I need to enable something to see it?

$ cd /mnt/Data/.system
cd: no such file or directory: /mnt/Data/.system

Update:
Found it with

$ /usr/sbin/zfs list

NAME                                                                       USED  AVAIL  REFER  MOUNTPOINT
Data/.system                                                              2.09G  7.20T  1.47G  legacy
Data/.system/configs-ae32c386e13840b2bf9c0083275e7941                     16.0M  7.20T  16.0M  legacy
Data/.system/cores                                                         127M   897M   127M  legacy
Data/.system/netdata-ae32c386e13840b2bf9c0083275e7941                      483M  7.20T   483M  legacy
Data/.system/nfs                                                           149K  7.20T   149K  legacy
Data/.system/samba4                                                       2.51M  7.20T   687K  legacy

Should I delete the whole .system maybe? My other data pool doesn’t have it at all and mountpoint says legacy :thinking:

4

I wouldn’t advice removing the whole system dataset unless you really had to.

I believe the cores dataset is mounted here:
/var/db/system/cores

2 Likes
5

Would it be safe to use this?

$ /usr/sbin/zfs destroy Data/.system/cores
6

No. Its a folder not a dataset

2 Likes
7

Aren’t those core dumps? How often do core files get created? I’ve only ever had one core file in my entire time using FreeNAS and TrueNAS Core.

Maybe it’s apt for a bug report?

8

How?

I can’t cd to it as .system doesn’t exist on the datasat

9

Because there is no folder named “.system”. That’s the hidden dataset’s name. Datasets can be mounted to any path.

The cores are found under /var/db/system/cores/

10

There’s none at presnt

TRUENAS-SCALE% pwd
/var/db/system/cores
TRUENAS-SCALE% sudo ls -la
total 33
drwxr-xr-x 2 root root 2 May 20 01:34 .
drwxr-xr-x 9 root root 9 Nov  3  2024 ..
TRUENAS-SCALE%

I managed to see some brieftly before it disappeared earlier due to perodic job


TRUENAS-SCALE% cd /var/db/system/cores/
TRUENAS-SCALE% ls -la
total 384338
drwxr-xr-x 2 root root         3 May 20 01:09 .
drwxr-xr-x 9 root root         9 Nov  3  2024 ..
-rw-r----- 1 root root 393219960 May 20 01:09 core.python.0.e95a7a4259574f679dc5f1e0831d4832.4066500.1747699778000000.zst
TRUENAS-SCALE% ls -la
total 187422
drwxr-xr-x 2 root root         3 May 20 01:10 .
drwxr-xr-x 9 root root         9 Nov  3  2024 ..
-rw-r----- 1 root root 191726646 May 20 01:10 core.python.0.e95a7a4259574f679dc5f1e0831d4832.4067678.1747699804000000.zst
TRUENAS-SCALE% ls -la

is there any way to increase the quota? so I don’t keep getting the alerts - it was fixed but having it set at 1GiB seems to small

11

Generally it’s fine. Corefiles are very compressible and you shouldn’t be generating them. Usually their presence in a stable release indicates:

  1. broken custom application
  2. failing boot device
  3. unexpected OS changes
1 Like
Quota Exceeded On DataSet after uploading a lot of files at once
12

I’m also getting this issue periodically,
Quota exceeded on dataset tank/.system/cores. Used 98.02% (1003.77 MiB of 1 GiB) - yet not on a pool that manages booting or anything.

For me it seems to be qbit (?) which seems to stay relatively stable… although am monitoring whether it grows, or whether it was another app that’s now been cleared.

juni-nas@truenas /var/db/system/cores
 % ls -lahs
total 70M
33K drwxr-xr-x 2 root root   3 Jul  4 08:59 .
11K drwxr-xr-x 8 root root   8 Jan 28 21:29 ..
70M -rw-r----- 1 root root 70M Jul  4 08:59 core.qbittorrent-nox.568.c852a959f53d41948751b3d70a3bc621.1287972.1751585243000000.zst

Did anyone find a way to increase core size, or troubleshoot what this means/what is generating them? I’m using mainly trueNAS-store apps, so not a heap of custom ones (e.g. the one showing up, qbit, is a TrueNAS store app), so I thought they’d play nice with the system cores given they’re purpose-built for it… has anyone had any success in why apps are generating/behaving like this?

So, if it’s not the app (which i feel it may be…), would be a tad worried about the other two options you mentioned ( 2. failing boot device, 3. unexpected OS changes) and thought I’d ask if you had any advice on where to go next to troubleshoot/diagnose these as they’re both quite vague. I haven’t had boot issues or really any signs of system-level issues - my disk checks are all good, routine scrubbing for boot drive, data drives, etc…

Am booting off a single NVMe 476GB SSD… but the dataset that is throwing the space concerns, tank of tank/.system/cores, is referring to my 1 x RAIDZ1 | 3 wide | 21.68 TiB HDD pool which I use for data storage, so I don’t see how it would be a boot issue?

2880×1176 268 KB

Just a bit confuzzled.

13

Did you ever figure out what was causing this issues? i only have 2 apps on my TrueNAS - Immich and JellyFin and i have been getting this alert non-stop

14

For me it’s been a while since I had this error, but I still get them occasionally :man_shrugging:

15

i noticed that i got the alerts when i installed immich and tried to use CUDA machine learning. my video card in my server is a GTX980 i think it maybe be under powered or too old for Machine learning so i did a re-install not using the CUDA machine learning and changed the temp file location for machine learning to a separate Dataset. So far so good. i am still really new at Unix and TrueNAS.

17

If anyone is still struggling with this I found a fix.

For me I was getting the same error

Quota exceeded on dataset Vault/.system/cores.

So first I went to the shell and logged into root user with

su -

Then I ran

zfs list

Under names I located Vault/.system/cores and saw that I had 813M used and Refer was 1G.

So then I used the command

zfs set quota=10G Vault/.system/cores

Then my quota was at 10gb.

So what I think was happening is that even thought I deleted the core dumps in that folder since I have periodic snapshots enabled I don’t get the data back until the snapshots expire in turn using up my 1G. So I just settled to increasing quota size to 10G and making the snapshots expire a bit sooner.

I hope this helps.

18

That’s not a solution. The question is why your cores dataset even exceeds 200KB usage?

There should never be anything inside. If it’s ballooning, then it means you have an issue that must be investigated.

2 Likes
19

What should I do to investigate?

20

You can check your notifications, log files, drive health, and pool status.

If you submit a debug attachment and core dump to a TrueNAS dev (on their Jira) they can probably pinpoint the culprit.

21

I have been having the same issue with the alert “Quota exceeded on dataset Pool/.system/cores. Used 100.00% (1 GiB of 1 GiB)”. It occurred after installing Immich app and imported around 15K photos so I am sure Immich is causing the issue and consistent with earlier posters.
All drives, shares datasets etc are showing healthy with no issues.