I posted this issue before I started using Immich and I’ve been using it for at least half a year now without issues 
Hey !
I’ve been having the same issue with a quota exceeded for dataset /.system/cores and did not found the answer online.
Deleting the cores in /var/db/system/cores did not do the trick (only around 70MB in there).
BUT I managed to find the solution. The exceeded quota was due to recursive periodic snapshots on the root database. The snapshots includes /.system/cores and if not deleted will bloat the database.
You can verify the snapshot size with (change ROOTDATASET with your actual dataset) :
zfs list -o used,usedbysnapshots,usedbydataset,usedbyrefreservation ROOTDATASET/.system/cores
You should see that most space is taken by snapshots. You should manually delete snapshot on /.system/cores with the truenas WebUI (Dataprotection → snapshots). You can filter using the search bar and delete all relevant snapshots (this path hold no value in being snapshot).
Then you can exclude the child dataset from the snapshot periodic task to prevent this from happening again (Dataprotection → click you periodic snapshot → edit (three dots on the right) ). In the exclude field, add ROOTDATASET/.system/cores (still replacing ROOTDATASET with your actual dataset).
You’re good to go 