A similar post on the forum the other day made me think about the fact I needed to investigate what is slowly eating space on the server. It turns out it is docker containers. Specifically, the official app containers as I don’t have any custom containers. The log files grow silently, continuously and with no limits.
Is there a setting I failed to see or find when setting up the apps to clear or rotate docker container logs in Truenas CE? I found that I have one official app container that is generating an ever-expanding log file which is currently at 6.5GB. At the moment that is what is silently eating disk space.
Reading up on Docker in general, I found that log files don’t rotate or get limited in size and that there is no default setting either globally or in containers to do so. (IMO - that’s just stupid). Which means log limits and rotations need to be manually added.
If you’re using Docker, container logs can silently consume a significant amount of disk space over time. By default, Docker uses the json-file log driver, which stores logs in large, ever-growing files unless configured otherwise.
Now there are suggestions all over the place to add to the /etc/docker/daemon.json a code block to limit and rotate log files globally. Great, I get to muck around with something I’d rather not have to worry about that probably should already be in the Truenas CE app config setups.
Except I found vague references to if I manually add the code to the json file then on the next update or migration the edit(s) will disappear. I also found an older suggestion (closed) to add log limits/rotations which was denied as unnecessary?
Just out of curiosity, I found a container with a 2.5G log file (on my non-truenas docker node). I’ve redeployed the stack with max-size: 5M. After reaching 4.8M in size, <container-hash>-json.log became <container-hash>-json.log1. Perhaps, I should also have specified max-file.
This doc states that the default logging setting can be changed at docker’s daemon.json. Not sure on how to do it properly in TrueNAS, though; perhaps post/pre-init magic can save the day.
Thanks. I was surprised not knowing much about Docker in general or how the developers implemented the docker apps I thought I was missing something obvious.
I was going to use this globally which sets a max log size and also rotates the logs so there will be only 3 log files for each container. It would go in the /etc/docker/daemon.json which was the file suggested to add the lines to for global log control.
This would be something good I think to add to Apps → Configuration → Settings to set the log size and set the max log files globally for the official apps. If not set then it defaults to what it does now - silently eat drive space. If not then maybe their should be an option to add it individually to each app as parameters during app setup so it is more flexible in how big a log and how many log files one wants for each app. Default could be some sane number.
The Docker Docs link was the place where I started from on research.
I did some tests on both a Truenas test vm setup and also on a real in production system that is using several Truenas Apps. Short answer is still “No It can’t be done.End of story.” The longer answer is how hard do you want to work at it?
In a normal Docker setup (not Truenas) you could easily add the code to the global /etc/docker/daemon.json file to limit and rotate the logs for the containers, then you just do a sudo systemctl restart docker to get Docker to reread the file. All would be good and the container logs get limited in size and rotated. As shown by @swc-phil, you can also do this individually.
However that won’t work in Truenas at all. It appears that the global /etc/docker/daemon.json file is recreated on restart of the system or a restart of Docker (unset then, set pool to use will restart docker, but I don’t advise that) so a script would need to be run to get file contents, compare then modify if necessary with the extra info.
Then the issue becomes how to get docker to re-read the /etc/docker/daemon.json file to pick up on the addition without a restart. Then a script could be run to handle the file check, addition if necessary, and re-read file and could be set to do so either by cron or by an (post) init script call. Cron would be safest to check because it can be run periodically in case docker crashed or restarted itself for some reason.
FYI I found that rebooting, restarting docker etc. does not destroy the log file or truncate it. If the log file was grown to 40Mb before a reboot, then it will be 40MB after a system reboot and the docker container restarts and the log will continue growing from that point.
One way I think may be to create your own custom docker container and control what is placed in the containers json file to limit and rotate just that containers log file, much like in a normal Docker setup. Then other than using the Truenas Apps as the initial base setup and then converting it to a custom app that you can control, your choice is to just do straight docker apps from whatever place one feels is best.
Another possibility for log control would be to look at manually truncating the individual container log files or set up a script to find all the (running) containers log files then truncate them.
Yeah, it is stated in the doc that the new (global) logging setting would only be applied to the newly created containers. I did test it myself (again, on non-truenas); restarting of the container won’t apply it, but stopping and starting the (portainer) stack will. Aiui, your container has to obtain a new hash, so the old log would be removed.
I found a decent script (not an AI generated script) that was commented well in an article about what each part does that would when given the correct path to the containers list all the container log files, find the size of each, and truncate any over a certain size. There are also examples to truncate log files in the Docker docs.
I did find in the Truenas version of Docker, that manually truncating the log file of a container is possible from a script, but you will have to restart the container from the Truenas GUI after the log was was truncated to get the Truenas Docker System back to a place where it can read the truncated log file again.
If anyone decides to experiment with truncating container files. You must also make sure you have the correct path! Most of the examples I have seen expect the container log files to be in a particular default place so verify the path location is correct. (hint: it is not in Truenas)
I wouldn’t recommend truncating logs in Truenas docker for the reason of:
After truncation, the truncated log file is actually being written to again if you look from the command line, but the Truenas Docker GUI interface will not acknowledge there is anything in the log, throwing up a simple message of there are no log files yet. If you use Portainer to look at the container log that was truncated, then the Portainer log screen just spins when looking for the log file of the container not that the log went anywhere it was just truncated.
None of this makes any real sense at least to me, but truncating a log file might break something. If done on a system you care about, and it breaks something, then the saying of “If you french fry when you should pizza, you will have a bad time.” (South Park) applies.
Welp, imo, 2.5G log for a single container is close to the “now logs are shipped with FedEx”. There is a bit of exaggeration, of course, as in many jokes.
