TrueNAS fails catastrophically when boot-pool fills to 100% with no protection or warnings

TrueNAS General
1

TrueNAS newbie here…created this situation with some help from Claude. Probably an idiot move, but still probably shouldn’t have happened. Thanks, Tom

Severity: Critical - System becomes unbootable

Description: When boot-pool filesystem fills to 100%, TrueNAS becomes unbootable with cascading service failures and no recovery mechanism.

Steps to Reproduce:

  1. Create directory in /mnt (e.g., mkdir /mnt/temp-ssd)

  2. Write large files to this directory before mounting external device

  3. Files write to boot-pool filesystem instead of intended external mount

  4. Boot-pool fills to 100%

  5. System crashes and cannot boot properly

Expected Behavior:

  • Warning alerts when boot-pool reaches 80%, 90%, 95% capacity

  • Quota limits preventing user operations from filling boot-pool

  • Reserved space for critical system operations

  • Graceful service degradation instead of catastrophic failure

  • System remains bootable even if boot-pool is full

Actual Behavior:

  • No warnings before failure

  • Boot-pool dataset (boot-pool/ROOT/*/mnt) consumed 212GB

  • Services fail to start (middleware, apps, etc.)

  • System unbootable until manual dataset destruction

  • No automated recovery mechanism

Impact: Complete system failure requiring manual intervention and potential data loss risk.

Environment:

  • TrueNAS SCALE 25.10.1 - Goldeye

  • Boot-pool size: 222GB

2

While I 100% agree that there should be warning as your boot pool fills-up, I’m curious if there would be a realistic situation outside of someone doing it on purpose.

There are alert settings for regular pools, but I’ve never even considered if they’d also be set for the boot pool.

Anyway - the fix is surprisingly simply; re-install TrueNAS & then import your old config file. All the junk that was put on boot pool would be cleaned on fresh install. I’ve had to do clean installs many times because I also screw around with things I shouldn’t & have never had any problems with getting any data back from my non-boot pools.

You really shouldn’t need anything outside a few KBs of scripts, or a couple GBs for older boot versions on your boot pool. Outside of that, I’d argue that there is nothing else that should be on the boot pool other than the OS. (I’m sure someone smarter will think of something reasonable, but even then it’d maybe be a few GBs).

If you want to be very safe, you can shut down the system, unplug any non-boot drives from sata data & power, then connect your usb for install - this’d make sure you don’t write over any data from your actual pools.

Don’t use the boot pool for general storage purposes - TrueNAS is rather different from most OS in that regards.

Edit: I mean, you could also boot into another OS that can speak zfs & then manually remove some crap from the truenas boot pool & get it back to <80% full, but that seems like more of a hassle than clean installing TrueNAS unless you carry a portal debian boot drive around (yes I know, someone here just might).

Edit 2: it might be worth creating a post in the ‘Feature Requests’ category to have storage alerts on the boot pool - I can imagine this being overlooked originally.

3

I appreciate the workaround suggestions, but I want to clarify the failure scenario:

What happened:

  1. Created /mnt/temp-ssd as a temporary mount point

  2. Mounted external SSD there… or so I thought

  3. Ran rsync to copy files

  4. The mount failed silently, so rsync wrote 212GB to /mnt/temp-ssd on the boot-pool filesystem instead of the external drive

  5. Boot-pool filled to 100%, system became unbootable

This wasn’t “using boot-pool for storage on purpose” - it was a failed mount combined with no safeguards.

The bug is multi-layered:

  1. No warnings as boot-pool filled (80%, 90%, 95%)

  2. No quota preventing user operations from consuming boot-pool

  3. No graceful degradation - complete system failure

  4. No recovery mechanism - required manual dataset destruction

The fix isn’t just “don’t do that” - it’s:

  • Alert users when boot-pool reaches 80%+ (like regular pools)

  • Prevent user operations from filling boot-pool beyond safe threshold

  • Reserve emergency space for critical operations

  • Keep system bootable even if boot-pool is full

Yes, reinstalling TrueNAS works, but requiring a clean OS reinstall for a disk-full condition is a design flaw, not a feature.

I’ll create a Feature Request for boot-pool storage alerts as suggested

4

Out of curiosity, what were you trying to do?

Copying data directly from the NAS to an external (usb?) ssd drive?

5

We used to see it a lot (well, more than we have recently) with misconfigured FTP placing stuff in /root.

There is no bug.

Really, yes, it is. As TrueNAS warns you every time you log in to the shell,

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

Not to say your suggestions aren’t valid, but you’re using the system in a way it isn’t intended to be used, and it isn’t especially surprising that this has resulted in a bad outcome.

2 Likes
6

Don’t trust AI without verify, and dont take what It say as good as argumentation , at least half of your claims are really baseless.
Even though I also think the warning about space usage in the boot pool can be useful, it wouldn’t have saved you anyway… you probably wouldn’t have had time to stop the process.

You lose a boot pool following AI, but we see a lot worst happening to other user. Take that as experience :smile:

2 Likes
7

You were root at least the moment you created /mnt/temp-ssd. Pretty much every unix based system allows you to shoot yourself in the foot if root is involved. Don’t do that :man_shrugging:

Warnings might be a good idea though.

For some reason /var/log is on the boot-pool so there is some remote possibility that something going haywire overflows the pool before logrotate rotates it away. I don’t think anything should write to the boot-pool except when configuration changes are needed to be honest, but the devs have clearly a different opinion.

1 Like