TrueNAS Scale - Middlewared fails to load after enabling snapshots 2 days prior

Hello Everyone!

I recently ran into a none boot issue with my TrueNAS scale on Dell hardware. The nas was running quite well for about 3 months and was being used as an active mirror target for production shares. When I enable snapshots on these AD integrated shares, about sometime last night, the TrueNAS Scale seewms to have lost network connectivity. I idrac/console into the server to observe a single harddrive failure that had self-healed as it was showing good in the idrac console. I am using 4TB SSDs non-enterprise.

After my reboot, the middleware fails to load at step 110/111. I was hoping someone could help me get this booted back and disable snapshots until I can figure out what caused it to fail. Thanks!

It seems to be the (audit) plugin!

anyone have any ideas where to start looking first? Thanks!

After further digging, I found that this might be related to the NETDATA log spam bug. Im seeing messages popup on screen for that. Does anyone know how to clean this up via CLI so the middlewared service can boot? Would appreciate any help out there. Thank you.