I underprovisioned ix-applications dataset space. So installing an app resulted in exceeding the dataset allocation.
To my amazement, all my apps stopped running.
The problem was that a key file used by the system was totally clobbered (set to 0 size).
IX won’t fix this bug so I’m noting it here so when it happens to you, you know the fix.
See: NAS-128741 for details if this happens to you.
You can restore the zeroed file from the snapshots I know you are all taking regularly