Recently we got hacked. Without going into a lot of details, some pondscum got in and wiped us out. They encrypted our infrastructure and then took out the Truenas NFS data store and encrypted the SMB share. The later I can understand but they completely wiped out the NFS datastore…gone…bye bye (including snapshots). We got most back from from other data I had saved.
Lets just say that getting repeated phone calls from these people is disconcerting.
There are weird entries in the syslog around the time of entry.
The $64,000 question is, other then how they got in, how do I harden Truenas and prevent it?
What @Stux said, as well as, did you check the audit logs?
If you’re running a current version of SCALE you will have audit logs that can tell you if they logged in to the server.
There’s a chance they got control over a computer with saved admin credentials to the TN box (either saved in browser or ssh private key).
Im really sorry to hear this. As the others have said it’s hard to understand how this could have happened without admin/root access to the TrueNAS itself be it via the UI or SSH.
Although it won’t be helpful in this instance yourself and others may want to consider using zpool checkpoint to periodically take pool wide checkpoints. This not only protects you against accidental deletion of datasets and snapshots but also helps protect against malicious attacks like this one. In this instance you would be able to rewind the pool to the previous checkpoint before the attack happened be it temporarily in readonly mode to recover the data or permanently read/write.
You can currently only have one checkpoint at a time and this is currently not built into the UI so needs to be scheduled via CRON but the commands are simple enough. Perhaps consider taking one every week or month or perhaps even longer depending on your data flow and then release the old one and take a new one a minute later.
Would love to see this functionality brought into the TrueNAS UI at some point and perhaps be an auto default upon pool creation a bit like scrubs. Would save quite a few people I bet.
The thing is that it is harden by default. The question is: how did it get soften.
Treat it as if you had to access it from a secure bunker.
I would have a user that is exclusive for the function.
The browser would be a portable one on a folder that is encrypted for that user only.
That’s the way I did it for sensitive stuff. If someone gets my laptop and clears the password to gain access, everything on that encrypted folder is gone.
Obviously, you login to that user for sensitive stuff. Use your regular user for your other uses.
For more hardening, you’ll have to share your environment better than “don’t mind how it happened”.
TrueNAS is not hardened such that it can withstand being in a DMZ or direct access from the internet. While there are some security features, it’s not intended to be placed in such a situation.
If the OP’s system wasn’t accessible directly from the internet, the next likely vector would be that the intruder gained access through the computer used to manage the network (or at least manage the TN-box).
Lastly, there are ways of getting in even if all you have is access to the network the server is on, if you’re skilled and motivatedenough.
Sorry for the long delay. Its been a terrible week gettign it all back.
Various questions people asked:
No…I did not expose the TrueNAS GUI or SSH to the internet.
I don’t see any odd entries in the audit logs around the time it happened except this:
middlewared.log
[2024/10/15 01:58:09] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:09:04] (ERROR) ServiceService.reload():369 - Service ‘ssh’ not running after reload
[2024/10/15 02:12:54] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:14:26] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:17:15] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:17:33] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:19:29] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:20:25] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:20:54] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:22:29] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:36:56] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:40:22] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
[2024/10/15 02:41:23] (e[1;33mDEBUGe[1;m) ZFSDatasetService.add_name():125 - Unable to resolve group id xxxxxxxxxxx to name
I have now added 2fa in the TrueNAS login.
I have removed password login to ssh (as well as any ssh access).
I don’t understand the zpool check point suggestion.
The #$%^& jerks completely wiped out the entire NFS dataset. That served to wipe out the snapshots.
Again, I apologize for not better explaining the environment when I first wrote this.
In the end, Truenas serves up an NFS target for backups.
First of all, sorry you had to go through all that pain. So happy to hear your backup strategy worked at least to some extent. Now it’s on to recovery.
Without elaborating details though, it is really hard to opine on how to do better. To what extent did you expose and or harden the NAS? For example,
did the WebGUI get protected by SSL? How about 2FA?
What level of protection did SMB enjoy, ie did you force SMB2+ authentication and encryption?
Did you use passwordless SSH authentication?
Was the IPMI inaccessible to folk on your network?
Have you eliminated the possibility that the admin computer was/is infected, keylogged, and that it gave up the keys to the kingdom?
were any passwords reused, especially between shares and admin accounts to the machine?
Which version of TrueNAS do you actually run? I don’t see it mentioned anywhere.
When you mention that you couldn’t find anything in the audit logs, did you specifically look at the Audit information in the GUI or did you “just” audit the logs?
I ask because SCALE has information hidden under “System Settings” → Audit", and looking at that will tell you when certain logins (to the GUI & SMB happened).
/var/log/samba4/ also contains some logs (for example log.smbd) that might show interesting events.
SMB protocol can’t be used to destroy ZFS datasets. Ditto for NFS protocol. There’s no libzfs plumbing for this in Samba or in knfsd.
zpool history can perhaps give some information about how datasets / snapshots were destroyed. Changes initiated by py-libzfs have a py-libzfs prefix assuming this is SCALE. That said, I don’t think TrueNAS version is listed here. 13.0 / 13.3 have a samba authentication log in /var/log/samba4. SCALE starting in 24.04 has SMB authentication logs presented in UI Services page.
My reasoning when it comes to SMB was that since the contents of the SMB share was encrypted, did something get logged when it did? It could happen from an authenticated SMB client. It could also have happened using shell access from the TrueNAS of course.
