I’m not really sure what’s going on here. I recently opened up port 22 on my truenas scale box to the Internet so I could transfer some files directly to the box. I thought that I’d hardened sshd, or at least disabled root and password authentication. Then, this morning, I discovered the box had powered off overnight, and when I was able to get it turned back on I found that it’s IP address had changed–possibly because I forgot to set the DHCP server to maintain a constant IP for that box, which I fixed.
Then I discovered that the truenas box was binding a second IP address on the same eno1 interface, and running a sshd (which was allowing cleartext passwords ) on that second IP address (i.e. when I “ssh user@secondIP” I get a password prompt. I dug a little deeper and found that this second IP address apparently showed up about the same time I got the truenas box running, so I don’t think my initial “I’ve been hacked!!” panic was warranted, nor does it seem to be solar flare related. Even weirder is that the command “ip address” shows the main IP address bound to the eno1, but there is no mention whatsoever of the sneaky secret second IP address. I only discovered it thanks to some good network monitoring tools on my switch!
Does anyone know why truenas is, apparently out of the box clean installation, sneakily running a second IP address on the main, and only, connected network interface, not documenting this in “ip address” accessible database, and running an OPEN sshd on this secret IP?
Or am I seeing something else entirely and misreading the situation?
FWIW, I should be on the latest non-beta Train of SCALE. I just clean installed it in mid April. This is my first rodeo with truenas.
Can you show the additional configuration on eno1 you’re talking about?
Yeah… no! Do not do that. Use a VPN connection to gain access to your machine. Best practice would be to run the VPN on your router. Since I have a friend also using a VPN to connect to one of my servers, I run wireguard on virtualized pfsense on my server. Opening a port for wireguard should be very safe from my research.
I don’t know which logs to look for on top of my head but I’m sure there’s a log that should show ssh logins.
Possible.
Also possible.
Until you get to the root of the issue id probably advise you to take down the NAS from your local network.
cat /var/log/auth.log | grep ssh2
If you’re also concerned about someone having logged in, a good place to check is ~/.zsh-histfile and ~/.bash_history for command execution logs
What is this ‘secret’ IP? I’m guessing you can check whether or not it was picked up by DHCP in your network logs (as you mentioned configuring a DHCP server). I can’t 100% remember how to check DHCP client logs in Linux, but cat /var/log/syslog | grep dhcp is probably a decent start.
Thanks y’all. Checking /var/log/auth.log was a good idea. Now I’m almost certain that, despite being HAMMERED by kiddies from mostly China, no one but me ever actually got in. Anyway, that wasn’t the point of this thread. The point is:
Why is truenas creating a second IP on the same eno1 ethernet port, and not recording this in ifconfig/ip address results, nor anywhere else I can think to look, but it is responding to two disparate IP addresses and is only connected to the one LAN via one single ethernet cable.
To answer your question, the main ip is 192.168.1.42, but I can ssh into a password prompt via 192.168.1.173, and I can see this DHCP slot taken by looking at my router’s DHCP table. This is a simplification of the situation.
The only other thing I can think is that maybe my ROUTER is being an idiot and reserving two IPs for the same MAC, routing to both of them, and this isn’t actually a truenas problem… but this has ONLY exhibited itself on the truenas box. Everyone else on the entire network, LAN, WAN, VLANs, etc, gets one IP per MAC.
I was hoping that this would be a “known problem” with a known solution. Thanks anyway!
Maybe a dumb question, but are you sure the 1.173 IP actually is going to your NAS? Because it seems more than a little unlikely that that interface would have that IP, but not report it with ip a.
But I suppose your NAS holds the answer to life, the universe, and everything, so you’d definitely want to be careful with it…
Your router should show the Mac addresses for both reservations. Check that.
I don’t know that you can set the authentication method per Network interface. So I to get it right, on your main IP it’s authentication via ssh keys and on the other via password?
Set a static IP for truenas and reboot your router, see if both IPs are still directing to your NAS.
Does the DHCP server have a MAC associated with that IP that you can see connected to one of the interfaces on your machine?
From TrueNAS, see if it’s aware of it via ip nei sh | grep 192.168.1.173 and see what MAC it returns. If it really is the same MAC as eno1 then it’s very odd, maybe some bug with your DHCP server not properly clearing the initial lease from TrueNAS?
You could also just set the SSH service to listen on 192.168.1.42 specifically, instead of 0.0.0.0.
I found something odd while looking at data on the console. I see that eth0 is set to the primary IP address, with IPv4 configured properly, but two IPv6 addresses that I do NOT want configured. I don’t’ want this machine on IPv6 at all!
Worse, I see that the “fifth ethernet port” which isn’t “eth” anything, but has a 20-ish-character long name, and I assume is the “console port” completely separate from the other eth0-eth3 on the back of the box, has a mysterious IP (private IP when dhcps fail I think 169.254.something). When I disable this fifth ethernet port that has no business being enabled, eth0 gets disabled.
Maybe it’s something wrong with the hardware, but at least I found the place in truenas that gives me an inkling of a clue to what’s going on here. Not much help, but I thought I’d share what I discovered.
FWIW, “ip address” returns eno1-eno4 and that 20-character device assigned 169.254.x.x.
If anyone knows how to disable that non-ethernet port without mucking up my ethernet ports, and maybe where that second private IP address 129.168.x.x came from and how to disable it, I’d love to know. If not, I guess I’ll let this project die until I have time to read up the 2348 pages of documentation to regain some semblance of sysadmin knowledge. ;-p
Re. “do not open ssh, only VPN”. Easier said than done. I have many headless boxes lying around the world, and I don’t even know how to configure VPN on a headless Raspberry Pi or VM on some random hardware in a closet in Prague. ;-p
I’m not usually a complete idiot, and I turn off cleartext passwords for all ssh servers. I’d like to think that with relatively updated ssh servers and RSA keys, I’m pretty safe from black-hats, especially if I only turn on ssh for a few hours/days (and set up PAM to block attempts–I’m not sure I’m doing that right, but I try when I think I know what I’m doing).
Anyway, if a post-buggy ssh is a security hole, I’d suspect that a VPN is a much bigger one. FWIW, I do run a VPN for my Macs to be able to screen-share across the world. It’s like MAGIC!
Did you ever figure out what was going on with the 1.173 address? If not, answering the questions you were asked six months ago might help. As to the other question, it would help if you shared the interface name.
…and it’d be easy enough to test whether this is the case by trying to browse to that IP. But I asked back in May (and it’s now December) whether OP was sure that IP was actually going to TrueNAS, and still don’t have an answer.
Edit: I’ve edited the topic title as OP hasn’t given any indication that this has anything at all to do with TrueNAS, much less being “sneaky.”
I’m sorry that I have nine other lives outside of managing my NAS and cannot stick with sleuthing until I solve everything. It may be another six-twelve months before I can really look into this, if ever… FWIW, my kludge was to simply BLOCK all traffic at the unifi router to the truenas’s IP that I did not configure. If it is the Bios Management Crap opening up ssh ports then I’m just as mad as I was scared. This is how you get h4x0r3d.
I really want the NAS to be secreted away, well behind many walls of protection, with only the slightest availability to well-authenticated machines on the physical LAN, and the even more secure VLAN. One of the big points of building this thing was to have a failure-tolerant giant repository for ALL of the data I have collected over the past 50+ years of shooting film & video and doing audio productions… so that I can (some day) sort through it all, cull the junk, and leave something beautiful for a legacy.
All that said, I will come back here and document any findings of value that I’m able to make. The main thing I’ve learned from this is to at least check that all traffic on the LAN is not unexpected traffic to IPs that shouldn’t be there.
Thanks to everyone who gave me some pointers on this. Merry Christmas y’all!
You asked for help made a sensational accusation about TrueNAS seven months ago. You still haven’t given basic information that would have let us help you. Up to you, I guess.
Yeah, this totally wouldn’t be documented anywhere. Like in the motherboard manual. And since you haven’t given any information about what that is, we can’t help you there either.
If you want help, we can probably help you–but you’re going to need to work with us. Here’s all we know at this point:
There’s a device you don’t recognize on your network, with an IP of 192.168.1.173
That device is running an SSH server on port 22
That address doesn’t appear when you run ip address on your NAS
You see an interface on your NAS that you don’t recognize, with “a 20-ish-character long name” that you haven’t shared, that you think has an IP of 169.254.something
You believed when you started this thread that the 1.173 address somehow belonged to TrueNAS itself, but have never said why you believe that to be the case. And if that address doesn’t come up in the output of ip address, I think we can safely rule out that possibility.
So the obvious follow-up question is what it is, and I think Eric’s made a very sensible guess: it’s a remote management system on your server. Most servers that have such features have dedicated network interfaces for them, but it’s at least a common (and possibly universal) default for those features to share the primary network interface if nothing’s connected to the dedicated interface. And it’d be trivial to determine if that’s the case: browse to 192.168.1.173 and see what you see. If it looks something like this:
Fun fact: It’s not universal. Supermicro X10SDV-TLN2F boards do not support sharing a host NIC with the BMC, probably because the Broadwell-D integrated NIC doesn’t support this. Only -LN4F and -LN4F boards support it on one of the I210s.
That board does have IPMI, so my first guess is that Eric’s guess is right on the money. But let me check the manual.
After taking a quick look there, I don’t see a direct answer–but you’d configure it in the BIOS settings under IPMI → BMC Network Configuration. I expect it’d be under the setting there called “IPMI LAN Selection,” and you’d probably want to set it to “dedicated.” My guess is that it’s currently set to something like “failover” or “shared.”
) on that second IP address (i.e. when I “ssh user@secondIP” I get a password prompt. I dug a little deeper and found that this second IP address apparently showed up about the same time I got the truenas box running, so I don’t think my initial “I’ve been hacked!!” panic was warranted, nor does it seem to be solar flare related. Even weirder is that the command “ip address” shows the main IP address bound to the eno1, but there is no mention whatsoever of the sneaky secret second IP address. I only discovered it thanks to some good network monitoring tools on my switch!
