NFS shares - IP vs FQDN

I finally have multiple DNS servers set up as primary and secondary in my home lab and thought I would change my Truenas server’s NFS share configuration entries from IP to hostname to make things easier on the eyes. Things went sideways. Everything from the GUI freezing and rejecting the changes to TN claiming it couldn’t resolve the hostname in DNS. My hosts are defined with FQDN and short name in both DNS servers. nslookup from every VM shows them without issue in either form. I also updated etc/resolv.conf on the TN server to search both DNS servers with options for retries and attempt counts (options retries: 2 attempts: 2). I have also ran nslookup on the TN server and even after a clean boot it cannot find a record one time but finds it the next time!

Obviously TN does not play nice with DNS so it looks like I have to go back to putting IP addresses in the authorized host entry in the GUI.

Is there a “best practice” method that I should be following here? Just put in the IP in the GUI and get over it? Add an entry to the local hosts file and use the FQDN or shortname in the GUI? Maybe some option/parm to make DNS actually work?

“Shortnames” if you mean only hostnames, is not something that exists in DNS records. That’s a shorthand stemming from automatic dns suffixes on clients, and netbios things that has stuck around forever. A shortname to DNS would actually be like a TLD and might not function the way you expect.

DNS organizes resources records in a tree format, starting with the root zone (.). Next are TLDs (also known as first level domains). Then second level domains (domain.tld.). Yes the dot is technically there in DNS, but it’s omitted in queries for a good reason, explained later. Then you have third level, fourth level, etc until you get to a hostname.

Groups of this DNS tree can be controlled by one DNS server, and this is called a zone. Typical deployments means your DNS server will be in control of one zone, and in that zone, it should have control of the domain level that your hosts are being added to.

When you add records, your DNS software should be presenting the resource record only if you are querying the FQDN, and true “shortname” queries should be returning NXDOMAIN (doesn’t exist). There’s a catch to this though.

In both windows and linux, there is a function called “dns suffix” and “domain search lists”. They perform the same thing. Basically, when you type in a short name, it will append the dns suffix and walk the domain short list until it gets a satisfactory response. This is how you should be configuring your network and clients for using “shortnames”; give them a dns suffix or domain search list if you are using more than one domain, and what they are actually doing is converting it to a proper FQDN before doing the lookup.

Just be advised: if you have dns suffixes and domain search lists defined, this will get in the way of your troubleshooting. There’s an easy way around that though, all DNS clients are supposed to support a way to bypass all of that, by terminating a query with a dot (.). So if you are testing your DNS functionality, hostname.domain.tld. (dot on end included) should query “hostname.domain.tld” and ONLY that. If you need to test and make sure hostname bare doesn’t return anything, you would query for “hostname.” and unless it’s a known TLD, it should return NXDOMAIN.

So to summarize: you should only have DNS returning records for FQDN queries. You can configure clients, such as truenas, with their DNS suffix or add it to the domain search list so that “shortnames” work too. That way when the clients consult DNS and are only given a hostname reference, they will append your domain automatically.

Shortnames, which to DNS are hostnames at the root zone, should not exist in DNS. The root zone is special and should not have entries in it other than TLDs for your use case.

1 Like

Thanks for the explanation. To be clear, I have defined the FQDN’s in my DNS and I am pretty sure that I did test using FQDN in the NFS Share “Authorized Hosts and IP Addresses”. I’ll go back and change all IPs to FQDN’s and see if that works.

I don’t have experience with NFS, but it may be possible that you also need to define the dns suffix in truenas for correct operation. ESPECIALLY IF KERBEROS IS IN USE. There is one place for the SMB service, and one somewhere in settings, maybe it was network configuration which I think is setting it at the host level. Once set, restart truenas for good measure.

So I am working my way through my NFS shares changing them from IPs to FQDN and everything is going well. After saving the updates to 4 or 5 NFS shares I edit my next NFS share and switch about 4 entries changing from IP to FQDN and when I try to save I get the popup telling me that TN cannot resolve every one of the FQDN’s I just changed. The odd thing here is that the names it cannot resolve are the very same ones I just entered and saved in multiple prior shares. I then had to close/cancel these updates so I went immediately to the Truenas shell where I entered nslookup on the short name (which still resolves) and also the FQDN and it works fine. While still in nslookup I changed the DNS servers hitting each one separately and they both returned every lookup perfectly. I then went back in to the same NFS share and changed the IPs to FQDNs and it took them just fine. Any thoughts?

I really cant say unfortunately. However, you really need to make sure that you are using NO OTHER DNS SERVER in your network’s clients and servers EXCEPT the one(s) you are managing. Full stop, period, no exceptions. That DNS server needs to be configured as both a server and a resolver. It should NOT be forwarding requests outside of its zone (asterisks here, but you will already know why you need to do this if you need to do this). Should you be asking for records outside of its zone, your DNS server should be acting as a resolver and forwarding the requests, not your internal clients!

The idea of “primary, secondary, tertiary” DNS servers going in sequential order for lookups is ANCIENT and hasn’t worked that way in a long, long time. Different OSs do things differently. I believe linux sends requests to all defined DNS at once and uses whatever comes back first.

1 Like

In my experience the resolv.conf file in a unix OS resolves lookups in order of the nameserver records listed. In a config with two servers I have taken down the “first” server and watched the resolution work fine with the “second” listed server so I am not that concerned.

While I don’t have the expertise to build my own DNS server with the appropriate software I at least knew my limits and went with pihole. It does what I want for local DNS records, resolves to an upstream DNS server (Quad 9) and keeps me out of trouble, mostly. All of my systems have two nameserver records in resolv.conf one pointing to each of my pihole instances.

This is not big leagues just my home lab. One pihole runs on my Ubuntu desktop which is pretty solid and where I control everything from so if its down then it is the priority to fix, not the rest of the servers. The 2nd pihole runs as a LXC container on a Proxmox cluster of two DL380 G9’s that point to Truenas2 for NFS Datastores. Before you say it … the 3rd “node” is just the haproxy app running on another Ubuntu box which makes a quorum that allows HA failover. The 2nd TN system, which is the one that has been giving me all the fun today is now stable since I changed all the host records to be fully qualified names. TN also has the correct domain name set as well.

Based on what I have experienced and read I will keep the FQDN format in all of my NFS shares and try not to break what’s working fine. At least for a while. Now with all this mess the PBS container running in TRUENAS1 is a mess and probably needs to be reinstalled. With any luck I won’t lose the few days of backups that it did as I used the internal ZFS bind-mount, but if so I can still fall back to my old snapshots which I have yet to whack. Thanks for the help and advice.

FWIW, both Linux and FreeBSD documentation explicitly state that the order of nameserver entries in /etc/resolv.conf is respected. On Linux there is an option that can be set to rotate queries as a way of load-balancing instead of always hammering the first one. Also, nowadays this is often a generated file and various network management setups define how to influence the order, especially in situations where multiple interfaces and/or VPNs can come and go. Windows is - as usual - a completely different beast; I think the parallel queries can happen there under some circumstances.

Still, all DNS servers mentioned there should be indeed functionally equivalent. Anything else is a recipe for hard to debug problems.

Now that the dust settled with my first TN system does anyone know if it is “legal” to make manual changes to file /etc/exports on a Truenas system instead of the GUI? It looks pretty basic but I have a feeling that someone is going to tell me that it gets overwritten by some function at some point and my changes would be lost :frowning:

It will be overwritten by every reboot and by every save operation in the UI.

Not surprised.

After changing the order of my DNS servers and during a reboot I noticed that every single host entry that I added to my NFS shares gets a timeout on the DNS server that I know is currently offline. I am assuming it was able to resolve through the 2nd DNS server that is listed in the Network settings.

I also noticed messages from dnsmasq saying:

“reading /etc/resolv.conf"

“using nameserver … (1st ip)”

“using nameserver (2nd ip)”

then …

“read /etc/hosts - 8 names”

Those 8 names appear to be the default entries that were created during install. There is also a comment at the bottom of /etc/hosts that says “# STATIC ENTRIES” so I tried to put a few entries in the /etc/hosts file. But … as with resolv.conf … it is overwritten at reload/reboot (yeah, I tried). What is the point of that comment if there is nowhere in the GUI to actually make static IP address entries stay around after a reload/reboot?

The bottom line is that I want to make NFS connections as resilient as possible but also be able to easily know the name of the server that has access to the share. Not asking for too much, am I?

System: Network: Network Configuration - click on “Settings”, scroll down to “Host Name Database”.

Put static /etc/hosts entries there.

In the end … I entered my IPs and fully qualified hosts in the Host Name Database in the Network section and tried using short names on the “Authorized Hosts and IP addresses” fields but there were still weird DNS lookup issues (watching /var/log/syslog).

Final config is to use FQDN in Host Name Database and FQDN in “Authorized Hosts and IP addresses” field for the NFS share. Note: if you look at your syslog after a reboot, of which I did many times here (use journalctl -b) you will see that TN specifically says that it is “watching” several files including resolv.conf and hosts which is why as previously mentioned manually edits will be lost… It would have been nice to be able to run a script to update them. Maybe the API offers that but I’m not really an API kind of guy. All done.

numo68 & GordonShumway, I had to go back down that rabbit hole to respond to this, it had been a while. I do know that priority order as listed in the manpage is only referring when using the glibc resolver.

If systemd-resolvd is involved, which often is, /etc/resolv.conf could be a symlink to stub-resolv.conf! That’s not even the only screwery that can go on with this file either. Just be aware that the manpage documentation for this file applies to only the glibc resolver unless it says otherwise!

Systemd-resolvd, if involved, can have pretty complex logic. See this issue thread with a lot of people finding out. systemd-resolved considers all DNS servers equivalent · Issue #21174 · systemd/systemd · GitHub

There are various other potential resolvers in use and the logic they may use too.

To even take this a step further, systemd-resolvd has hardcoded fallback DNS as well, and a “sticky” function where it wants to stay with a known working DNS for a period of time.

Unless you are absolutely sure of how your DNS resolution is occuring not just the first time but at all times, and especially when troubleshooting, you should only configure the DNS servers you are meant to be querying to rule out unexpected DNS resolver logic, it can have you chasing your tail. Been there, done that.

Anyways GordonShumway, I apologize again, I haven’t any experience with NFS shares, so I don’t know where else to go with this. I can only address the DNS part specifically. The nslookup and dig tools will be your friend to verify that clients’s resolvers are getting the correct host addresses. After ruling out DNS, it’s going to either be an NFS client or NFS server issue, or a broken network issue (i’ve had overlapping IPs before with truenas by mistake and it manifests in all sorts of weirdness)

1 Like