Adapter - X520 DA2
Server Dell r510
Switch Unifi Pro 24 port
Had truenas core and decided to upgrade to Scale. Disabled all onboard nic on Motherboard. Did fresh install and only nic appears to be the x520 DA2. Unifi switch shows a 10g connection on switch. Global network setting would not allow me to save the gateway ip.
Turned on onboard nic cards and used DHCP with both nics and had not issue logging onto server.
Only 1 nic card is set to DHCP
I tried to use Linux so that I could verify that the 3 nics installed but could not enter the correct commands to get vendor. I would like the correct command to use to check this.
It’s not entirely clear if you currently have one more more NICs set to get IP using DHCP, since you mention several configurations. You also finish by saying you have three NICs, but only appear to name the make and model of one.
Not being able to set a gateway is an indicator that you are doing something in an unexpected way.
Please post screenshots of your current network setup from the TrueNAS side.
Also please include complete HW details.
Disable Dell r510 nic’s through the bios. Only Nic card active was the X520 - DA2 name “enp5s0f0”. Could not set gateway even though the Unifi switch said it was connected. 10g nic had no IP and gateway was not reachable.
Removed the X520 - DA2. Enabled Dell r510 nic’s through bios. Both nic on motherboard they are named “eno1” and “eno2”. Both were able to get an IP and I was able to set the gateway and log on.
Put back in the X520 -DA2 and left the Dell r510 motherboard active. Presently what my system is. Only eno2 is set for DHCP.
All 3 install were new installs.
Hopefully I explained it better. I have had truenas core running for many years. I just decided to reset everything and replace my 4 port Link aggregation with a 10GB card.
Unix does not allow more than 1 network interface when using DHCP.
Next, DHCP does not allow setting gateways, as it should add one for you.
If using more than 1 NIC, ALL must be static IPs and then you choose one to have a default gateway. In theory, you can extra static routes but that type of network design is generally beyond normal NAS usage.
If the TrueNAS GUI implies the ability to use more than 1 NIC when the first is DHCP, it is a bug. Please report it.
All that said, Apps, Docker and VMs may be able to use additional network interfaces.
I am aware of only 1 adapter set to DHCP. My first install had only the 10gb card active. It had a connection according to my switch right after the install. I tried to set the network but it could not find the gateway. I did try setting the 10gb card to a static IP but it also made no difference. TrueNAS is only using 1 IP at a time so there is no bug. After spending 3 days trying to resolve this issue, I have decided to buy new transceivers 10G SFP+ a RJ45 to see if they make a difference. I am running now with the 1gb and the system is running great so I can continue to resolve this issue. If the new transceivers do not resolve it I will test the card in a windows desktop.
The cards come in several flavours (firmware), depending on the manufactor (well rebrander). And at least the intel branded cards, want’s intel branded SFP’s. What brand (HP, Intel , Dell , …) is your model ??
Try one of the “fix’es” mentioned in the url’s above.
If i were in your situation, i would probably be testing the fix, initially on a system that was booted from a “clean Debian 12”. Either a test machine or the “NAS” booted from a Deb12 live-install.
Edit:
I have an X520 in the drawer, but have not used it yet.
So i just have “off the web” knowledge about the sfp issue.
Btw:
I actually bought two solarflare cards, due to the x520 sfp issues, and x520 power usage. They’ll definitely come out of the drawer before the X520.
This is the first fibre connection I have done. I did verify what cable I required when I bought the card with 2 transceivers. Fiber Patch Cable - LC to LC OM3 10Gb/Gigabit Duplex Multimode. The transceivers are avago afbr-709asmz-e2. I thought the important thing was the transceivers were to eliminate compatibility and did not consider compatibility with hardware. I would have thought if I bought the sfp with the card they would be compatible with the card but it is possible that they are not compatible with the switch. I looked at your link on possible fixes and will check at least for the x520 card.
I have seen enough other implementations to say that it’s not uncommon to have an option to get the DHCP IP and then override any DHCP assigned DNS/GW with manual entries.
Admittedly, since I don’t use DHCP on my TrueNAS servers, I haven’t tested this there.
Are you sure about this? In the numerous discussions in the forum about multiple NICs, the statement is usually that you can only have one NIC set to DHCP as opposed to none, if using several. The main discussion point tends to be around the validity of having more than one NIC assigned to the same subnet.
Also, when setting the gateway in TrueNAS, you do not tie it to a specific NIC. The GW setting is set globally. It’s looks slightly different under the hood, but at least that’s how it appears in the web interface.
I am positive I am only connecting with 1 DHCP connection. Once I resolve the issue with the 10GB card I will set the system to a static IP. I have bought new transceivers for my Ubiquiti switch that are Ubiquiti to eliminate if the transceiver I have now is incompatible. I have also bought an Intel transceiver to make sure it is compatible with the card. I have also bought another brand of cable. This is the most logical step to me as Truenas recognizes the x520 and the switch says it is connected; the web is full of people talking about incompatibility of transceivers. Once I complete this next step I will post the results.
@neofusion - No, I am not certain about anything with DHCP. I simply don’t use it enough to memorize it’s details.
Yes, I am aware that gateways, (default or additional static routes), are not tied to a NIC. They are however tied to a specific sub-net’s router. Which generally ends up going to a specific NIC.
Solaris 10 & 11 are different. They support multiple NICs on the same sub-net and do the proper traffic routing when using IPMP, (Internet Protocol Multi-Pathing). This would in theory allow multiple default routes, going across multiple NICs without using LACP.
My TrueNAS is mainly a backup to my computers and my 2 Synology Nas. I have lost all my music once and now I have it located in 4 places. I only need 1 IP address and will change the DHCP to a static once I get the 10gb card working. I have done a lot of reading about compatibility and I now think my issue is the Transceivers or could also be a bad cable even though the cable is new.