I believe you should be able to update any vendor specific firmware/card to the standard Mellanox firmware. Just like flashing an HBA.
It is easier said than done, shell only tools, there is no gui and I still have no clue how to flash, mlxconfig there is a while language around it…
If only they made a simple graphical interface…
There must be a million threads over on ServeTheHome for the CX3 
2 Likes
I believe the GUI you’re looking for is called GNOME or KDE. 
yes you are pointing out the one model and Card that is VPI and accepts mode config, I was considering buying this exact card, however after buy 4 cards…im kindof burned out on rolling the dice.
Ah I see, so can your particular model not be flashed? Or change between IB/eth modes? Honestly asking as I don’t know.
I also went down this rabbit hole, there are no videos about or walk though, mostly you get bits and pieces like download TOOLS from NVIDIA SITE. I tried Windows version but there is Linux too…
Just be ready to use script language specific to these tools, like -d, -v q…mlxconfig, something to query, there is another to burn or write flash…
my best advise from TrueNAS perspective,… avoid ConnectX chipset, TrueNAS does not like Infinibans mode…if you can convert to Ethernet mode, there appears to be a generic driver loading at TrueNAS start
If you are a Sadist and love torture, best odd with VPI and the MCX354A chipset,
all will most likely come infiniband mode. I connected on card to my windows and infiniband is in Title of card detected… “IPoIB” in title you will need to change mode.
I even saw some screen shots where there were able to install a more official driver for network card and you can change mode in windows under properties…but I don’t think its permanent in Config… remember these card will work automatically with windows, I have a hard time getting rid of generic windows driver,
Like the helpful people above, pay a little extra for Chelsio T580 or a little more for Intel 740 i think… anyway I hope it helps,
I’m going to make a video about my experience to hopefully help other people avoide some of the mistakes I have made along the way .
1 Like
Yes but the tools provided by NVIDIA for Mellanox, are shell only, no matter what OS you use, you will need to open shell window where you will run all of the queries and commands, its also possible to brick the card…
Was only joking
I’ve been using MLX cards with TNC for ages*, and yes it has not always been smooth sailing, but most of the time it worked fine.
Perhaps not as performant as hoped/expected, but basic functionality was present and it always was a good value. Dunno if I was lucky to skip a lot of the problems that ppl seemed to have or why I never had those.
Anyhow,
“normally” handling the MLX cards is relatively simple.
Yes you need to use the CLI, but thats based on the fact that they use the same tools on different platforms on most of which Admin usually don’t use GUIs (ie all except Windows).
Things get clearer if you consider that MLX cards come in two flavours, VPI and ETH only, and their switches used to come with the same two flavours.
Originally they only did Infiniband (actual IB or IPoIB), the ability to run Ethernet came later and was a very expensive experience back in the day (licensing).
Because that is their root, most VPI (ie IB+ETH) cards come in IB mode when they are sent out (as that was their original/primary use in datacenters).
Now converting a card from IB to ETH is simple
identify the card, set port status.
here an example i had saved from an ESXi host:
[root@2heesx:~] /opt/mellanox/bin/mdevices_info
PCI devices:
------------
ConnectX3(rev:1) mt4103_pciconf0
ConnectX3(rev:1) mt4103_pci_cr0
[root@2heesx:~] /opt/mellanox/bin/mlxconfig -d mt4103_pciconf0 set LINK_TYPE_P1=1 LINK_TYPE_P2=2
Device #1:
----------
Device type: ConnectX3
PCI device: mt4103_pciconf0
Configurations: Next Boot New
LINK_TYPE_P1 VPI(3) IB(1)
LINK_TYPE_P2 VPI(3) ETH(2)
In windows you’d usually install mst tools, run a “mst status” to get the device IDs and then run
mlxconfig -d LINK_TYPE_P1=2 LINK_TYPE_P2=2
Thats it, its a one time thing, and the card is good to go.
(Note that you need to setup the port on the switch to ETH as well o/c)
However, with all cards that get OEM’d and where you have 15 years of drivers and different agendas over time, sometimes things are not so simple and you have problems like some firmwares not accepting ETH any more.
The beauty of the CX3 is that it is easily cross-flashable between vendors and different firmwares, no lock in like in newer cards.
It (used to be) widely compatible, excellent bang for the buck [show me another 40/56g card for under $20], flexible, readily available … no reason not to like it. Well except maybe not really consumer oriented;)
p.s. IB support in TNC is nonexistent, so that is doomed to fail. You might get it going on TNS, but its not supported, probably not recommended, and not point and click at all.
*I guess I should say “I used it ages ago (CX3), then didnt use it (virtualization) and now used it for a couple of years with CX5”
2 Likes
That is Very helpful, I will try that just for the challenge, but the scrip[ting thing was not easy for me to figure out…
I have ordered already Chelsio T580, they were so cheap i not have 3, which will be good for NAS and workstations, I really hope speed improvements will happen, from 1gb to 40gb…who knows considering all the elements, but its a good experiment, Do you nkow how to check licening on a SX6036 switch?
I don’t have a 60xx running atm, but on my SN2100 its under Setup/Licensing.
The 60xx come without Ethernet license usually, so unless you’re very lucky…
Also, if you have a valid license (and don’t buy one from the scalpers at Ebay),
you still need to configure the switch to allow ethernet and config the port (or all ports) to Ethernet too
Given that you said your card worked out of the Box with IPoIB I assume your switch runs IB.
There are threads over on ServeTheHome for both of these issues too o/c
is there a Mellanox software for windows to read licensing? what command might i need if its like network card>, thanks in advance
There is no windows program to connect to the switch to read the license, except of course that you can ssh in and then run the appropriate cli command.
Accessing the webgui is as close to a Windows GUI as you can get here.
lol I would need to run my microtik router into this switch, assuming there is Ethernet port working on it.,…and i need to convert from cat6 to QSFP+… 1gb…lol
I think im going to ssh this switch…somehow looking up commands for sx6036
What’s outlandish about a management network on 100M/1G copper?
The switch has an ethernet mgmt port and a (cisco compatible iirc) console port.
sn2100 [standalone: master] > enable
sn2100 [standalone: master] # show license
License 1: ...
1 Like
I know I have a typical home network, 1gb, but recently excited to upgrade to 40gb, however my internet will never keep up with 40gb…thinking of just running Fiber as intranet, no point in integrating Fiber to internet, just keep it as local transport.
By the way I want to thank you guys for helping me figure this stuff out, im not new to computers, but TrueNAS and Fiber, its virgin territory…
If I would have just limited myself to 10gb, I bet would be no issues at all, but the Switches were so cheap and I hear this switch can be full unlocked in hybrid mode…lol also easier said than done, oh well here i go…
2 Likes
Ok Chelsio T580, Out of mailbox working, I still need to program the switch, but no issues on 40gb card on TrueNAS core…
But I still need to use Connectx3 infiniband to initially connect to SX6036 to program licenses etc… and make sure I have ethernet ports available…
No, you should connect to the management port of the switch. Or the console port, either works.
1 Like