In seeking 10G+ solutions for my home, it occurred to me that I only really care about 10G routing for my TrueNAS Scale machine and my main PC.
I came across the MicroTik CCR2004-1G-2XS-PCIe and in theory it sounds like I could put this in my TrueNAS box, have WAN from ISP going into one SFP port, one SFP port going to my PC, and maybe the gigabit port to a WiFi router (set up in AP mode). Then I’d have full speed (10G ISP connection) for my PC and server, gigabit WiFi for everything else, and it would all just live in my TrueNAS machine.
Has anyone done anything like this? Does this even look possible or am I misunderstanding its capabilities?
Any feedback, advice, appreciated, welcomed.
I think you’re probably right and that many of us home-server/self-hosted folks are well known to do that to ourselves.
You raise an excellent point though. My thought was “I’ve already got this great machine on a UPS with space for a PCIe card”, but a small wired 10G router would definitely do the trick with a bit less fuss, if slightly less integrated, and certainly cheaper than a wireless router with two 10G ports. I’m new to SFP so I’d have just have to figure out what cards to put in the server and PC to support that connection to the CRS305. Any recs?
Solarflare SFN 5122F/6122F/7122FF, Chelsio T520, Intel X520… whatever you find second-hand/refurbished. Should be $50 apiece or less—just check that Intel NICs are genuine.
Looking more at the CRS305, I’m concerned an 800 MHz single core processor might not be enough to actually support routing at 10G vs just switching. I’ve had issues in the past with similarly spec’d wireless routers struggling to hit 1G. The CCR2004 has a much beefier processor, and I suspect in part because it’s meant to be a high throughput router.
Switch where you can, route where you must…is the oldest network motto there is and is still true today.
Buy a cheap 10 gig L3 switch and put a 10g generic nic in your Truenas server. Let the switch do the routing. Use vxlan or a virtual firewall if you need to isolate vlans from routing between each other.
That Card is no different than running and x86 or ARM Linux router with FRR package installed.
It will actually become the bottleneck as it’s not an ASIC it’s a 4 core ARM cpu that’s forwarding in the CPU. It’s no better than using your NAS cpu. Its value is as a mid priced 25g nic. Functionally it’s basically a 25g 4 core ARM DPU.
Opnsense or pfsense in a vm will be far more useful if you need isolation of host vlans. You just need decide if you run that on the truenas host or a dedicated box.
That decision really comes down to how many cores you have available on your system.
Personally I get 10+ gbps through my Nas with opnsense no problem and just isolate 4 cores to it with a good Intel 10g NIC that has decent buffers\memory space and stable Linux drivers.
My internet connection is direct into my switch, I take the vlan up to the opnsense vm and route from here to secured and unsecured vlans. The switch does a single L3 routing function on the trusted side, and some isolated vxlan forwarding for some host/vms that support it.
Thanks for all of the thoughts! After evaluating my current and potential future networking desires, I’m going to give the EnGenius ESG620 a shot. Not a lot of reviews, but their other stuff seems to have decent reviews and the combination of SFP+ and 2.5G POE+ is going to give me a lot of flexibility for the price. The TP-Link ER8411 is probably next in line if the ESG620 doesn’t work out well.
I would recommend you do some research on the EnGenius ESG620.
Contact their sales team and find out if that box has a real asic onboard that is servicing those host ports. You might get a rude surprise that it does not. Given the the fact that thing is running an ATOM CPU and the sub 10G forwarding specs (over all ports), I would guess that it’s just an ATOM CPU with a bunch of NIC interfaces wired direct to a PCIe 3.0 x8 or x16 bus.
The point is that if you need to be clear what your goals are.
If you only have a <100Mbps WAN/Internet connection, then this device may not be very useful.
If you are trying to run 2+Gbps between VLANs on your local network, this is probably not a good device, especially if you are using WIFI connected to it.
Most SD-WAN / Secure Gateway / Firewall all in one boxes are just small CPU devices with no onboard ASICs. The Port to Port switched traffic performance is usually all through the CPU, and as such they are not good options where you are planning on doing multi-gigabit traffic between VLANs or Security Zones.
If that is what you need, buy a real network switch with an ASIC. There are a lot of second hand options around depending on your knowledge/skill levels. EnGenius and Ubiquity both have good options if you don’t have experience using a switch CLI.
Thanks for that feedback. I tried it out and ended up returning it and getting an Omaha 8411 instead. The ESG fans ran constantly CPU usage seemed to never go below 20%, which is likely due to it being Atom based without ASICs) and it was just too loud for my space. 8411 is doing the trick just fine. Runs fast and quiet.
I’m fairly sure that 8411 has no asic either. Just based on the throughput. But if it meets your needs then kudos.
The thing is you really don’t need it unless you are constrained on CPU cores. opnsense and pfsense vm’s will do everything that box does and they’re free.
The max throughput of that device in real world 256k mtu is about 3Gbps one way. Pay close attention to the 1500 mtu throughput. It’s not great.
This is what we mean when we mean when we say;
Switch where you can,…i.e. locally
Route where you must, i.e. wan/internet
Personally I would have bought a cheap switch and upgraded your hardware or ran second truenas unit (even on an a cheap generic atom/nuc or old laptop), if there was budget left over.
Old used laptops make great donor boards for a truenas host running vm firewall / virtual appliances. I’ve seen people 3d print cases for them if you want to ditch the laptop chassis for example.
Appreciate the extra insight/tips. To clarify, I’m using the ER8411 basically for the 10G WAN port and low throughput VLANs. The other 10G SFP+ port is going to another switch (currently 2.5G with SFP+ passthrough, eventually will be an SFP+ switch, probably Microtik to move data between VLANs). I’ve seen over 8 Gbps down and 6 Gbps up to my ISP from my PC and my server is saturating the 2.5G port until I get the SFP+ switch.
Just home use, and yes, it’s uncommon, but I’m lucky enough to have a local ISP that serves symmetrical 10Gb fiber for $50/mo. I got those speeds over speedtest with a Mellanox Connect3 SFP+ ↔ Mokerlink SFP+ (switch) ↔ ER8411 SFP+ ↔ Fiber ONT.
You won’t see those speeds when you are routing on the same hardware. The process of routing requires the CPU to perform a number of additional cycles per packet. Each packet processed slows down the data rate and there is little to no benefit from previous packets with the same src/dst because the host is not designed to delegate that to a separate system. Generally a dedicated routing OS can make some improvements as it’s sole focus is on forwarding traffic, unlike your host PC. Even still it will never be as fast as a pure end host.
Traffic to or from your end host is very simple and far quicker to process. The host is just looking at the destination ports and which application/process to send the payload to.
This is why ASICs exist and why for the past 30 years people have used them and not just high end servers/CPUs. They generally don’t put ASICS in servers because it still requires the host OS to update and maintain the ASIC forwarding table. Supporting that is not simple.
The vast majority of firewalls/sd-wan devices/small routers etc are just CPUs without an ASIC. 10Gbps one way is about their natural limit. It’s not a size issue it’s a packets per second issue.
