I’ve had a variety of personal systems in my home lab over the past 15-20 years. I had rack mounts with all sorts of official ‘iX’ gear (TrueNAS, FreeBSD poudriere builders, PC-BSD hosting, etc etc).
Until last year I had a rack in my home office, with nearly 100T usable and somewhere around 100 drives spread across many systems. It was way overkill for what I needed and far too noisy for my home office.
This past year I downsized into desktop chassis and went all Flash (spinning needs to die), one system with 4x NVMe M.2 devices, the other using a 9500 Tri-Mode HBA so I can use 4x U.2 NVMe drives. Both systems are using the same 12th Gen Intel(R) Core(TM) i5-12600 CPUs, with onboard GPU for some apps, and one of the boxes has an ARC A380, for testing and Apps as well. One system always runs latest release of SCALE, the other I keep on nightly images for dev work and testing. I like to re-build my TrueNAS systems every 3-4 years, so I can get the fun of “system building” while also trying out new hardware combos first-hand.
Maybe in a future podcast I’ll go into more detail on the home rigs if anybody is curious. To answer your question of tech “worker” vs “enthusiast”, I’d say I’m a mix at this stage. I build systems for fun, have RPI devices aplenty, contribute the occasional code and App to TrueNAS. My last “for fun” tech experiment was installing TrueNAS on a Radxa x4, which worked surprisingly well
The n100 in my mini lab also works surprisingly well for the needs of the lab. Running Scale because the 10gb adapter isn’t supported in Core (and scale is really the way forward). Only 6 SATA spinning drives right now, didn’t want to spend the money on SSD yet.
Wish I had more time to throw into figuring out how to build Truenas on Arm.
Looks like this is going to be tough to do on a new platform. make checkout works fine. But then make packages, which is supposed to build all of the packages from the source that just got checked out, immediately makes a truenas scale chroot to build in. Which of course only exists on x86 right now.
The README says to use a debian chroot to build it and then it just makes its own environment? What the heck? Hopefully it won’t be too hard to have it create a debian chroot from the debian repos and build from there. But from the looks of it, one can’t simply build the os, one would need to setup repos as well.
@bobpaul - If I remember correctly, one of the iX people said that if someone found things to change, that help ARM64 builds, to make a Pull request.
It is completely understandable that a company developing software for a limited hardware set, did not take into account building on other Archs. But, they do seem receptive of making general purpose changes that allow building on ARM64.
Does it, though? How important is caching in an all-NVMe system?
Obviously it isn’t going to run TrueNAS, and if it did, it’d be missing the switch management aspect. But it really seems like they’d be better off working to integrate an existing NAS OS than trying to shoehorn those features into RouterOS.
ECC RAM, after all, every last bit goes through the RAM, whether it is network packet or NVMe block
It mentions 2 x M.2 SATA slots, can those be used for a larger OS? And booting? Or would the NAND need to be a pre-boot code location?
The U.2 storage is through 16x PCIe 3.0 and multiple PCIe switches, limiting each U.2 device to 2 lanes of PCIe 3.0
The SFF-8644 shows 4x PCIe 3.0 lanes, so their is some expansion.
CPU is wired to the Ethernet switch via 4 x 25Gbit/ps Ethernet. That limits individual streams.
The 1Gbit/ps Ethernet port is not part of the switch chip, it truly is out of band management.
However, all those questions / comments aside, for the price, it looks good. (Though for specialized hardware like that, it either needs Open Source software support or compelling vendor software.)
Fair point…
I guess it depends what is required of the NAS. But if we were to put ZFS on it, I guess ZFS would at least want all of its metadata cached in RAM. With U.2 drives now exceeding 100 TB apiece and 20 U.2 drives in total that could be A LOT of metadata.
Of course, there are some “hammer and nail” issues lurking around. Mikrotik has its own RouterOS, so they want to start from it and add a NAS component. We are a TrueNAS forum here, so we tend to put/want/dream ZFS everywhere.
I genuinely have no idea whether it would be easier to bring OpenZFS to RouterOS—and then decent NFS, SMB and iSCSI capabilities—or to bring all the niceties that Mikrotik invested into RouterOS into TrueNAS. I guess, however, that neither Mikrotik nor iXsystems are eager to go for such a merger
Probably not, though HexOS suggests iX is interested in working with others.
You’re right, of course, that this device isn’t ideal hardware for TrueNAS, even leaving aside ARM vs. x86–but it’s among the closest I’ve seen to good hardware for it, and to hardware that might start driving a port of TrueNAS to ARM. Something like this, but without the switch, would border on “compelling.” Give it expandable ECC RAM, well…
Run for the hills. The SMB file sharing presently built into routerOS can only be described as primitive. I have no doubt that Mikrotik is working on something better but I’d give them at least a year or two before the biggest issues have been uncovered and perhaps resolved.
I have a lot of respect for Mikrotik but even something as seemingly-simple as hosting a iTunes repository for a Sonos 1 system was beyond the capabilities of a HexS. Apple solved that problem with far less capable CPUs over 20 years ago. Speeding up the back end by deploying lots of NVME won’t solve the atrocious UI nor the wonky behavior.
Good point: RAM is ECC.
Bad points: RAM is soldered. U.2 bays are 7 mm, no hot-swap (powering off the switch to replace a drive? oops!). And storage configuration, including creating RAID arrays and shares is through the CLI.
“Badly” and “primitive” are hereby nominated for the Understatement Of The Year award…
If Patrick at STH comes away confuddled re: how this thing is supposed to work as a NAS, good luck to me or anyone else not extremely conversant re: MikroTik / routerOS terminal speak to make it happen.
I agree that this is an interesting gateway / router to help run a decent sized business. It has tons of connectivity! But using this as a NAS for hundreds of users will likely cause multiple admins to quit in protest.
I would hope that if you have hundreds of users, you will buy/build a real NAS. This Mikrotik device is designed for a handful of users, getting past about 10 and I would be looking for a different solution.
I have an odd setup that I wish I could 100%. The last bit I cannot implement is Truenas on native aarch64.
I have a Mac Studio I got for a good price with 96GB of RAM. I run OPNsense on aarch64 (built be a community user) and have a thunderbolt-attached SFP+ cage so I can directly connect to AT&T’s fiber. OPNsense is running in a VM, and is acting as the core router/gateway for my home network.
I then have another thunderbolt-to-SFP+ cage connected with a DAC cable to a 10GB switch. I could have done something trunked, but I connected the Mac Studio’s existing 10GB Ethernet to the switch that distributes for my home network. I can remote into the Mac Studio to restart the router VM, etc.
I still run an x86_64 Asustor 6407T NAS with TrueNAS on it. I wish I could move this to the Mac Studio as a VM, and connect it as a guest (on the network). I can’t because UTM does not yet support USB 3.2 Gen 2. I have an 8 x SATA cage that is USB 3.2 Gen 2 and it provides just enough bandwidth to get decent performance across a raidz1 like this.
I can successfully run TrueNAS x86_64, the stock image, in UTM as a VM on an arch64 host with great performance, but because UTM doesn’t support USB 3.2 Gen 2 I cannot connect my SATA cage.
If it were native arch64 the USB 3.2 support would be there (I gander).
PS: I run Podman Desktop off native Mac OS (the host), and from there I host ollama, open webui, and comfyui in containers.
I really with TrueNAS gave aarch64 a shot… the Mac Studio is attractive for its performance per watt. Way overpowered, but I’m loving this otherwise.
Running TrueNAS in a type 2 hypervisor for production is highly advised against.
As for the router, do as you please, but I wouldn’t want an essential piece of my network infrastructure to run vitualised on a desktop machine which can crash, be switched off, or reboot for reasons unrelated to update OPNsense itself.
…but not as a NAS. A closed machine with no user storage storage, no extension possibility whatsoever, and which uses a proprietary, undocumented, boot process. That’s about the worst possible hardware suggestion for a port.
Is there a good thread to read up on the dangers of this? I would appreciate becoming more informed. It has been stable for me for ~2 years, but I have recognized USB as a fragile/finicky protocol in the past.
The main draw for me here is how much functionality I’ve been able to consolidate into a surprisingly low-powered (and arguably over-engineered) single device:
It serves as my core router (running as a VM).
It’s on its way to becoming my NAS (also as a VM).
It hosts LLMs for LAN clients (currently using Ollama in Podman on macOS, but I’m planning to migrate this to Fedora CoreOS or Talos Linux in a VM).
It runs a ChatGPT-style web interface for local access (Open WebUI).
It powers ComfyUI for image generation (because memes matter).
It supports my SDR setup via OpenWebRX.
It also provides Headscale, Nextcloud, and other essentials for mobile devices in the family.
Admittedly, it’s a bit of a Frankenstein build—but considering how expensive utilities are here, I’m pretty pleased with the consolidation.
Without running TrueNAS in a VM, the whole setup idles around 20W. It does spike when all 8 disks spin up or when I push the LLMs hard. I’ve even considered ditching my laptop entirely and just using this Mac Studio server/router hybrid as my main device—essentially routing internet ↔ everything through it.
Previously, I used a Banana Pi BPI-R4 as the network core—router, switch, AP, the whole deal. Now it’s dedicated to AP duties, connected via 10Gbps backhaul to the switch. One of the major wins with the BPI-R4 is its dual 10Gbps SFP ports.
I’ve never had macOS completely fail on me—it’s not perfect, and I do have my privacy concerns—but its recovery system is something I really admire. That said, I think atomic Linux images are the future. You can trust an immutable image. I’m also keeping Mac OS available so I can use Jellyfin with hardware decoding. Support for the Apple GPU is not available yet on Linux (afaik).
Bonus: I can pull close to 10Gbps over Wi-Fi in my living room via the Banana Pi AP.
All of this is backed by a UPS, giving me roughly 2 hours of runtime during outages.
)