Talk me out of ASUS Pro WS W680-ACE (separate boxen)? :D

I am enamoured with the idea of a single box to host a new (proper, with ECC) ZFS array as well as Frigate (NVR) which requires some GPU for processing the video (and upcoming AI features in all likelyhood) as well as its own storage[0] of course.

Of course this particular combination narrows the field quite a lot, but I zeroed in on the ‘ASUS Pro WS W680-ACE’ motherboard as it seems the cheapest/best way to get to ECC support and still have some capable GPU and not too bad power consumption (by using 12th gen Core i3 perhaps), with plenty room for future expansion.

However maybe this is the wrong approach altogether? Until I stumbled upon W680 chipset I was thinking that 2 separate boxen would be necessary: one for the ZFS NAS and another for Frigate.

However my worry was that might be pushing too many bits over the network (unnecessarily), and it might be better to be writing all that IP camera capture video locally on the same box.[1] So I keep going back and forth. And that was before I saw what DDR5 ECC [2] was going for (DDR4 is not an option with this board, unfortunately). :rofl:

I have been reading a number of posts here recommending repurposed server grade hardware (Xeons and the like) but it seems this is more geared toward the “pure NAS” use case, rather than what I have in mind. Anyway, I thought this might be a good place to ask for people to poke holes in my plan, as I admit to knowing much less about this kinds of hardware (but very willing to learn). :slight_smile:

I have been researching all of this for what seems like months already (off and on) with no end in sight. Well I thought the ‘ASUS Pro WS W680-ACE’ might be the end finally, until I stumbled across some posts here. So I joined the forums to make this post and seek some feedback.

Because most of these threads seem to boil down to “what are your criteria”, I will list mine (in roughly descending order of importance):

Criteria (in Detail)
  1. Reliable data storage of important personal media like documents, photos, etc. (hence ZFS) [3] which I have wanted for literally years now (and can finally afford to do properly (i.e., with ECC)).

  2. Ability to self-hosting some services:

    1. Frigate surely being the most demanding.

      1. Possible expansion into local hosted AI (maybe) so I want to leave room for a dGPU.
    2. But there are already some others:

      1. Home Assistant probably being next most demanding, which is no where near Frigate. Although potential locally-hosted AI would of course increase the requirements a lot.

      2. And sure to be more in future.

    3. Media (to me anyway, not all of this (e.g., movies) requires ZFS, but maybe some does (carefully curated music library)).

  3. Expandability (is more important than…).

  4. Power consumption (which would be nice to get “as low as possible” but not high on the list; power not terribly expensive here).

  5. Noise (I don’t want screaming rack gear, but WS W680-ACE is full ATX, so we are already probably thinking large case so this implies large fans which should already be plenty quiet).

  6. Size of the box (is not a concern, in fact I would prefer full ATX for other (more important) reasons above).

  7. Cost (I want the best “bang for the buck” but not worried about spending a little more for the proper solution; there is no “budget” per say).

Maybe I should mention that this is a GNU/Linux “shop” (home :slight_smile: ) and I already have some years experience self-hosting on a number of ARM based SBC, which I could still use for some things perhaps going forward. But no SMB nor Windows boxen, we use NFS here, As God Intended. :smiley:


[0] Non-ZFS, could even maybe be SMR drives(?).

[1] I am sure we will probably get up to at least 5-6, maybe as many as 10 (as high definition as necessary) cameras, eventually.

[2] At least according to ASUS QVL for this board, but maybe I should broaden my search.

[3] Yes I know that high availability != backups. I do make backups, in fact eventually the plan is to build another box for a friend and cross locate each others backups off site (automated with ZFS snapshots).

So one immediate problem I see, if you want ECC then I’m not aware of any i3 that supports it. Although the motherboard may support it, you need to make sure that the processor does as well. ECC is recommended but not absolutely necessary.

I’m not sure what you mean “cheapest way to get ECC support and still have some capable GPU.” Any motherboard with a 16x PCIe slot can support any PCIe GPU.

It’s good to have an idea of what you might want to expand up front. One downside to consumer CPUs is the small number of PCIe lanes. I think with non-server Intel CPUs, that comes out to ~20. Chipset usually breaks it out into more at slower speeds, and the amount is fine for most uses.

When thinking about power consumption, also consider the up-front cost. Spending hundreds more to save a few watts may take decades to break even. It just depends on your rates and the power saved, of course. The GPU will potentially be the largest consumer.

Personally, I prefer to put all my always-running services and storage in a racked server. The equipment can be found for relatively cheap, there’s plenty of expandability, and I can stack chassis of disks. Not suggesting you should do the same, just depends what you’re after.

What future expansion would you like to plan for?

For ECC that’s i5 and above in these generations.
But the cheapest way to get ECC support is generally to go for older-than-last-gen hardware. PCIe 5.0 motherboards are getting crazy expensive. And DDR4 is cheaper :wink:

Since your use for iGPU appears to be analysis rather than transcoding, you may look into Ryzen as well.

“Power consumption” here applies at idle, which is a typical load level for a home NAS. If Frigate is doing image processing 24/7 this does not apply and the benefit of a consumer-class CPU (idle <10 W) over server-class (idle over 50 W? why not? what’s idle anyway?) somewhat disappears.

dGPU + expansion = PCIe lanes, more so than what consumer-style platforms provide,
so you might drop the iGPU and go for Xeon Scalable/EPYC with a dGPU from the start.
This boils down to how much compute load you expect.

Why? Surveillance drives are not SMR, that would be bad for continuous write workload. And SMR now provides minimal size benefits over CMR.
TrueNAS however only supports ZFS, so non-ZFS is either “another box” or means that TrueNAS is virtualised—implying a dedicated HBA and more PCIe lanes on the host.

Thanks Guise for the feedback, this is exactly what I needed. I think I will need to sleep on it, and think about it some more. And probably, do even more research (unfortunately). I used to love doing this kind of research (when I was younger) but now I am starting to grow weary of it (to be perfectly honest) as I am now middle aged and have a family, and a house, and therefore many other projects and responsibilities to attend to. And never enough time. :frowning:

Some more specific replies…

I found i3-12100T[E] which do, according to Intel Ark. Is that incorrect? And appear to be plenty available on eBay. I’m not hung up on that anyway, it just seemed sufficient for my needs. Certainly there are many 12th gen Core that would fit the bill (and likely some others, but I know less about those).

Sorry, I meant iGPU. Leaving the PCIe for other things.

I guess here lies the rub. Compared to what I have been used to (tiny ARM SBCs), moving to even an N100 (what Frigate recommend) would feel like a huge upgrade probably. :smiley:

But I mention N100 specifically for exactly the same reasons you guys do, lack of expansion. Here we are talking only something like 9 PCIe lanes (I think?). Now THAT is what I consider “too limiting” (and thus, what began this quest). But people are running like 20 cameras on setups like that these days, just amazing what the newer hardware can do. It’s really hard to beat that, in a tiny box, for <$200 USD! And only uses a few watts!

EDIT:
Of course, this is a completely different category of machine, no ECC, (essentially) no expansion, etc.
/EDIT

I only have some vague ideas about future possibilities, maybe some image gallery like Immich or perhaps NextCloud, something like that, other services like an RSS feed aggregator, home inventory (grocery) management perhaps.

Getting into AI, I already know, would be a huge jump up from things like this. And I am not even sure I need / want / have time to do so (realistically). Just kind of “dreaming” so to speak. I have much more immediate concerns (and no where near enough time to implement even those)! :smiley:

I suppose if I can handle my current needs (primarily Frigate, and handful of relatively lightweight services, and important personal media for the family), that might be enough “for now” (i.e., the next several years, at the rate I seem to do things).

I don’t want to be limited (within reason), but I don’t want to go crazy over-specifying the hardware, either.

In particular, this seems a bit overkill to me for my needs (unless maybe I really want to get into AI stuff and really hosting a lot more intensive loads, which I don’t think I will be). Although, it would certainly be “future proof” (everyone’s favorite saying :smiley: ).

But I guess I need to think long and hard about what my needs really are, before spending the money on hardware. And so I will at least look into these things you mention, thanks.

Yeah, I was all but ready to start ordering hardware, when I realized this mobo only accepts DDR5 (not DDR4) and started to price that. :astonished:

EDIT:

So I guess by “older-than-last-gen hardware”, at this time, means what? Gen 10/11 Intel (or AMD equivalent)? It’s been a long time since I’ve built a PC, so I’m a bit out of the loop here.

/EDIT

And as Jorsher correctly points out (and I am aware):

Power is relatively cheap here, I think. I just like the idea of an efficient (as possible) server, if it is going to be running all the time.

Yeah, no transcoding. But I suppose I assumed the iGPU was just better for whatever Frigate needs are, but I admit I am no expert in this area.

I did look at Ryzen (first, actually), as they do not do this silly market segmentation nonsense that Intel do (wrt ECC). But I found some other comments about high power usage (and other issues) that kind of turned me off as I recall.

Yeah, I was already wondering about this as well, thanks.

Yeah this was on my list of things to research, hence the ‘?’.

I was not aware of this. Are you saying it is not even possible to have a main ZFS pool/array (for important data) and one (or more) others for things like surveilance video (or even movies)?

:point_up:
That. And your explanation that Frigate is far less demanding than I thought. This rather points towards Core (Xeon E)/Ryzen.

That would be AM4 Ryzen, or going back to Gen 8/9 on the Intel side because Gen 10/11 does not support ECC unless you go Xeon E-2300/W-1200/W-1300. (In both cases on the Intel side, you’d have to research and price what you can actually find now.)
DDR4 ECC UDIMM then, not RDIMM.

You can, of course, have as many pools as you want, including a non-redundant pool for “non-important” things but lack of redundancy somewhat defeats the purpose of using TrueNAS and ZFS.

So I think that the next step should be to decide on the general setup:

  • TrueNAS, and Frigate as an app/VM (ZFS storage)
  • Frigate on another box with its own non-ZFS storage
  • virtualised TrueNAS and Frigate on the same host, each with its own distinct storage

How much storage/drives for TrueNAS and for Frigate? What’s the possible extent of “future expansion”?

1 Like

I don’t see anything suggesting the i3-12100T supports ECC. Do you have a link to something saying it does? For iGPU with ECC, there are some Xeon e3 (and maybe other models) with iGPU. For example, I was using an e3-1275v6 in the past.

AMD has been kind of whipping Intel’s butt with efficient CPUs for years now. If they have any server CPUs with integrated graphics, I believe they will only be the latest gen which may not be very price-efficient. However, with the right motherboard and consumer CPU, you can use ECC. I don’t have any specific suggestions on that front.

On that subject, props for not thinking you need a monster of a CPU to share files and run Plex. A lot of people go overboard with their CPU, including myself (but at least knowingly). You don’t need a monster of a CPU for anything you’ve listed except maybe Frigate.

I’ve never used Frigate. I understand it’s all the rage with kids these days. I asked ChatGPT for an idea of hardware requirements, and was recommended a Google Coral TPU. This reminded me of a couple of my nerd friends that were using Frigate + Coral. Unless I’m being lied to, each will only support ~4 x 1080p feeds at 5fps detection speed. A slower detection framerate or lower resolution would allow more feeds. Educating myself, I see there’s an Asus PCIe card that supports up to 8 of the m.2 Coral accelerators. This is probably much more efficient than doing it with a GPU, and way way way more than CPU.

Older hardware would be fine for all of this. I’d look for DDR4 boards as etorix suggested.

Yes, you can have multiple pools.

Yes, you want to run from SMR drives with ZFS, and I’d say for anything that isn’t purely archival.

Exactly. This is essentially what I keep asking myself.

In my mind anyway, I don’t think of surveillance footage as “critical” enough to warrant ECC protection. What might a bit flip result in? A tiny dot in one frame of a video?

OK, maybe the “detection” footage could be moved to a ZFS pool. But this is a much smaller subset of the “recording everything for 30 days on 10x 4K cameras” (getting ahead here, more below) that I want to be able to do.

I kind of view movies the same way, but who knows, HDDs are relatively inexpensive nowadays, and if I were to build a nice big (ZFS/TrueNAS) array maybe I could afford to store even higher bitrate movies than I am now in there, without even hardly noticing it.

I only even bought one camera so far. :smile: But looking around the property, I don’t think I would have too much trouble putting up 4-5, maybe up to 10 (eventually). But since I’m just getting started, I really don’t know, TBQH. :man_shrugging: And it might take me years to do all those installations at the rate I work (I am kind of a perfectionist, but The Missus would just say :turtle:). :smile:

As I started to mention above, I think I want to record all cameras 24x7x365, for 30 days. At first I did not think that was necessary (isn’t the point of Frigate to surface relevant events?) but after discussing with the community there the answer was “well maybe I just want to look back and see what may have happened which did not trigger an event/detection, but for whatever reason might be interesting (e.g., a tree fell down, did it rain on this day, etc.)” which I suppose I became convinced of.

Now of course there are ways to calculate this, and I sort of did some rough ones but anyway of course we could be talking about a significant amount of video here (not initially, but eventually, as I add more cameras) and that’s why I didn’t want to necessarily put that all into ZFS.

The “important/personal” data is what I really want to put into ZFS/TrueNAS, I guess I should add up just how much that is. But it can’t be much, as it is currently all fits on a single 4 TB HDD with (plenty of) room to spare (last I looked).

So my needs really, are quite small (compared to what I think you guys might be imagining). :slight_smile:

Again sorry, I was slightly imprecise. Looking more closely now, I think I should have said “i3-12100[T]E” instead of “i3-12100T[E]”. Anyway, here is what I was looking at:

https://www.intel.com/content/www/us/en/products/sku/132225/intel-core-i312100e-processor-12m-cache-up-to-4-20-ghz/specifications.html

https://www.intel.com/content/www/us/en/products/sku/132224/intel-core-i312100te-processor-12m-cache-up-to-4-00-ghz/specifications.html

It’s very exciting! Essentially nearly cutting edge AI object detection, in a F/LOSS software package! And integrates very well with Home Assistant (or other systems) as it simply sends the events it detects via MQTT! It’s all quite lovely! :slight_smile:

They still are officially recommended (and likely very useful in many cases). However if you carefully follow the hardware discussions (in r/frigate for instance; which I have been doing for months now) and read between the lines, many people think that the future is more open GPU based models. The Coral was a pet project of like one single Google engineer (or something like that, to my understanding) and does not even compile any longer if you follow the official instructions. Many people consider it some level of abandonware (I know, such a shock, coming from Google! :smiley: ). A couple years ago, it was the bee’s knees, especially for the money. But by now, more and more people are reporting like 5-10 millisecond detection times on modern Intel iGPUs, which rivals the Coral.

Anyway Frigate is forging ahead on the cutting edge and support many different hardware setups and models, which is part of what I find so fascinating about the project (and also want to somewhat leave hardware options open).

I am not saying your friends are lying, but they may have something misconfigured. Or this may be out of date information. As the field is rapidly advancing. I just set up a test instance (on some existing hardware) and the documentation today recommends 720p for example on the detect sub-stream (while still recording in full resolution from the main stream).

And I have read some almost unbelievable reports of camera counts and resolutions (on fairly low-end hardware), but I think the state of the art is just advancing that rapidly now (hence my excitement).

I knew that about ZFS, but for some use cases (sequential writes in particular) I have read that it might be OK? And isn’t that exactly what surveilance video would be?

I am not trying to argue, rather understand. As etorix also mentioned above that “surveillance” drives are not SMR. I really don’t know, so I’ve added this to my list of things to research. Some times it can be hard to differentiate between marketing / (IMO, often times artificial) market segmentation, and what is really necessary.

I see now on the CPU. Yeah, looks fine for ECC.

I will bother with Frigate when I move back to my own country :slight_smile: It’s a future project I’m interested in, just haven’t bothered while at a temporarly location. I only recently installed Home Assistant just to get familiar with it.

And no, I didn’t mean friends lying. They were using it for object detection at a very low wattage. I meant the bots lying :slight_smile:

SMR write speeds drop to ‘garbage’ when the cache runs out. They are intended for dropping a file on it here and there or archive, not for write-heavy workloads – like video feeds. I understand for sequential writes it might be alright, but for writes from multiple cameras I’m not as confident. Most will tell you to just avoid them. Stick with CMR.

I started perusing the hardware recommendations and other resources. Some of them are out of date, but I guess the takeaway is to use reliable (“real server”) hardware.

So I started looking around eBay for old Supermicro X10 based boards, which seem to be available for very reasonable prices, pair that with something like an E5-2600 v4 (?) which can also be had pretty cheap, and some DDR4 ECC and for like a few hundred bucks we have the makings of a very nice and reliable TrueNAS / ZFX box, with like 40(!) PCIe lanes to work with!

I could then buy a separate N100 to do the Frigate processing and probably still be coming out at less than what I had calculated for the ASUS Pro WS W680-ACE build.

I am not saying that is even the right thing to do (perhaps I should add some dGPU or something to the Supermicro board instead) and I don’t hardly know the first thing about Xeons and stuff but I guess my point is that everything you guys have been preaching here is starting to sink in.

I guess I just need to start to learn more about what are relevant model numbers/ranges of these kind of boards to start narrowing it down. I would welcome any suggestions in this regard.

Also, what kind of “idle” (I know, kind of misnomer) power draw should I expect with something like that? And how well do those kind of boards support things like C states (if at all)? I just have so much more to learn about these, it’s like a whole another world to me.

I use the Asus W680 (+IPMI card) for my proxmox server with a 12700k. Complaints; the IPMI card isn’t fun to wire up, I can only boot on BIOS version 3101 (though it has been about a year since I bothered trying any bios newer than that), DDR5 ECC UDIMMS are expensive & very hard to find (been ~year since I last looked though).

Other than that? No real complaints; it does the needful. I think it currently has an uptime of like 8 months.

The TE (Embedded) actually does. Good luck finding one at retail!
Regular desktop i3-12xxx do not support ECC.

This unfortunately stopped with Xeon E-2300 (LGA1200).
LGA1700 Xeon E-2400 have no iGPU.
And the price for C256 motherboard (X12STH and the like) with its Xeon E-2300G, or a W480/W580 motherboard with Xeon W-1200/1300, and DDR4 ECC UDIMM might not be much better than W680 with Core i5 and DDR5 ECC UDIMM.

PRC-style AI-assisted surveillance and social scoring at home. Yummy! :male_detective:

So that would be X10SR_ (or X10DR_, dual socket) with Xeon E5 v3/4 and cheap second-hand DDR4 RDIMM. Very cheap overall, with lots of PCIe lanes and possibly lots of computing power.
At this point, you may as well look into 1st/2nd Gen Xeon Scalable, which would be X11SP_ (and X11DP_) boards in Supermicro nomenclature (C621 chipset). Same cheap DDR4 RDIMM. But this is precisely the platform that is known to idle above 50 W.

If your redundant storage is very small, meaning few disks, the efficient platforms to look for are embedded boards with soldered-on CPUs:

  • X10SDV (Xeon D-1500, typically coming as a mini-ITX board with 6 SATA, 1 M.2 and one PCIe x16 slot);
  • A2SDi (Atom C3000, 8 or 12 SATA in mini-ITX).

Both take DDR4 RDIMM, and idle below 10 W. Enough computing power for light containers alongside storage. X10SDV would best be chased second-hand while you might have to buy A2SDi new (but it begins to trickle down to second hand).
And then take Frigate to its dedicated box, with non-ZFS storage.

Correct… for a filesystem which would return the file anyway. ZFS would actually detect the corruption and flatly refuse to serve data that is known to be corrupted. So you’d be better using ext4 or XFS for non-redundant, non-critical storage which you want to access regardless of corruption.
Note, though, that the main risk is probably bit rot on disk over bit flips in RAM.

I believe ECC is supported in the I3 14th gen, at least according to some sites I’ve seen. The ASUS site for the mobo does say it’s not supported, but I think that’s referencing the earlier gens.

ARK says: “No!”

I’ll simply add that I have found the frigate setup process in TrueNAS to be better left to folk who have a lot of time or who already have considerable experience setting up the config file.

To call it an annoying experience is the understatement of the century, especially if your camera streams are not public (ie free to use by anyone on your LAN vs. password protected)

Also, I heartily agree that a coral USB-based TPU is the way to go - low power, high performance, fewer implementation issues vs. the PCIe version.

For the time being I will continue to endeavor to make Frigate work but blue iris (which has user interface issues galore) is still a much easier NVR to set up and it also can make use of Coral TPUs. Unfortunately, blue iris is only available on windows OS.

Interesting. I’ve seen about a dozen review sites that explicitly state ECC is supported.

The manufacturer probably knows best, but it’s odd that so many other sites think it is.

1 Like

Just gonna throw my 2 cents in re: ECC memory & LGA1700:

I have a Raptor Lake 13600K, an ASRock Industrial IMB-X1314, with W680 chipset, and 4x Kingston KTL-TS432E/32G. The CPU, Motherboard, and memory are all capable of ECC. Support is explicit for all of these pieces of hardware.

However. There is no EDAC module for Raptor Lake CPUs on W680. Which means as much as I have hardware support, there’s no module in the Linux kernel to surface bit errors and correct them.

You can verify this on your own systems by proceeding to the directory /sys/devices/system/edac/mc – if there’s no mc0 entry for your memory controller, you don’t have ECC support.

I’d love to be corrected if needed, but this is my understanding. ECC support isn’t easy to come by cheaply.