Help deciding on a custom build vs used server

TrueNAS General
22

This pool will be faster overall than a RAIDz pool, not just during a resilver operation. Looks good to me (but I ran for years with 3-way mirrors :slight_smile:

I’m going to tackle this one in pieces. This pool is for a local replicated copy of Pool A?

I’m not sure I understand what you mean by the second sentence.

The way ZFS replication works is that it is always based on snapshots. And snapshots are, by definition, read only. Let me walk through an example.

PoolA@1 is the first snapshot of Pool A
PoolA@2 is the second snapshot of Pool A
PoolA@3 is the third snapshot of Pool A, etc.

The very first replication is copy all of the data and metadata from PoolA@1 to the copy, assume we are putting the copies in PoolB. After the initial copy you will have

PoolB/PoolA@1

When it is time to replicate PoolA@2, ZFS knows the blocks that were changed between PoolA@1 and PoolA@2, so the replication process just has to copy those blocks. You end up with (on the replica side):

PoolB/PoolA@1
PoolB/PoolA@2

Repeat for PoolA@3 and now we have (including both source and replica

PoolA@1
PoolA@2
PoolA@3
PoolB/PoolA@1
PoolB/PoolA@2
PoolB/PoolA@3

If you ever destroy (delete) the last snapshot of Pool A, then replication will fail until it is restarted from the beginning as there will be no snapshot to base a replication on.

If the snapshots on Pool A have a retention period, they will be destroyed (deleted) based on that retention period. The replicated snapshots on Pool B may have the same or a different retention period than the snapshots on Pool A. I use the same retention period on all my replicas as the primary. But different snapshots have different retention periods, for example, hourly snapshots are retained for 1 week, daily snapshots are retained for 1 month, etc. But both the primary snapshot on my production system and the replicated snapshots on my backup system have the same retention.

Yes, as copying to the same pool will use the same drives for reads and writes. When you create a snapshot, no data is copied, just internal ZFS metadata, and that operation is very, very fast.

I am very confused by the above. ZFS snapshots are not stored separately from the pool, they are a read-only point-in-time copy of the data in a dataset and are part of the pool. They are very lightweight, containing only data blocks that have been changed since the snapshot was taken. They have no performance penalty because of the copy-on-write nature of ZFS. ZFS protects against bit-rot for all the data, both active datasets and snapshots. Having a copy of the snapshots in a separate pool on the same server protects against data loss if the source pool is damaged and cannot be recovered.

On my production server I have a variety of snapshots with a variety of retention periods. I replica all of them to my backup server located a few feet away. I have a plan for a backup server on the other coast of the country at a friend’s house that will also replicate all the snapshots from the production server.

This also makes sense (except the snapshots to an external drive).

1 Like
23

Well, my system is as follows:
(you can see it in my signature for details, however the full structure is under restructure (again))

  • Main file and backup server.
  • it is a chinese LGA 2011 MoBo AN E5-2695 v2 CPU, 28 GB DDR3 ECC RAM, * 120 Gb M.2 for system, a
  • 1 (or2???) TB M.2 for apps and VMs
  • 2x6TB WD Blud Drive for Backups (I live with mysons, so there are about 4-6 WIndows based machines to back up) in a mirrored config
  • 6x4TB WD Blue HDDs (removed from external USB cases) for all, non critical data, mostly ISO-s and media files
  • Backup server:
  • Similar, Chinese LGA 2011 MoBo, 32 GB DDR3 ECC, E5-2630L v2 CPU
  • 64 GB Samsing sATA SSD for TrueNAS system
  • 1x6TB WD BLue (same as in the main server) to back up the other pool
  • 1x16TB Seagate HDD for backing up my media library
  • Remote backup server in another country
  • very similar to the backup server, LGA2011, E5-s650L CPU, 64 GB DDR3 ECC,
  • M.2 500GB NVME forsystem and Proxmox
  • 1x6TB for backup important data
  • 1x16TB to back up media files
    They run this config since like mid 2023. No data loss or HDD lost so far.
    What I have a lot of probelms recently is my network infrastructure, my router died, my mesh wifi APs are dying one by one, and finally PLex decided to discontinue the only important feature I used them for, the remote streaming option.
    So I am busy right now to rethink my whole network infrastructure and buid it.
24

I realize my mistake. I thought that snapshotting and replication are done in a single operation. And after that, there’s only a single file which represents the difference since the last snapshot.

Having that in mind now, is this kind of a replica (pool “B”) actually useful? Like these points:

  1. It reduces wear on the source drives as further replications to external storages only use the primary replica (pool “B”), so the source pool has less wear. But then, money saved by excluding pool “B” of the system could be used to update drives on the source pool sooner and compensate for the faster wear-out.
  2. Also, replication from pool “A” only occurs once (because replications to multiple destinations will be done from “B”), so “A” throughput is more available for intensive read/write operations of user data. That intensive use most probably won’t happen, but still…
  3. It could rise reliability and failure resistance in cases when a source pool fails. There’s easier and probably faster access to the latest snapshots to restore the original pool.

Also, “A” and “C” could be the one pool with different datasets. At least, in the beginning. Thus, drive expenses are distributed more evenly over time. I can’t understand if my insights on having several pools for different purposes do really make sense… or are they feasible if this will be the very first experience? I’d be happy to build a cheaper system and gain some experience and statistics. Then I’d know what I need. Now I may just spend money on a suboptimal solution. And then spend weeks or months before actually utilizing it.

25

So now I think it might also be wise to have two smaller systems instead of one big. E.g. to have second NAS for primary replication destination (pool “B” in my example).

1 Like
26

i5 and higher… Trust Intel to NOT make it easy.

Er… WS W680 boards only support DDR5.
Savings would come from going for older hardware. E.g. a ca. 200 € AsRock Rack E3C246D4U2-2T from eBay with DDR4 and a dirt cheap Core i3-9100 (ECC here!) or second hand Xeon E-2100/2200.

If you saw my opinion on the old forum, I certainly was not impressed with my own Define 7, the many screws and the fiddly trays. In this mid-tower size, the Nanoxia DeepSilece 8 Pro is much, much easier to work with a storage case.
With a micro-ATX board, the Node 804 is also worth considering.

Oh, you’re another case of a very unusual layout.

You get a larger drive and leverage copies=2 :wink: (just don’t tell our resident jester Winnie…)

Raidz resilver is slower… but at these sizes it’s nothing to worry about.
As for performance, if you had 12 drives, then 6 mirrors would beat two 6-wide raidz2 on IOPS. With your single vdev pools, no layout is going to shine (but raidz2 will beat mirrors on writes).
You’re overthinking optimisations—especially on your first NAS. Keep it Simple.
And a second copy on another pool but in the same machine is only half of a backup: A rogue RAM module, drive controller or suicidal PSU will take them both down together.

A single large raidz2 will just offer more resiliency to all your data, with less drives. (9 HDDs with W680M would require a HBA… for just one extra port.) You might still segregate in datasets by value, and replicate separately.
Although with so little data there’s an easy performance optimisation available: Put what needs most performance on SSD! You may then look into Ryzen (AM4 or AM5, the latter including EPYC4000) for x4x4x4x4 bifurcation on the PCIe slot. Or use SATA SSDs.

16 GB Optane M10 make great boot drives. And so would 256 GB M.2 drives sold after a laptop upgrade".
Redundant boot is, in my opinion a total waste on a home server. Keep a copy of the configuration file. Installe new drive. Install TrueNAS. Load the configuration. Done. Less than 30 minutes downtime… and I have yet to personally see a boot drive fail. (Data drives? Yes, let’s not talk about the body count…)

Which wear? Wear from reading?

Another sentence which does not make sense.
Different pools will necessarily have different datasets. But you can have different datasets in the same pool, with different parameters for different purposes.
Which purposes and use cases by the way? Storing video files and hosting VMs are different purposes requiring different pools (raidz / mirror). Storing large and smaller data files, or serving different files to different users, can be done with datasets from the same (raidz#) pool.

27

All very fine and accurate info… but your drive count and use case (VMs) is totally at odds with the OP here, isn’t it?

Much longer than a 4 TB drive, but that’s why it’s at least raidz2 with this size and counting ONE failure only. And if the failure occurs in a stripe of mirrors, the table turns: A 4-wide raidz2 will resilver safely (again, from ONE failure), while the stripe of mirror will have an “at risk” window that is too large for comfort.

28

Oh well… another mistake. I thought RAM controllers are backwards compatible with older modules.

I understand now how hugely wrong way I was going to go. Will try to find something like that.

OK, wrote this down. But the Node form factor is weird with big footprint. Ideally it has to be something either slim or tall.

Considering my initial misunderstanding of snapshots and replicas, this additional replication pool probably just adds more complexity instead of reliability. If replica is made from the original pool, it means that snapshot is already protected from a bit-rot if that original pool is redundant.

Overprovisioning appears to be a good strategy when you don’t know what you’ll (actually) need. The only problem is $$$.

I understand this part. It was meant to “relieve” the original pool of replication (to multiple targets one-by-one). And if something(one) breaks the data pool, snapshot replicas are just by hand. But again, it depends on what blows up first - a pool or common hardware (e.g. PSU as you mentioned).

Alright, wrote this down too.

Yes, that’s what I meant.

The idea in that sentence was: “Instead of having two different pools for different types of data, I could have one pool with two datasets.” Just to save on complexity, power consumption, motherboard and case requirements… alright, even on total drive expenses.

The important data includes graphical projects (the working copy will be on a PC/laptop), family pictures, art library.
The unimportant data includes footage from camera (edited mostly), and similar stuff. It’s not completely expandable, I’d like to have it for as long as possible, but definitely not the kind I can’t survive without.

Probably there will be 1-2 simultaneous users. Not sure about video streaming yet.

29

So if we are talking purely about storing lots of data, I would go for a RAIDZ2. And the new RAIDZ2 expansion feature will certainly makes things easier in terms of capacity planning.

You will need very little CPU power (like 4 cores) but should be generous with RAM. If you cheap used DDR4 ECC RDIMMs I would not go lower than 64 GB.

Is power consumption a factor?

And I join those who don’t recommend the Fractal Define R7 for many drives. Cooling is a major issue because the drives are mostly covered by the fiddly mounting sleds. So I had to add high-pressure fans, which basically make it as loud as. a rack-mount case.

1 Like
30

Not necessarily.
Your speed most likely will be heavily bottlenecked by your network speed. (if its 1GB/s)
Snapshoting should faster than a normal backup.
Also, if you install a new system just for that, you will get increased power consumption, and higher risk of something breaking down.
And you have to spens some (and maybe a lot) of money to do it.
BUt it all depends on you!

31

Snapshots and backup are two different things for two different purpose.

  • Snapshots are to provide an option to revert any unwanted or malicious activity.
    (like, if you delete something by accident, or you got infected by a file ecripting virus. Those malwares work like they encrypt the files one-by-one on your storage. If your important files are on a separate server, they get indeed encrypted, but in the background, TrueNAS recognises that the file is about the change, so it doe not overwrite the original file with the encrypted one, but creates a new copy of it. YOu WILL SEE it encrypted, until you righ click on the file and select “restore older version” and then you will get back your original file.)
  • Backups are more against losing your whole system mostly because of any catastropic failure.
  • Of course, a backup can be used to restore after an attack too, but doing a full (but even an icremental) backup every hour is really resource intensive task.)
32

I also recommend ALiexpress for looking around.
RAM is really cheap there nowdays.
I bought 64GB DDR4 ECC for 60EUR 2 weeks ago.
YOU can buy second hand, new or refurbished RAM there.
(And, I suspect that chinese also recycle RAM in industrial scale (they buy used RAM desolder the chips and resolder them on brand new PCBs), just like they do with old Chipsets with their X79 and X99 motherboards.)

33

I can’t tell. I don’t even know the consumption of my current PC. Probably depends. The lower, the better. But when a difference in price is 400 eur vs 1500 eur, I can stand some difference in power consumption.

It’ll be 2.5gbps. I thought it’s enough for the raidz2 as its throughput is equivalent or even lower than 2.5gbps.

Aren’t snapshots + replicas the “backup tool” coming out of the box with ZFS?

What? You mean they sell used stuff priced as new?

34

Someone mentioned used workstation. I found a used one for 260 euro.

Dell Precision T7610 Tower
Intel Xeon E5-2696 v2
64 GB DDR3 ECC Registered
480 GB SSD
SATA III
1300W PSU
DVD-RW
NVIDIA Quadro K620 2GB

Seems to be a suitable thing, right?

Unfortunately, it doesn’t have IPMI, but for this price I forgive it. Server (for IPMI) will cost much more (at least where I searched them).

35

A backup tool is not a backup.

The critical thing is to understand what event can cause which damage. If you have separate pools in the same machine, the backup pool would survive the loss of the primary pool because of HDD/SSD loss. But if the primary pool is killed by a voltage spike from a faulty power supply, the backup pool will likely be killed by this event, too.

If you have a separate machine for your backup pool, this scenario would be covered. But your house burning down would not be covered.

And so on …

Lastly, one needs to be aware that the vast majority of data loss is due to human error. This has a profound impact on the scenario analysis.

The key to not loosing data is to be paranoid and keep things simple. At least that is what has worked for me since 1991, when my 42 MB Seagate ST251-1 disk had faulty sectors.

1 Like
36

IMO you shouldn’t consider buying ddr3-era tech. Unless you take saving another hundred euros as a matter of sport.

1 Like
37

Generally not. Alder Lake is actually compatible with either DDR4 or DDR5 but motherboard manufacturers hardly ever implement two sets of RAM slots to allow either option on one board; at best you can choose between two board models, one for DDR5 and one for DDR4.
I have not seen a W680 board with DDR4.

Your criteria, your pick…
The Node 804 looks bulky because it’s nearly a cube, but the actual volume is not larger than your average mid-tower. Good airflow but not silent (drive noise goes straight out the top mesh).

Key here is to keep drive count low. With only “bulk” data (and not so much of it), you could do with a raidz2 pool, 4 to 6 wide, fitting in a Node 304, which is quite a small case, well-cool and reasonably quiet.
The only constraint is that the Node 304 only takes mini-ITX. A Supermicro X10SDV is a perfect fit… if one can find one for a decent price. (A Chinese seller on eBay has one slightly more modern AsRock Rack D1622D4I left. I should get mine next Tuesday; accepted offer at $230. Can’t help hoarding hardware… :wink: )

Old and bulky, but certainly suitable. Your searching skills are improving fast!
:+1:

38

Sample more or less in my signature line. Over fanned because I had them. :grinning:

My Dell R720 server just hums along but it is noisier then my builds with more drives.

39

There were ssome exotic MoBos in rthe DDR-DDR2 and the DDR2-DDR3 transition era, that had support for both typeof RAM, but then Intel moved its memory controller interface from the North Bridge into the CPU, so it got hardwired.
Also, some CPUs in the DDR3-DDR4 transition ERA had variants, that were DDR3 and DDR4 compatible, so you could put them into both kinds of MoBos (but I am not 100% sure about this, only have some faded memories as if I read about this.)

40

Then I would say, its not :smiley:

41

Well, this is not htat simple!
YOU can buy new stuff, you can buy used stuff, you can buy recertified stuff, and you can buy “recycled” stuff. (this last category is really is a Chinesess thing.)
Usually, I pick the “within my budget” stuff.
I am totally OK to buy second hand RAM cheap.
I always run extensive RAM testing after I receive them and return them if they aree faulty.
Aliexpress has a rather good kind of customer service. YOU dont even have to bother with the original seller, you can directly open a dispute with ALiexpress itself, you write down your problem, and attach some pictures or videos of it.
The only stupid thing is when your package ie stolen (when the courrier reports it delivered, but you dont get anyithing). They require a photo proof of that you did not received it. That is really stupid :smiley: