SYSTEM INFO:
MOBO: ASUS Prime B350-Plus
RAM: 16 gigs
CPU: AMD r5 2600
GPU: AMD RX580
STORAGE:
1tb HDD SEAGATE - ST1000DM003-9YN162
1tb HDD WESTERN DIGITAL - WD10JPVX-22JC3T0
64gb SSDGAMERKING SATAIII
(these are two pretty old drives that I plan to replacing in a few months/in the next year
I need some personalized noobie advice about my TrueNAS server. I started my server last month in order to have an addition backup to my important music projects/files/samples as well as self host my own immich server (photo/video server) for me and my family.
For now everything’s working great and I can even access immich remotely with tailscale but eventually I want to expose immich to the internet with nginx and cloudflare. I am running into a few issues: (other than some checksum errors on the HDDs )
nginx requires being installed on an ssd. This means I can’t just replace my two old HDDs with newer ones and I don’t want to buy an ssd strictly for nginx.
If i’m getting an ssd, I should get two right? so I can mirror them?
2a. I’m thinking that the photos will reside on the faster storage and the file backups that I don’t need to access will go on slower HDD storage that I already have.
I saw the other special vdevs and looked into what they do but I don’t understand if it would benefit me to have them and how I would implement them into my system (theoretically with 2 ssds and 2 hdds).
3a. How would I understand what the bottlenecks in the system are? Again, would these special vdevs even help in my case because what if my internet is the bottleneck.
I have no idea what I’m doing and I would love an alternative build idea to the ones I proposed. I’m also thinking of changing out my cpu to one with integrated graphics so I can ditch the gpu. Thank you in advance for anyone who decides to help me out. I really really appreciate it. I’ve been finding all of this server and networking stuff really fascinating so i’d appreciate advice and any links to readings to better help me understand what the best build is for a photo server + file backup server.
There many people much more knowledgeable than myself here. This guide might be a helpful reference. A few more details about your use case and future plans will help members give solid advice. This is my two cents, please take it with a pinch of salt.
I don’t think this is a required. A quick Google and I don’t see any other mandatory requirement.
Most of mirroring or RAIDZx is to provide drive redundancy increasing uptime. I can’t stress enough redundancy is not a backup! Backups are meant to save you from catastrophic data corruption and/or hardware failure when the system is not recoverable. To truly protect against both of those scenarios a off-line backup and/or cloud backup with the following strategy. Follow 321-backup-rule. How much redundancy and number of backup comes down to your risk tolerance and use case.
Backing up applications are immensely more complicated. Why? Because the application that ingests the data like pictures becomes just as important as the raw data itself. Think about it yet a terabyte of photos that tagged, organized into albums, shared with other users and so on. That data would be lost even if you could recover the pictures from a backup. Not to mention how these apps are deployed can have an impact on how easy it is to recover your data from a failed application.
Backing up an application and restoring is not really supported yet by truenas. A key piece to backups that are automated and actually tested.
If you implement a backup system take heed of the following advice before you invest utilizing your server. Backup the server and at least one other place, then wipe the server clean including the OS!
This will force you to prove your backup system and give you experience of how to actually restore from the worst-case scenario. I myself am holding off until there’s an official backup and restore method for applications.
You are correct that applications and data such as photos stored on ssd will have more performance. There is something to consider before committing to a ssd. SSD’s are expensive 1 terabyte drive this can be anywhere 60$-100$. What is the cost of expanding that storage in the future? What happens when you longer have physical space for a drive. You have to migrate the pool to larger ssd drives 1 TB to 4 TB for example. Something to think about.
RAM plays a critical role in ZFS performance. 16 GB is a good place to start but ZFS will utilize all RAM available. Calculating RAM requirements is something loosely based on “network throughput”, “number of active users”, “storage size” and x running applications/services.
Immich utilizes AI for image categorization and so forth. Your GPU AMD RX580 might be helpful there however there is a cost of power consumption. Others might have better advice. I’m not sure even cutting edge APU (CPU with integrated GPU) can be utilized as an alternative. The AI equipped APU market is still so new in the software might be under baked for support.
Every app can be installed on rust disks, but SSD are faster. Personally i install app in SSD, same for db/metadata/config… Instead, media are on rust disks.
Have redundancy Is strictly adviced. But if you have budget/other limit( ex. no more sata available), on ipotetical small app dedicated pool, you can consider to just keep 1 disk and setup a dayli replication task on rust large pool to recover in case of need.
Like I said, my use case will strictly be for storing photo/video on immich and having a file server and I don’t have any future plans for this besides working on the networking in order to access immich through the internet.
Some more details:
The file server will not exceed 1 tb (float around 500gb) and will not be touched except in the case of a disaster on my local system. On the other hand the immich server will constantly be accessed and I expect it will also be around 200gb for the next few years, of course inching toward 500gb in a decade or twoish.
Is there a way to calculate how much RAM I should have like you said with this use case in mind? I think the idea of having a pair of HDDs and a pair of SSDs would be best and I don’t mind paying a little extra for more performance. How should I set up my vdevs and where should my apps be?
Sooooo, i’ve tried multiple times with nginx and it seems people have the same problems that I have with it deploying forever. My only theory is that it’s my slow HDDs causing the problems. I’ve tried adding on a string of code onto the env variables section of nginx with no luck.
Thank you for this, I will definitely take this to heart as my photos are one of my most prized possessions.
I’m looking into ways that I can backup the entire immich application like you mentioned and it seems to be impossible with my current configuration because I installed it in the IX-systems directory. This directory doesn’t show when I try to make an SMB share or RSYNC task…
So to sum it up, what I think I should do is:
Get 2 new 2tb HDDs and 2 new 1tb SDDs for a total of 3tb of storage with redundancy. This should last me at least two decades.
File server AND backup of Immich will be on HDDs (~700gb)
Immich (with data), Tailscale, Nginx (if I get it working) will be on the SSD vdev (~200gb)
Backup of entire Immich app on my portable hard drive
This seems to satisfy the 3-2-1 rule and gives me a good amount of security? All I need to do is figure out:
How to backup the entire Immich app?
How to I replicate the app onto the HDDs?
What would it look like to completely wipe the system and restore it back to it’s original state? Is this basically just uploading the TrueNAS config file and having the immich app restored and then I’m good? Or is there more to this?
Correct and it is problematic to use SMB with an application that’s not the set up for the explicit use case. Say if you choose your data directory for immich and also set up with SMB. immich is going to make assumptions about what is in the directory when Photos could be deleted, renamed or moved. SMB is going to need to apply permissions from the user to that existing dataset. Those permissions could conflict rendering immich data unreadable or SMB unreadable.
A rule of thumb storage is to never exceed 80% of total capacity. 800 GB for a 1 terabyte of space. Does that change your projected needs?
I would say start out with your 16 GB of RAM and you can always add more RAM if needed. It’s one of the easiest things to upgrade.
Your motherboard doesn’t support PCIe bifurcation. It enables a larger slot, such as PCIe x16, to be split into multiple smaller lanes, such as PCIe x8 or x4. With support you can get a cheap pcie4.0 x16 to m.2 m-Key nvme x 4 ssd Expansion Card the more expensive route Multi-NVMe (m.2, u.2) adapters that do not require bifurcation Down the road you’ll appreciate having NVME over SATA. So think about if you ever have to upgrade your motherboard. It’s the number one suspect for hardware failure years down the road.
Right now your motherboard supports the following PCI Express
PCI Express 3.0 x16 AMD Ryzen Processors
1 x PCIe 3.0/2.0 x16 (x16 mode)
AMD 7th Generation A-series/Athlon Processors
1 x PCIe 3.0/2.0 x16 (x8 mode)
PCI Express 2.0 x16 1 x PCI Express 2.0 x16 slot (max.@x4 mode) *3
*3 PCIeX16_2 slot shares bandwidth with PCIeX1_1 and PCIeX1_2.
PCI Express x1 2 x PCI Express 2.0 x1 slots
PCI Slots 2 x PCI Slots
For now SATA drives until your upgrade cycle for the server. I’ll doubt you get 10 years out of the motherboard.
Highly recommend Tailscale rather than exposing ports. If you must have exposed to the Internet Tailscale Funnel would be a better way to expose the server to the net. However as always use at your own risk.
I believe you’re describing the following Nginx Proxy Manager won’t deploy When setting it up in the UI add the ENV variable, S6_STAGE2_HOOK=sed -i $d /etc/s6-overlay/s6-rc.d/prepare/30-ownership.sh
If anyone says just use host path that is not a true solution. database corruption can happen with the running application with the database being backed up live (transactions with the database might be interrupted). In this case simple ZFS snapshot won’t guarantee data integrity. These kind of things usually reveal themselves during restoring data. Obviously that’s the worst time. Shutting down the container before backup should mitigate these issues however host path has other issues.
I’ve had issues upgrading applications that have utilized host path leaving them in a broken state. One upgrade might go well, later it might break next upgrade. Since then truenas electric eel leveraging docker/docker compose. I can’t say whether or not that’ll have any improvement with the issue above. Bottom line upgrading applications is unreliable and we don’t have a real backup solution for applications. I would recommend to wait for a proper backup solution for applications rather than relying on some community guide.
I gotta say even ZFS snapshots are stupidly confusing. If you restore early in the timeline of ZFS snapshots it seems you lose the ability to restore to a later point because data is lost during restore. Makes experimenting restoring snapshots seemingly one a time deal.
If I could wave a magic wand it would be being able to restore a snapshot anywhere on the timeline and temporarily evaluate restored data. if the temporary restore looks good then commit that snapshot for permanent restoration pruning other snapshots that rely on the restored snapshot.
There is the option to clone the dataset instead of restore and you’ll have to reset up your applications to point to those paths but to me this is too much complexity. Backups should always be simple, transparent and automated. Truenas needs to take the human and system complexity out of backup and restore as much as technically possible.
A while ago I did watch the video. Think about the complexity of the different containers and folders required for immich. The average person don’t think about ZFS data pools or how snapshots are handled from a global snapshot list that needs to be filtered. They think about an application name and the dates that they want to restore.
I appreciate your contributions Stux! I say that all above not to diminish the power of ZFS or your awesome video. It’s just a critique of the status quo in truenas following the ZFS paradigm with minimal abstraction that leads to a poor user experience. Truenas needs to provide best practice on the backend and a simple interface on the frontend. The front could have a simple interface with the standard ZFS exposed behind advance UI element.
I sincerely hope not in this case as truenas already has advanced UI elements for power users. It would be a travesty if ixsystem didn’t really implement robust backup and restore that’s accessible simply because they would push people to HexOS cloud-based subscription.
Out of all those features backup/restore support is the most important. They’ve acknowledged that backup/restore support is going to be implemented. It’s usability though be entirely dependent on how much they rely following the ZFS paradigm. Anyway I don’t want to co-opt this thread in to loosely related topics.
Thank you @Stux and @Trudge for helping me understand this complex topic. I think I’m starting to understand everything. Snapshots are so interesting!!! I read about them in the zfs101 document out there but It seems sketchy. If snapshots are so powerful can’t I just take periodic snapshots of the ix systems folder and effectively have my immich application backed up??? Especially if I can have the snapshots themselves be automatically backed up on my local system or my portable hard drive like I said before. Is this safe?
Well, yes, when you take a snapshot like that you are taking a “crash-consistent” backup, ie its a backup, as long as the system is capable of recovering from a random crash elegantly.
I would hope most systems are.
I run my servers with sync enabled on their datasets, which helps with crash consistency, but I also have Optane SLOGs.
Ideally, an app consistent backup would be used, which actually triggers a backup from inside, or alongside the app. Its preferred to use app consistent backup techniques to backup databases and other systems, but if the system should be able to recover from a spontaneous crash, then a snapshot should also be sufficient.
What does having “sync enabled on their datasets” mean? And how can I ensure that my system will be fine after said spontaneous crash? Automatic downloads of the config file and tiered snapshots?
)
