After trying to understand better the latest changes/moves of the TrueNAS Core development, I fear that the initial promise of an easy setup for a hyperconverged cluster is going to be a disappointment, or am I understanding things wrong?
In particular after dropping glusterfs and now it seems even kubernetes will be replaced with docker…
What I’m talking about is a simple setup with, say, a two node cluster (two HW boxes) each with at least two mirrored disks for storage, and the cluster should be able to automatically mirror the storage and ALSO be able to move containers from one node to the other seamlessly, depending on load and resource availability.
Is there an alternative hyperconverged platform other than TrueNAS that is striving into this direction otherwise?
Sounds like you should create your own hyper converged cluster with kubernetes with K3s, K8s or RKE2 with rancher at that point. Truenas is a storage appliance first with a few more bells and whistles. Not a one size fits all solution.
Exactly then you confirm it… somehow I was hoping in the beginning that with glusterfs etc one could have a solution that is still not too complex to install on 2-3 servers… otherwise doing what you suggest is surely possible but nothing streamlined like installing things ala proxmox (which seems to have at least ceph support, but not really good container orchestration) or anything similar…
Sadly it seems so… but it’s one of those examples were an open source project would be still so valid and based on a really great idea that it’s even more sad and almost unexplicable that no other group of people or company (like iX maybe?) could have picked it up and continued to run it…
In open source it should not be like if another company stopped putting efforts on it (had to choose between ceph and gluster), it means it’s not valid or doable any more…
“It seems”? Yes, this was announced months ago; iX is ditching the Kubernetes-based apps system in favor of Docker Compose because maintaining Kubernetes was too hard.
If you’re looking to cluster, no, that’s been permanently removed from TrueNAS’ scope. I think the best alternative in that regard would be Proxmox–it does support clustering for both compute and storage, HA with auto-migration of VMs or containers as necessary, and even some degree of ZFS management. What it doesn’t do well at all is shared network storage, though that could be added in a few different ways.
Regarding “shared network storage”: I heard that proxmox should support Ceph multi nodes with HA fairly well, or you mean specifically sharing that then via SMB/NFS/iSCSI?
your comment has me curious. are you referring to the cephs storage of proxmox? or its network shared storage? what types of issues have you had that makes you say that?
i ask as someone who is currently running a 3 node production system using truenas scale as the shared network storage location.
during our criticality testing we observed migration within 20 seconds on a 1g network, and have had several real world events with the same results. it has so far worked flawlessly. (not to say it wont ever have an issue)
wondering if there are issues i am not aware of that i may need to plan for.
The “issue” is that Proxmox doesn’t including any file-sharing features. It isn’t a NAS. I don’t know how I could have been more clear about that. I’m not talking about Ceph; I’m talking about sharing storage out over the network like TrueNAS, unRAID, OMV, etc. do.
i see i misunderstood your comment, i thought you were referencing running the VMs on proxmox from a shared network location not file sharing from the PVE installs themselves.
What I think people tend to do is to expose SMB/NFS/other file sharing protocol through a lightweight LXC, which in turn accesses the network filesystem which is managed by Proxmox via a bind mount (onto e.g. CephFS). Or alternatively a VM with storage on a virtual disk on top of Ceph.
Ceph by the way seems awesome. I wonder if iX would even consider integrating it into TrueNAS.
Do you have a link to the source repo? I could not find any. Website last updated in 2021. Inquiries about the source by me in the followup of EuroBSDCon 2021 were answered very evasively. Then communication stopped.
Perhaps you’d consider looking at SmartOS. It won’t do clustering ‘out of the box’ for 1 or 2 nodes, you need 3. This has fail-over and load-balancing capabilities by running TritonOS. They also have a product called Manta which is a distributed object store that is pretty rad. All of it is Open Source.
If you have any questions about installation etc, the folks hang out here: SmartOS | TopicBOX
Start with SmartOS to see if that works for you. If it’s what you need and you just want redundancy and load balancing you can roll your own whatever you need for your app and this will take care of the replication to multiple locations and instances.
Might be a bit more than what you wanted to chew on, but it’s a pretty amazing system that doesn’t get enough props so I thought I’d mention it.
