Hi all,
First post here. I’m very much new to all this and I’ve bit off more than I can chew, I think, so I’m doing my best to strengthen my proverbial jaw. I’m running TrueNAS Scale and the web UI refuses to load beyond the card backgrounds and such, but no data loads in the cards. I can access the shell and the settings and the Network page, but pretty much nothing else loads.
I am also dealing with tasks that refuse to complete, mostly pool.dataset.sync_db_keys, which I’m guessing is a snapshot task, but I’m not totally sure, and I haven’t found good information on it from Googling it.
I’ve been searching for hours trying to figure out what’s going on and how to fix this. I’ve seen various posts about middlewared, a potential issue with certificates, Linux kernel 6.6, and I’m sure other things that I’ve forgotten. I ran a SMART test on my boot drive and it came up squeaky clean. I’m not sure what middlewared is (similarly to the aforementioned task, when I Google it I just get posts describing issues with it but nothing that explains what it actually is) and I’m not the greatest network admin.
I’d be very grateful to anyone who can help me solve this issue (and help me learn more about how TrueNAS works in the process!).
System info:
TrueNAS Scale v24.04.1.1 running as a VM in Proxmox
Kingston 240 GB boot drive
Intel Xeon E5-2650
16 GB of RAM
(What other specs should I include here?)
Since virtualizing TrueNAS, we need to know exactly how it is setup.
Looking for info like how Scale is run on Proxmox, how you pass through devices, disk / pool setup.
Link to some resources
Thanks for the reply. Sorry I couldn’t be quite as timely.
To be quite honest, I don’t really know how to give you the information you’re looking for. A friend of mine helped me purchase and set up my server and everything seemed to be going (relatively) well until now. Unfortunately, he’s been rather busy lately and in an effort to become more self-sufficient and not have to rely on him/make him my tech support, I’ve been trying to take care of issues myself as much as I can. But the more I read, the more glaringly obvious it becomes how little I know.
That is to say, I don’t know how to give you the information you’re looking for (and yes, I did read both the articles you linked). I was able to confirm with the aforementioned friend that we set up the server with drive-based passthrough with plans to change to PCI-Passthrough, but we ran out of time and forgot to go back to it.
As for how Scale is run on Proxmox, I’m not entirely sure what you mean by that nor how to answer (it’s run as a VM, but clearly you’re looking for more information than that), and I’m not sure exactly what you’re looking for in terms of disk/pool setup.
I’m sorry I couldn’t be of more use. Help me help you help me?
Why are you running TrueNAS on Proxmox instead of TrueNAS directly on the machine?
What else runs on Proxmox?
The articles cover how to, more safely, run a Virtualized TrueNAS but I don’t think your hardware setup is right at this point. Your hardware specs didn’t list a separate HBA and the HBA needs to be passed through in Proxmox along with all the HBA attached drives. They will not be available for anything else on the Proxmox machine.
Your first post shows 16Gb of RAM. Is that the entire physical machine or it that just what is assigned in Proxmox for the TrueNAS VM?
I’ll point you to the documents section. Become familiar with TrueNAS Scale version you are running, look for the ZFS Primer. (Reference Section at bottom of page)
Go over the documents and articles and talk to your friend who did the setup.
I’m running TrueNAS on Proxmox because I have other VMs set up in Proxmox as well.
Turns out I do indeed have an HBA, an LSI SAS2008. And that makes sense because, as I’d mentioned, we’d intended to set up the drives using it, but we ran out of time (it was like 2am when we were doing this) and forgot to come back to it.
The total system has 64 GBs of RAM, 16 of which are assigned to TrueNAS.
I am working on reading the documentation, and because I’ve been using TrueNAS for the past six-ish months, I am familiar with much of it as far as using the UI is concerned, but where I’m unfamiliar, well, let’s just say that this stuff doesn’t come naturally to me. So I’m trying my best to learn, but it’s slow going.
I was going to ask, would rebooting the system cause problems? Maybe it just needs a restart? But I’m not sure how that would affect pending tasks (that, had the system been working properly, been completed a week ago).
Also attaching a couple sample images of the UI failing to load just so there’s some visual examples, so you can see what’s going on.
Are you able to backup your configuration and try doing an update to the latest version of TrueNAS Scale? TrueNAS-SCALE-24.04.2
I’m guessing you can’t even get the menus to load.
It is probably best for you to wait for your friend, go over the documents for running TrueNAS on Proxmox / virtualization and do a reinstall with the recommendations in the guide.
You could also try rebooting the TrueNAS and choosing a previous boot enviroment if you have one. (any previous truenas updgrades)
It’s funny you should mention that. This problem cropped up after I recently upgraded to the latest version (at the time, 24.04.1.1, also noted in my initial post in this thread). I didn’t have internet connectivity set up so I didn’t realize I was a couple major versions behind. I got the networking set up, updated, and a day or two later the UI went bye bye as seen in the screenshots above.
I definitely could try updating. That seems to be one of the few menus that does load, which is minorly encouraging.
I should note that I’ve also been talking with the guys on the Discord. Eventually we figured out that I do have an HBA card and it should already be in the correct (IT) mode. I just have to actually go into Proxmox and pass it through. And that’s where, once again, things get tricky. Watching this tutorial, there seem to be two ways to accomplish that passthrough (setting up a resource map vs. sending the card directly to the VM(, and I’m not sure which is more correct. Furthermore, it looks like I should have the VM powered off to complete/configure the passthrough and the idea of rebooting it has occurred to me a number of times (I wondered if the good 'ol turn it off/turn it back on again refresh might solve everything), but I’ve been mightily concerned that turning it off would cause some sort of damage to the pending tasks that never complete (I think the pool.dataset.sync_db_keys tasks are snapshots, but I haven’t been able to find a good answer on that).
All told, I can see two next steps: update TrueNAS and/or set up the HBA passthrough. I think both will require the VM to be powered off at some point, so I’m guessing I can stop worrying about that specifically. Between updating TrueNAS and setting up the HBA passthrough, do you think there’s one I should prioritize over the other? They both need to be done, of course, but in what order?
HBA passthrough is needed. It should be covered in the links in my first post above. Pass through the entire controller and all the HBA controller attached disks.
You can try an upgrade but you will end up doing a reinstall, I think, to do the passthrough conversion of the HBA.
Ok, so I got IOMMU enabled now (and in the inevitable reboot process, the TrueNAS UI fixed itself and is now usable! Hooray!) and based on what I understand from your articles, it is best to add the HBA to the TrueNAS VM directly (TrueNAS VM item>Hardware>Add PCI Device>Raw Device>HBA card) rather than doing a device map.
Here’s what I don’t know, though. Currently all my disks are on drive-based passthrough (did a whole bunch of qm set {VM ID number} -scsi{drive #} {drive serial number}). Do I first need to somehow unmount them or something (I have no idea whether or not that’s the proper term) before adding the HBA card to the VM? If so, I would assume I need to then mount (or whatever) them to the card before passing it to the VM?
I have never done a conversion like that. I would have a backup of the configuration and all the data in case of a big failure. Yes, you would want to attach all the TrueNAS related disks, boot can be excluded, to the HBA and then do the pass through setup. I don’t know if it will require a reinstall of TrueNAS or it will just work.
Maybe someone else will post that has Proxmox and TrueNAS experience. Otherwise, coordinate with the person who helped you set it up in the first place if they are more experience with Proxmox and TrueNAS.
I did have the foresight to back up the config already, so I’ve got that done, at least.
I am trying to work with my friend when I can, but he is oft busy and not often online. Getting ahold of him can be quite a task.
Nonetheless, thank you for all your help thus far.
