Why does my TrueNas Scale quit working after upgrade to 24.04.2.3?

This version quit working after upgrade after 24.04.2.3 upgrade

The Network cards did not activate. After shutting down and dropping back to 24.04.2 , things are working fine.

OS Version:TrueNAS-SCALE-24.04.2
Product:Z690M-ITX/ax
Model:13th Gen Intel(R) Core™ i5-13500
Memory:15 GiB
System Serial:To Be Filled By O.E.M.

Has anyone elese experienced this …??

1 Like

it broke my network as well.

like example jailmaker docker.

i see docker is running but the links don’t work after updating.

how did you downgrade? how do you get the previous version?

tried looking here but couldn’t find it
https://download.sys.truenas.net/

24.04.2

found this

is 24.04.2.2 ok?

downgrade failed

FAILED
[EFAULT] Unable to downgrade from 24.04.2.3 to 24.04.2
remove_circle_outline
More info...
 Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 469, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 511, in __run_body
    rv = await self.method(*args)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 187, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update.py", line 428, in file
    await self.middleware.call('update.install', job, destfile, update_options)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1564, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1428, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1321, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/install_linux.py", line 30, in install
    raise CallError(f'Unable to downgrade from {old_version} to {new_version}')
middlewared.service_exception.CallError: [EFAULT] Unable to downgrade from 24.04.2.3 to 24.04.2

how did u downgrade yours?

You can revert to a previous boot environment with

System → Boot → Select the 3-dot menu by 24.04.2.2 → Activate

Then reboot your TrueNAS system.

@theosparkx @mooglestiltzkin can you share your network card model?

1 Like

i use a 10Gtek 10Gb PCI-E NIC Network Card, Single SFP+ Port, with Intel 82599EN Controller, PCI Express Ethernet LAN Adapter

10Gtek 10GbE PCI-E X8 Network Card X520-10G-1S (compare to Intel X520-DA1 )

its installed in truenas server. i connect it to my network switch sfp+ via a OM3 sfp+ fiber optic cable. Why? cause low power and i figured i’d try fiber for once :laughing:

it worked fine before i applied the recent update 24.04.2.3

hm so i boot environment back to 24.04.2 which was my last working version

but seems to still have the same issue :smiling_face_with_tear:

the only change that cause the break was upgrading from 24.0.4.2 to 24.04.2.3

some bugs i noticed

  • previously i could do 192.168.0.x:5001 to go to my docker web page. but now i can’t. i checked jailmaker is running, docker is running, container is running.

but regular smb access via file explorer still works :sweat_smile:

This was how i originally setup the networking for jailmaker

and after doing that, to go to truenas > general

Web Interface IPv4 Address: set this to the other ip. Why? so that nginx proxy manager that also needs 8080 can still work, while allowing truenas to also have a functional 8080 without having to fight over it.

so my docker stuff was using a different ip to truenas.

i think i set the ip used by docker in jailmaker separately? can’t remember x-x;

anyway was working fine before this update. previous updates never had this problem :frowning:

*update

also after using boot environment to downgrade, i tried uploading the config prior to updating that caused the issue. but the problem still remains :confused:

Another thing i noticed broke was hostnames.

Seems it had reset from my own hostname back to truenas

so i had to edit it again.

under truenas > network > global configuration > (hostname) *change this back to yours. Noticed this got reset

for networking i check what i previously edited when i initially was setting up.

in addition to what i mentioned,

source

so i checked the file, it is still

# SPDX-License-Identifier: MIT-0
#
# This config file is installed as part of systemd.
# It may be freely copied and edited (following the MIT No Attribution license).
#
# To make local modifications, one of the following methods may be used:
# 1. add a drop-in file that extends this file by creating the
#    /etc/systemd/network/80-container-host0.network.d/ directory and creating a
#    new .conf file there.
# 2. copy this file into /etc/systemd/network or one of the other paths checked
#    by systemd-networkd and edit it there.
# This file should not be edited in place, because it'll be overwritten on upgrades.

# This network file matches the container-side of the virtual Ethernet link
# created by systemd-nspawn's --network-veth switch. See systemd-nspawn(1) for
# details.

[Match]
Virtualization=container
Name=host0

[Network]
DHCP=false
Address=192.168.0.24/24
Gateway=192.168.0.2
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=customer-bridge

[DHCP]
UseTimezone=yes

so that seems to be correct.

the other setting i checked is truenas networking. these are the active interfaces

enp19s0
br1 = 192.168.0.210/24
vb-docker

192.168.0.210 is for truenas ip. for docker it’s suppose to be 192.168.0.24

note: this br1 i can’t remember if it’s suppose to be 24 or 210. when i tested 192.168.0.24 test, i didn’t see any improvement, so i figured that doesn’t work if i edit there. so i figured just having the static ip set to 24 is at the 80-container-host0.network file

but when i check 192.168.0.24:5001 for dockge, it don’t load.

so i went to shell did a

jlmkr list, jlmkr status docker

jlmkr shell docker

then docker ps

it says jlmkr is running, the jailmaker docker is running. and the container is all running. so not sure what part broke or how to fix x-x;

problem began after the recent update

next i used stux guide to double check my settings

jlmkr list

NAME   RUNNING STARTUP GPU_INTEL GPU_NVIDIA OS     VERSION ADDRESSES    
docker True    True    False     True       debian 12      192.168.0.24…

so it shows the ip for it should be correct…

jlmkr shell docker

docker ps

CONTAINER ID   IMAGE  

louislam/dockge:latest                                    "/usr/bin/dumb-init …"    5 months ago   Up 6 hours (healthy)   0.0.0.0:5001->5001/tcp, :::5001->5001/tcp                                                                                                                                                dockge
root@docker:~#

tried https://192.168.0.24:5001

but not working

any ideas why it’s not working :cry:

*update

just tested today now it works

https does not work, but http does.

http://192.168.0.24:5001

not sure if updating again to the newer version would break it again, but i will wait for now ^^; cauz i got it to work again.

I admit I skimmed your post and didn’t watch any linked videos, but a quick thought is that if you are doing your network setup manually by editing /etc-files and the like, then your loss of settings between upgrades is is essentially self-inflicted.

TrueNAS throws away the OS-files during an upgrade and sets things to whatever the TN database indicates. That’s why the only supported method of changing the configuration is to use the GUI or included CLI-tool.

but this has never been an issue with previous updates. just only now o-O;

anyway i managed to fix it. simply change boot environment back to previous. and also fixed up another issue i had cropping up

https://notthebe.ee/blog/easy-ssl-in-homelab-dns01/

If your zone is in Setup for over 28 days, it will be automatically deleted.

auto delete for free cloudflare users. but for paid cloudflare users, it remains working even if u didn’t name server activate, unless u manually remove it urself. this broke so i redid it to fix it.

but i think this was a separate issue because that only affected when using domain url

e.g. mywebsiteurl.com

. i used lan ip url e.g. 192.168.0.24:5001 which normally just works regardless of whats up with cloudflare.

anyway works now.

ps: one docker app whoogle didn’t work, but restarting it, worked after that. so i guess either restart all my docker apps or just restart truenas is in order :sweat_smile:

That may be, but you’re not supposed to edit files in /etc in TrueNAS, as it’s an appliance OS.

I am not sure I understand what the issue is. Just complete the setup. If you don’t complete the setup you can’t use the domain anyway, even during those 28 days, so what’s the point?

i simply follow guide to get stuff to work. guide doesn’t explain what they ommited but i brought up.

its for working https cert using lets encrypt for a no remote homelab. all details on the youtube and website if u r curious
https://notthebe.ee/blog/easy-ssl-in-homelab-dns01/

anyway i digress.

just recap on update what i noticed broke

  • hostname, i think this got reset? or at least it worked before it didn’t after the update.

the solution i found was to edit the hostname and change it back. tested this then worked. before i did this, i had to use 192.168.0.210 for it to work instead of simply typing the hostname.

  • the next issue was docker urls no longer working. i also tested 192.168.0.24:5001 I think i also tried http when https didn’t work, but i’m not too sure if i did. this also happened straight after updating.

not sure which fixed it exactly, but i changed boot environment back to previous working version, then upload previous working truenas config. and also did the wolfgang fix as mentioned. and restart docker containers still not working. then this fixed it all back to working condition again.

It’s a broken Kernel, simple as that. I have a logitech KB/Mouse combo attached as well to a Chenbro case. Booted, selected older kernel via Web Dashoard, viola, we’re back. Disabled / deleted other kernels, set the keep flag for the 24.04.2 kernel. Might have to change the boot option permanently for now. Works great (very Fast) with a Asrock z690 ITX, I5 CPU. I looked at the Atom MB/CPU offerings and quite frankly cost wise while a little more. I feel it’s got a better deal than say turn key units (Netgear screwed me over) or QNAP.

boot kernel? where in dashboard?

network card is dual 10Gtek 2.5 rt6189 ??. Now not able to assign IP4 address after upgrade to 24.10-RC.

Got everything to work after some late night poking around.
#$$$$$ ******** Final ***************** $$$$$#
Used Chenbo SR301 Chaiss (solid unit) → Broken door sucks !!! , Chenbo support rocks, helped me fix the problem.
a) removed internal fan shroud
b) Inverted chassis HD fan to push upwards
c) Added second chassis fan 2200 rpm
d) theratake low profile LGA1700 fan

Intel I5 125000 CPU/GPU
ASRock Z690M-ITX/ac Motherboard Mini-ITX
16Gb DDR5 x1
M.2 Crucial 500gb M.2 MVMe drive (Truenas Scale 24.2.03)
M.2 6 Port sata adapter @ 6Gbs
San Disk 500Gb Sdd (Truenas 24.10.01 RC)
Segate Iron Wolf 1Tb Hybrid (win 11)
Not used → LSI SAS3008 12Gbps SATA/SAS 8-Port w/ unRAID+2*SFF-8643 SATA Cable
4x 14Tb WDC Utrastar HD (recertified by Amazon)
10Gtek 2x 2.5 Gbe PCIe adapter

Notes

  1. TrueNas needs to be connected to a managed switch to support more than one NIC
  2. LAGG really not tried yet
  3. ASUSTOR 5304T is the primary source
  4. 2x Linux Desktop Clients

So it had nothing to do with the “Kernel” and everything to do with inadequate cooling?

No. truenas needs a managed switch for multiple NICS it seems. I had it connected to an unmanged 8-port trendnet 2.5 spare. Nothing would trigger DHCP from the router. Connected both NICS to the QNAP viola. Extra cooling was ancillary, to make sure those HD don’t melt.

saw the new eel update and tried again.

after update, my dashboard was a mess. but after edit everything seems ok.

for jailmaker i delete old one and create new one. the nvidia card settings in the jailmaker config breaks jailmaker from working. so i disable that and it worked after that.

then i got jailmaker and docker in it running. i did a jlmkr list and it shows the 192.168.0.24 i use for the docker

then i deployed dockge docker compose, so that is running.

  1. normal usage of truenas for nas purposes work
  2. jailmaker works
  3. docker in jailmaker is running

earlier the url wouldnt work 192.168.0.24:5001

*yes i tried using private browser to make sure its clean but that didnt work

hours later i tried again and it worked


*screenshot as proof

So now i will procede to deploy the rest of my containers docker compose saved in the stack folder.

So successfully recovered after migrating from dragonfish to eel for jailmaker with docker running in it.

The networking for it might still be flaky (for me). Will continue to monitor if it’s the case or not.

But i did delete old jailmaker and redid that from scratch.

What i did not do from scratch was the networking, but maybe i should. But i can’t recall the exact steps i did x-x; scared i can’t enter truenas if i botch that up thats why i hadn’t done that yet.

This was the step to doing it

the sfp+ fiber connection is correctly being shown in truenas. this was the update in case there was confusion. It was simply a matter off editting the dashboard to choose to show it. Because prior it didn’t show it for some reason after updating to eel from dragonfish.

But it remains true that i keep losing access to my container everytime i try access via 192.168.0.24:5001 *this is for dockge as an example

but i deleted and redid and hopefully that problem goes away will see

*update

figured out the problem. it had nothing to do with truenas. it was my own misconfiguration in router in regards to networking for docker not working most of the time

my bad