Upgrade failing 24.10 -> 25.04 [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2580)

TrueNAS General
1

Fails no matter what browser I use, whether it is HTTP or HTTPS. Checked that the NTP time is enabled and correct timezone. It appears during or right after downloading an update after switching channel to 25.04 from 24.10.

Please advise on how I can do the upgrade without a full reinstall.

Full traceback of error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 443, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 566, in read
    data = self._fp_read(amt) if not fp_closed else b""
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 532, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 473, in read
    s = self.fp.read(amt)
        ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1314, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1166, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2580)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 627, in stream
    data = self.read(amt=amt, decode_content=decode_content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 565, in read
    with self._error_catcher():
  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 454, in _error_catcher
    raise SSLError(e)
urllib3.exceptions.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2580)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 509, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 556, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1367, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 183, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update.py", line 304, in download
    return self.middleware.call_sync('update.download_update', job, train, location, 100)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1654, in call_sync
    return self.run_coroutine(methodobj(*prepared_call.args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1694, in run_coroutine
    return fut.result()
           ^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update.py", line 309, in download_update
    return await self.middleware.call('update.download_impl', *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1460, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/download_linux.py", line 7, in download_impl
    return await self.middleware.call('update.download_impl_scale', job, train, location, progress_proportion)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1629, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1471, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/update_/download.py", line 61, in download_impl_scale
    for chunk in r.iter_content(chunk_size=8 * 1024 * 1024):
  File "/usr/lib/python3/dist-packages/requests/models.py", line 824, in generate
    raise RequestsSSLError(e)
requests.exceptions.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2580)
2

During is what it looks like to me. Can you download the manual update file from https://update.ixsystems.com/ and update that way?

3

I tried, got file `TrueNAS-SCALE-25.04.0.update1.54 GiB` – on upload, the system says

Error submitting file
0 Unknown Error
4

On your desktop client, what is the SHA256 hash of the file you downloaded?

It should be this:
4ce3e1e88bcc0fee44d9c7ac617c19b845d21b321ef420fbd3ce6e567334ffde

If the hash matches, then the problem is likely on your TrueNAS server, such as the NIC.

5

Bad RAM or CPU would be my guess.
What hardware do you have?

6

Bad RAM or CPU would be my guess.
What hardware do you have?

7 
$ sha256sum TrueNAS-SCALE-25.04.0.update
4ce3e1e88bcc0fee44d9c7ac617c19b845d21b321ef420fbd3ce6e567334ffde  TrueNAS-SCALE-25.04.0.update
$ export SHA=4ce3e1e88bcc0fee44d9c7ac617c19b845d21b321ef420fbd3ce6e567334ffde
$ export THEIR_SHA=4ce3e1e88bcc0fee44d9c7ac617c19b845d21b321ef420fbd3ce6e567334ffde
$ diff <(echo $SHA) <(echo $THEIR_SHA)
$

Didn’t trust my own eyes to diff that so used diff, looks the same.

8

A UGREEN DXP NAS. Not very convenient (it’s headless and on a shelf down in basement) but I can try to boot it into a memtest. No replacing the CPU so :crossed_fingers: it isn’t that.

9

Could also be a faulty NIC.

10

If a memtest passes, can you boot it into a live Ubuntu USB and try to download the update file manually to check its SHA256 hash?

11

Turns out it is bad ram and the UGREEN NAS uses DDR5. Price has gone up 5.6x! Yikes. I’m trying to A/B test each stick with memtest86 to see if at least one stick is good on its own for now. (Half the RAM would still be plenty. RAM was cheap in 2024!)

Otherwise I might part this out and look into a way to move my drives to a new build while cannibalizing ddr4 from my current VM machine, combining compute + storage into one for as many watts savings as possible. (My network rack also runs off solar+battery with grid as a backup power source, although it pulls from grid more than I’d like right now. Having everything in one server would improve battery usage. But that’s a topic for another thread!)

12

If you’re lucky, it might be a setting in the BIOS, such as running it at a certain voltage or speed that is making the RAM fail. See what options are offered in the BIOS.

If it’s not that, hopefully you can remove the bad stick and have enough memory to continue running TrueNAS without issues.

I have symbolically added your post to this satirical list. Pretend you made it on the list. :wink: I had to stop adding to it some months ago since it was getting too long.

13

I understand, but we’ll see what happens. I didn’t touch the settings for RAM and this isn’t a gaming PC motherboard, so I don’t think it is trying to overclock it or anything. I’ll go back and check the original specs on the RAM I purchased and make sure that voltage, timing, etc., is correct in the BIOS. (I’m not super impressed with the UGREEN version of basic BIOS either, so I’m leaning towards not recommending these things at all – but now that DDR5 is so expensive and other things like refurb 3.5” server drives are much more expensive than November 2024 too, I can’t really recommend any of this for a server/NAS.)