[updated] Can't log in, console screen blinking (25.04)

TrueNAS General
1

I don’t know what to call this. Please help.

System was working fine this morning. I spilled a cup of water on the keyboard so i unplugged it. Grabbed a new keyboard. Old keyboard was plugged into back of chassis, hard to get to so i plugged new into front USB port. Console screen flashed some message about hotplug i was not quick enough to see, then server powered off and restarted.

99% certain i don;t have any security features like that turned on, but going to check BIOS now to confirm.

Long story short, my server seems to boot up, with errors, but i can’t log in or use the console.

IMG_7231
IMG_72311512×2016 1.35 MB

This is my console screen, over and over, all it does:
[video]: Imgur: The magic of the Internet
[screenshot]:

IMG_7230
IMG_72301533×721 75.8 KB

… And I get this when i try to log in on browser:

image
image762×824 44.4 KB

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/ws_handler/rpc.py", line 323, in process_method_call
    result = await method.call(app, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/server/method.py", line 40, in call
    result = await self.middleware.call_with_audit(self.name, self.serviceobj, methodobj, params, app)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 906, in call_with_audit
    result = await self._call(method, serviceobj, methodobj, params, app=app,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 715, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 93, in wrapped
    result = await func(*args)
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/account.py", line 1256, in has_local_administrator_set_up
    return len(await self.middleware.call('privilege.local_administrators')) > 0
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1000, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 715, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/account_/privilege.py", line 437, in local_administrators
    users = await self.middleware.call('user.query', [['local', '=', True]])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1000, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 715, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/api/base/decorator.py", line 93, in wrapped
    result = await func(*args)
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/account.py", line 430, in query
    result = await self.middleware.call(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1000, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 715, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 174, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 147, in query
    result = await self._queryset_serialize(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 199, in _queryset_serialize
    return [
           ^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 200, in <listcomp>
    await self._extend(data, extend, extend_context, extend_context_value, select)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 213, in _extend
    data = await self.middleware.call(extend, data, extend_context_value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1000, in call
    return await self._call(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 715, in _call
    return await methodobj(*prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/account.py", line 287, in user_extend
    user['sshpubkey'] = await self.middleware.run_in_thread(self._read_authorized_keys, user['home'])
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 622, in run_in_thread
    return await self.run_in_executor(io_thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 619, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/account.py", line 272, in _read_authorized_keys
    with open(f'{homedir}/.ssh/authorized_keys') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 5] Input/output error: '/mnt/kobol/Stuff/users/colin/.ssh/authorized_keys'
2

IO/ Errors from your data pool aren’t a good sign. Do you have SSH access to it?

3

Nope. like a total noob dope, my private key is stored on a pool on the NAS. Lesson learned.

4

Well, can you provide more information about the pool characteristics / hardware? Getting IO errors from the pools is a sign that something is seriously amiss (this is an unusual situation – you shouldn’t be getting IO errors for filesystem IO).

5

i wonder if my SAS cards are poorly seated or something, or possible the SAS-to-SATA breakouts are poorly seated on the cards…

6

[edited for clarity]:
Sure! System specs in signature, and this linked post from earlier this week contains my zpool list and status, and a table of how my drives are installed/connected:

7

Having I/O errors for filesystem paths is not a UI error. If I were to hazard a guess, it’s hardware-related, but people in the forums will need more information about your actual hardware.

1 Like
8

the HW info was why i linked the post, i edited it for better clarity of my intent - zpool list, zpool status, and a table of all my drives and how they are connected - plus my signature has the rest of my system config in it… if anything is missing, please let me know, i do aim to be thorough.

9

If for some reason you can’t ctrl+c out of the CLI prompt in console, you can switch virtual terminals (ctrl + alt + F2) for instance and try some credentials.

10

OK thank you - I shut down and checked that all my SAS cards and cables were seated properly. I found one drive with a bad connector on it, it is probably hanging by a thread and maybe was causing the IO issues? Either way my system has booted up now.