Storage disk resize failing (SLOG Over-Provisioning)

TrueNAS General
1

Hey, everyone!

I’m using TrueNAS SCALE version ElectricEel-24.10.2.2.

I want to add a dedicated SLOG device to my RAIDZ3 pool. As the “SLOG Devices” documentation says:

The current TrueNAS project recommendation is a 16 GB SLOG device over-provisioned from larger SSDs to increase the write endurance and throughput of an individual SSD, though this is not the only usable size for SLOG devices.

, so I went go through the “SLOG Over-Provisioning” documentation (mind you the latest available version is for 23.10).

The “Resizing a Disk for Over-Provisioning” section mentions using the storage disk resize command. It seems that the disks parameter’s syntax has changed, as when I went to <server_address>/api/docs/restful/#/disk/post_disk_resize, it says it needs to be a list. Since I was still getting some unknown error via the cli command, I used this command instead:

midclt call disk.resize '{"disks": [{"name": "nvme0n1", "size": 32}], "sync": true, "raise_error": true}'

When I inspected the output of midclt call core.get_jobs, I see this output:

{
    "id": 452,
    "method": "disk.resize",
    "arguments": [
        {
            "disks": [
                {
                    "name": "nvme0n1",
                    "size": 32
                }
            ],
            "sync": true,
            "raise_error": true
        }
    ],
    "transient": false,
    "description": null,
    "abortable": false,
    "logs_path": null,
    "logs_excerpt": null,
    "progress": {
        "percent": 0,
        "description": "",
        "extra": null
    },
    "result": null,
    "result_encoding_error": null,
    "error": "[EINVAL] disks: Not a list\n",
    "exception": "Traceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/middlewared/job.py\", line 509, in run\n    await self.future\n  File \"/usr/lib/python3/dist-packages/middlewared/job.py\", line 554, in __run_body\n    rv = await self.method(*args)\n         ^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3/dist-packages/middlewared/schema/processor.py\", line 178, in nf\n    args, kwargs = clean_and_validate_args(args, kwargs)\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3/dist-packages/middlewared/schema/processor.py\", line 172, in clean_and_validate_args\n    verrors.check()\n  File \"/usr/lib/python3/dist-packages/middlewared/service_exception.py\", line 72, in check\n    raise self\nmiddlewared.service_exception.ValidationErrors: [EINVAL] disks: Not a list\n\n",
    "exc_info": {
        "repr": "ValidationErrors([ValidationError('disks', 'Not a list', 22)])",
        "type": "VALIDATION",
        "errno": null,
        "extra": [
            [
                "disks",
                "Not a list",
                22
            ]
        ]
    },
    "state": "FAILED",
    "time_started": {
        "$date": 1747568891000
    },
    "time_finished": {
        "$date": 1747568891000
    },
    "credentials": {
        "type": "UNIX_SOCKET",
        "data": {
            "username": "root"
        }
    }
}

You can see the error is disks: Not a list. I’ve tried multiple different ways to provide the disks argument, but it still fails.

Any ideas? My hunch is that there is a bug in the parsing of the JSON arguments (specifically the disks list).

2

I though there was a button in GUI for that. But my suggestion would be to do it outside of TrueNAS using the relevant utility (manufacturer’s, nvme or nvmecontrol).
JSON does not look it it is meant for humans to parse, much less type. :scream:

3

I just did this through the TrueNAS CLI shell #7. Not the API or SSH-cli.

image
image347×356 13.7 KB
.

Even if this doesn’t solve your problem through your method. I do believe that it would solve it nonetheless. :smiley:

4

If you need performant [sync] writes, why do you have raidz3? Looks like a contradictory configuration.

1 Like
5

Thanks for your replies!

I though there was a button in GUI for that. But my suggestion would be to do it outside of TrueNAS using the relevant utility (manufacturer’s, nvme or nvmecontrol ).

I couldn’t find such a button in the GUI. I’m considering using the nvme CLI tool :+1:

I just did this through the TrueNAS CLI shell #7. Not the API or SSH-cli.

I imagine it is the same as what you get when calling the cli command in shell.

If you need performant [sync] writes, why do you have raidz3? Looks like a contradictory configuration.

I want redundancy first. I’m hoping an SLOG device would speed up the occasional SMB [sync] large file transfer a bit, as the current speed is abysmal.

6

SLOG is not a write cache, so this is not likely to work as you hope.

7

I’m assuming that the real question, given the original post, is “why a SLOG with z3?”.

I’ve had configurations where the most important factor was high capacity at least (reasonable) cost. This implied z3 on spinning rust. Without a SLOG and when using sync traffic (NFS), that means that the client was waiting on two writes to spinning rust; first the on-disk intent log and then the final data. This resulted in periodic pauses visible to the client. In some cases, poorly written clients would detect it as a problem network mount and would detach, thus killing the user’s workflow.

By adding a fast SLOG there was only one write to spinning rust and those pauses and related cascade errors went away; the client saw the sync commit right away and could continue.

3 Likes
8

I understand how separate intent log works, but that’s not the point really.

Why does it automatically imply RaidZ3? RaidZ1 will yield much better performance, and at the same cost you can have 3 raidz1 vdevs tripling the IOPS ceiling.

Have you actually calculated how much redundancy you need? Raidz1 is great, for example for 5-disk array this brings annual probability of failure from 10% to under 0.01%. Raidz2 costs another drive to make make that abysmally small probability even more abysmally small, below that of other system components. It’s almost always an overkill.

Raidz3 is therefore unthinkable – if you have calculated RaidZ3 to be appropriate – I’d love to know your usecase and see the calculations.

  1. I would turn off sync for SMB transfers altogether.
  2. I don’t think improving performance of occasional IO will meaningfully change performance.
9

I wasn’t trying to say that it always implied z3, just that it did in this particular case. (And actually I’d misspoke; we deployed z2 rather than z3, but the argument still stands.)

In this case we were deploying multiple 8-wide vdevs, 24 disks per shelf; z2 gave us 120TB per vdev (20T disks). RaidZ2 was considered to be the minimum acceptable level of redundancy as losing an additional disk during resilvering would take down the whole pool, with a long TTR. Despite the best of intentions, it’s not always possible to avoid duplication of manufacturer and lot number within a vdev, so there’s always the worry about coincident failures.

This was a small operator so keeping the initial cost down and adding additional 8-wide vdevs to the pool over time was a lot more paletable than a large initial outlay, especially when it came to adding more shelves.

So in this case it was a large z2 pool where a SLOG was warranted. I suspect that if I’d gone with a z3 the results would have been substantively the same, but of course I have no proof.

Also, unless I’m missing something, the OP never mentioned the geometry of their pool other than having at least one z3 vdev of unspecified width, so I’m not following the numbers on “3 raidz1 vdevs cost the same as a single raidz3”. Certainly the redundancy differs (as you mentioned elsewhere). I guess 3 3-way z1 vs a 9-way z3 might be one possibility, but I think we’re assuming too much.

2 Likes