Truenas-csp error

TrueNAS General
1

I am attempting to setup the hpe-csi/truenas csp for kubernetes pv via iscsi. pvc are stuck “pending” with the logs stating:

Wed, 02 Apr 2025 13:44:03 +0000 backend 7 ERROR Not found: Volume with name pvc-aaa7c306-971e-4432-8285-171affccf9ac not found.

Wed, Apr 2 2025 9:44:03 amWed, 02 Apr 2025 13:44:03 +0000 backend 7 ERROR Backend Request (POST) Exception: 422 Unprocessable Entity: Traceback (most recent call last):

Wed, Apr 2 2025 9:44:03 am File “/app/backend.py”, line 446, in post

Wed, Apr 2 2025 9:44:03 am self.req_backend.raise_for_status()

Wed, Apr 2 2025 9:44:03 am File “/app/lib/python3.12/site-packages/requests/models.py”, line 1024, in raise_for_status

Wed, Apr 2 2025 9:44:03 am raise HTTPError(http_error_msg, response=self)

Wed, Apr 2 2025 9:44:03 amrequests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https redacted api/v2.0/pool/dataset

Wed, Apr 2 2025 9:44:03 am

Wed, Apr 2 2025 9:44:03 amWed, 02 Apr 2025 13:44:03 +0000 backend 7 ERROR Bad Request: TrueNAS API returned: {

Wed, Apr 2 2025 9:44:03 am “path”: [

Wed, Apr 2 2025 9:44:03 am {

Wed, Apr 2 2025 9:44:03 am “message”: “null not allowed”,

Wed, Apr 2 2025 9:44:03 am “errno”: 22

Wed, Apr 2 2025 9:44:03 am }

Wed, Apr 2 2025 9:44:03 am ]

but in the audit log I see the method call:
Method: pool.dataset.create
Params:

  • Type: VOLUME
    Comments: >-
    Dataset created by HPE CSI Driver for Kubernetes as
    pvc-aaa7c306-971e-4432-8285-171affccf9ac in hpe-storage from test
    Name: main/iscsi2/pvc-aaa7c306-971e-4432-8285-171affccf9ac
    Volsize: ‘10737418240’
    Volblocksize: 8K
    Sparse: True
    Deduplication: ‘OFF’
    Sync: STANDARD
    Compression: LZ4
    Description: Pool dataset create main/iscsi2/pvc-aaa7c306-971e-4432-8285-171affccf9ac
    Authenticated: True
    Authorized: True

however, when looking at the datasets, I do not see it is actually created and I cannot find or do not know how to find any system logs indicated why the api call to create the volume has failed. How to proceed with troubleshooting this failure?

2

You can run midclt call audit.query | jq to see full audit entries. /var/log/middlewared.log also will contain information about failures.

3

the first command does show “success: false” but I have exported and searched the middleware.log and it does not show any error related to this

can’t paste screenshot

in fact, it has very few entries for today’s date at all, less than 5

4

What were the pool.dataset.create parameters from the audit.query response? What version of TrueNAS is this?

5

I actually got this sorted out, the extent was a block device but needed to be a filesystem

6

Generally zvols should be used for iscsi, if for some reason the HP integration is pushing you towards using file-backed extents then it is doing the wrong thing.

7

On the truenas side the iscsi block share is on a zvol dataset but yes, the hpe-csi is requiring filesystem extent

8

The root dataset where the zvols are created needs to be of filesystem type. It’s a bit foreign to me that you would create a zvol dataset and nest other zvols in that dataset.

Is there any benefit/drawback for one over the other?

9 
root@truenas[/home/truenas_admin/debian]# zfs create -V 1G dozer/zv1
root@truenas[/home/truenas_admin/debian]# zfs create -V 1G dozer/zv1/zv2
cannot create 'dozer/zv1/zv2': parent is not a filesystem

If you’re talking about this, the main disadvantage of trying to nest a zvol in a zvol is that it doesn’t work :wink:

1 Like
10

Yeah, this makes sense. Maybe I should just pass that error message along for the folks who skips the manual.

11

I think in general it would be best to pass along validation errors to consumers. We generally try to be verbose in error messages. For that matter, as we transition to a clearer versioned API you can probably add your own error handling for common situations.

It looks like our API validation here leaves a bit to be desired. I’ll make a pull request with a change to improve the error message.

12

The problem on my end is that users are getting this message thrown at them:

requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://10.233.0.11/api/v2.0/pool/dataset

Wed, 16 Apr 2025 11:46:12 +0000 backend 8 ERROR Bad Request: TrueNAS API returned: {
 "path": [
  {
   "message": "null not allowed",
   "errno": 22
  }
 ]
}
Wed, 16 Apr 2025 11:46:12 +0000 backend 8 DEBUG Falcon Response (to HPE CSI): 500 Internal Server Error

Which demands “path” is being set for the zvol, while it’s being inherited for a filesystem. So, I guess things could be improved in both ends.

13

Yes. That’s the validation error I just fixed in above commit. The error was being raised by an internal filesystem.stat call.

1 Like
14

One other tidbit you may not have seen with 25.04:

>>> requests.get('http://127.0.0.1/api/versions').json()
['v24.10', 'v25.04.0', 'v25.04.1', 'v25.10.0']

Starting in 25.04 you can get a list of supported API versions through api/versions while unauthenticated. This can be used to for version detection and to simplify maintenance on your end.

So you can develop for instance against the v25.04.0 API and continue to use it while it’s in that list (connecting to api/v25.04) and be comfortable in knowing that there are no substantial changes that will require work on your end.

15

Oh, interesting, so I can use the v24.10 API on a v25.04 box? I’m going to switch to the WebSocket API for v25.04 onwards.

16

Try to stick with 25.04 +. What language is your integration in? If python you can user our python truenas API client.

17

I’m using Python and the plan is to rip out Requests for the TrueNAS API client.

18

Feel free to ping me if you run into any problems with the API client.

1 Like