How to obtain file system disk usage from truenas

[orsiris]

Hello,

I’m sorry if that question has been asked multiple times, but I can’t get to wrap my head around it.

I’m using Prometheus / Grafana to monitor my whole infrastructure.

I’ve installed a TrueNAS Scale 24.04.2 recently.
So far, I’ve added a graphite endpoint to TrueNAS to get the metrics into Prometheus.
But on both Graphite and NetData, I can’t seem to find metrics for file system usage.

I’ve configured /etc/netdata/netdata.conf file (quickly) by enabling diskspace plugin with following conf:

[plugin:proc:diskspace]
        update every = 1
        space usage for all disks = yes
        remove charts of unmonted disks = no

I only get available disk space, but every metric that should show used disk space is 0 bytes. Any idea how this is supposed to work ?

Also, as side question, how am I supposed to monitor replication status ?

Best regards.

[MikeO3]

This is probably not helpful as I do not use SCALE or Prometheus or netdata… and will assume when you use the term filesystem you mean pools.

I just started looking into this myself due to the lack of “disk space” measure I decided to investigate pool fields size and free. Not decided how to use or present the data but calculated field can be used to get available as you can see some examples in my mock-up.

This is influxdb with the graphite feed from TrueNAS CORE so I don’t know if your able to expose the same series/measures. Works for me as the pools on servers are standard names of p0,p1,p2 etc. so easy to expose on any host. Tactical for now but probably need do a dynamic pool list based on the host disk drives.

Feel free to flag the post to have deleted if this is not helpful.

[orsiris]

@MikeO3 Thank you for your answer.
While I do make those kind of calculations on other systems too, I cannot find any logic in the metrics I gather via graphite:

disk_space{filesystem="_mnt_stor",instance="truenas01p",job="truenas",op="avail"} 53331.45
disk_space{filesystem="_mnt_stor",instance="truenas01p",job="truenas",op="reserved_for_root"} 0
disk_space{filesystem="_mnt_stor",instance="truenas01p",job="truenas",op="used"} 0.0002441
disk_space{filesystem="_mnt_stor_es",instance="truenas01p",job="truenas",op="avail"} 53331.45
disk_space{filesystem="_mnt_stor_es",instance="truenas01p",job="truenas",op="reserved_for_root"} 0
disk_space{filesystem="_mnt_stor_es",instance="truenas01p",job="truenas",op="used"} 0.0009766
disk_space{filesystem="_mnt_stor_hv",instance="truenas01p",job="truenas",op="avail"} 53331.45
disk_space{filesystem="_mnt_stor_hv",instance="truenas01p",job="truenas",op="reserved_for_root"} 0
disk_space{filesystem="_mnt_stor_hv",instance="truenas01p",job="truenas",op="used"} 2688.947

While the available disk space can make sone sense (eg 53331 GB available), the used metric doesn’t make any sense to me:
For example
Dataset /mnt/stor/hv uses 2630 GB, which show well.
Dataset /mnt/stor/es uses 4000GB, but shows a very small usage.
I guess that’s because it’s just an empty dataset that has child dataset which uses those 4000GB, but there is no metric for child datasets.
Also, zvols don’t show either, so I have plenty of fun trying to get storage space utilization.

So this could be a graphite mapping issue, but then, even netdata shows bogus data:

/mnt/stor/ex usage as seen by netdata:

/mnt/stor/es as seen in TrueNAS UI:

So the question is, how do I configure Netdata to get child dataset sizes ?

[orsiris]

Looks like Netdata did something for zfs dataset sizes in add collector to monitor ZFS pools space usage by ilyam8 · Pull Request #17367 · netdata/netdata · GitHub which was released in netdata v1.45.2 or so.

TrueNAS Scale 24.04.2 still uses netdata v1.37.1.

@ TrueNAS team: Any chance 24.10 will see a more recent netdata version ? It looks like a real show stopper to me that the NAS can’t report zfs dataset spaces properly. Thanks for any insight.

[marcobe]

I also would love to see some historical space usage by dataset, that way you can anticipate dataset growth and see the potential weird stuff happening to your system.