TrueCommand Storage pool "" is predicted to reach 81% full in "" days

TrueCommand
1

Hi,
I’m wondering if anyone has any insight on an alert we’ve been getting from TrueCommand. It keeps sending every half hour even though “Storage Pool Expansion Soon (80%)” alert is set to disabled.
Does anyone know of any other setting that might be causing these email alerts to be sent out?

2

@UnknownDave I am trying to reproduce this now. In the meantime, here are a few steps you could perform:

  1. Ensure the container is on version 3.0.2
  2. Set middleware logs to trace level. /sys/log-level/set on the API page, input {"level": "trace"}
  3. Check output of the sqlite DB (/data/tc-conf/db). Ensure for internal-pool-80-full-estimate the isactive field is not enabled. You could try manually updating the column here, but if removed it will be added back as enabled on restart.
  4. Overwrite /etc/middleware/config.yaml and change sys.alert_email_threshold from 30m to 1d if you do not want frequent email alerts. Ideally copy this file from the container and then create a tied volume from a local copy to /etc/middleware/config.yaml with the field changed.
  5. Upload the SELECT * FROM alerts; output and the MW log to Jira.
1 Like
3

@iX_Zack_Welch Thanks for the reply.
I am able to verify the version is 3.0.2. I attempted to check output of sqlite DB but got an API error x.x.x.x/api/v1/data/tc-conf/db: 404 Not Found. However, I’m only a beginner in API so I’m not sure I understand the specifics of most of the steps unfortunately.

4

@UnknownDave Sorry, my comment was misleading. That path (/data/tc-conf/db) is a file path within the container, as is /etc/middleware/config.yaml. The database can be accessed through the sqlite client (sqlite3 /data/tc-conf/db) within the container, or if the container was created with a bind volume to the /data directory (-v local-dir:/data) then within that local directory. It is highly recommended to either have a bind or named volume to this directory in order to preserve data across reboots and upgrades. Additionally, copy /etc/middleware/config.yaml to a local directory and bind it through the same means (-v ./config.yaml:/etc/middleware/config.yaml) to preserve middleware config changes.

I’m continuing to try to reproduce this error and will provide a bug ticket if I am able to do so. In the mean time, try modifying the config file to at least cut down on the email chatter.

5

@iX_Zack_Welch I appreciate the clarifying response. Although I must admit I’m still not sure I understand the specific steps necessary.

By container, I’m going off the assumption it is in reference to the SAN itself as opposed to through TrueCommand? For the sake of brief clarification I just want to make sure we are on the same page that our environment is not running any containers such as the likes of Kubernetes etc. Likewise, we are not directly using sqlite on the TrueNAS SAN so I do not have any past experience with it.

What I can say is I did run these lines in TrueNAS CLI on the affected SAN using your provided instructions + what I could find on the web.

Sqlite3
.read /data/tc-conf/db

I then get “Error: cannot open “/data/tc-conf/db”

I also get no results from .tables. Whether that indicates anything or not I’m not sure.

6

Sorry, you’re a cloud customer then none of this is relevant. I’d encourage you to raise a support request in that case and we can lower the alert frequency now to avoid spam. A future update will also change the frequency and maybe make that directly configurable for cloud customers.

If you are hosting TrueCommand in a container locally then you’d have some more control over the config and the ability to run commands from within the container. I’d still recommend in the future, if filing a support request for a particular bug, to elevate the logs first through the API page, /sys/log-level/set, {"level": "trace"}, then download the MW log after reproduction and send it along with the request. In cases like this with the alert not being disabled, it’ll save you time getting it addressed. Support can download the logs themselves and elevate but they cannot access the cloud instance UI. I was not able to reproduce it myself, but I still encourage you to raise this particular issue with support.

1 Like
7

I actually have a case open currently but had made this post before realizing the proper channel for the issue. So, my apologies if that caused confusion here.

Thank you for the insight on everything, I will take it into account for future cases.

1 Like