ElectricEel-24.10.1 Issue

neofusion · December 29, 2024, 5:30pm

The JSON is only used for attachments.

The problem here (as I understand it) seems to be that midclt is choking on this line:
midclt call -job mail.send '{"subject": "'"${subject}"'", "html": "'"${html_data}"'", "to": ["'"${email}"'"]}'

The ${html_data} variable contains the contents of logfile_tmp.txt which is a text file with mixed in html.

As a comparison, even a file containing only the following results in the dict expected-error:

Hello
World

So the question is if one can have those line breaks without them being interpreted by the shell, or possibly midclt.

ruffhi · December 29, 2024, 5:43pm

I can get a simple table to work …

#!/bin/bash
midclt call -job mail.send '{
    "subject": "System Status Report",
    "html": "<html><body><table><tr><th>heading1</th><th>heading2</th></tr><tr><td>item11</td><td>item12</td></tr><tr><td>item21</td><td>item22</td></tr></table></body></html>",
    "to": ["email@gmail.com"]
}'

It sends this table. The issue is that the html line has to be one long line.

This will not work (below)

#!/bin/bash
midclt call -job mail.send '{
    "subject": "System Status Report",
    "html": "<html><body>
<table>
<tr><th>heading1</th><th>heading2</th></tr>
<tr><td>item11</td><td>item12</td></tr>
<tr><td>item21</td><td>item22</td></tr>
</table>
</body></html>",
    "to": ["email@gmail.com"]
}'

Screenshot 2024-12-29 124131

It will not let me use …

midclt call -job mail.send '{
    "subject": "System Status Report",
    "html": "<html><body><pre style=\\"font-family: Courier New, monospace;\\"><table><tr><th>heading1</th><th>heading2</th></tr><tr><td>item11</td><td>item12</td></tr><tr><td>item21</td><td>item22</td></tr></table></pre></body></html>",
    "to": ["email@gmail.com"]
}'

oxyde · December 29, 2024, 5:53pm

the midclt argument is in json format, and he will throw parse error with line break:

edit:
despite, encoding the html for json, the call go smooth:

midclt call -job mail.send '{
    "subject": "TEST",
    "html": "<html><body>\r\n<table>\r\n<tr><th>heading1</th><th>heading2</th></tr>\r\n<tr><td>item11</td><td>item12</td></tr>\r\n<tr><td>item21</td><td>item22</td></tr>\r\n</table>\r\n</body></html>",
    "to": ["asdf@gmail.com"]
}'

oxyde · December 29, 2024, 6:12pm

seems i get it to work with the joe’s file:

#!/bin/bash
LOGFILE="/mnt/--/logfile_tmp.txt"
html_data=$(cat "$LOGFILE" | jq -Rs .)
midclt call -job mail.send "{
    \"subject\": \"TEST\",
    \"html\": $html_data,
    \"to\": [\"@@@@gmail.com\"]
}"

but as you see i have encoding everything (the sintax for using jq is provided by chat gpt but having ask specifically for this should be fine anyway xD)

ruffhi · December 29, 2024, 6:33pm

Nothing wrong with ChatGPT as long as you test what you get back. Do not trust it :).

Also,

this forum uses this code for spoilers

[details=description] hidden [/details]

neofusion · December 29, 2024, 6:36pm

This is brilliant, thank you for show us how it’s done.

joeschmuck · December 30, 2024, 3:43am

@oxyde
That code worked ! Well for sending a single HTML file without attachments.

I’m still trying to understand why it works. I understand the \" is a literal quotation, but why does it need to be on multiple lines? Hey, it works and I’m not fussing.

I had to make some changes to Multi-Report so it would use sendmail on earlier versions of TrueNAS and this new stuff for 24.10.1 and later. Now I’m trying to figure out why curl is giving me a headache. I will work on it some more tomorrow but I’m close. I can attach multiple files, it is just the html code that is not longer liked. Simple html is fine.

A good night’s sleep will reset me.

joeschmuck · December 30, 2024, 5:50pm

And stuck yet again. @oxyde Time to roll out that ChatGPT solution again

The last issue is sending the HTML file in the body, and having attachments at the same time. It appears curl really doesn’t like semicolons, it fails on the first one it finds. I have found a bit of data on this issue however no solutions, well that were obvious to me. I have to say it again, I’m no programmer.

Here is what I have right now:
This variable works fine if I remove the semicolon after the word “file” and I can send the html and attachments without issue. But add that semicolon and it fails.

html_data="Check the attached <b>Chart.html</b> file; to view the chart that should be in the body of this email."

My curl command is:

curl -k –location-trusted -v -u truenas_admin:santababy -F 'data={"method": "mail.send", "params": [{"subject": "'"${subject}"'", "to": ["'"${Email}"'"], "html": "'"${html_data}"'", "attachments": true}]}' -F "file=@/tmp/attachment.json" http://127.0.0.1/_upload/

And you will of course need the attachment.json file pointing to at least one file to attach.

In my searching I never saw some “special” formatting as @oxyde had for the mail.send call.

And I am virtually done with the script overall. I can send out an email with attachments provided the HTML has no semicolons. With this fact, I can attach the actual HTML file as an attachment and a person would need to open the ‘chart.html’ file to see the typical chart I want in the email body.

I’m on hold for now, been working different angles all morning with just failure after failure, and frustration. I now need to 3-D design and print a garage remote control holder for my wifes new car. There is no good location to place it in the car right now. The design part will be tricky but it will keep me busy for hours as I learn more about Fusion 360.

oxyde · December 30, 2024, 7:46pm

@joeschmuck, the curl fail to me too now.
I’m quite sure that this time the problem is the “semicolon” break the form data

The json encode is quite useless there (no reason to ask to chat gpt now

), i will continue make some test to find the right pattern (or is something documented somewhere?)

oxyde · December 30, 2024, 8:23pm

Here i am @joeschmuck , got it work, just replacing -F with --form-string before passing data
Never understimate 8 years stack overflow thread!

curl -k --location-trusted -v -u davide_admin:*** --form-string 'data={"method": "mail.send", "params": [{"subject": "'"${subject}"'", "to": ["'"${Email}"'"], "html": "'"${html_data}"'", "attachments": true}]}' -F "file=@/mnt/SSDPool/Script/InvioMail/attachment.json" http://127.0.0.1:880/_upload/

As you can see from my snippet… for me the right endpoint is http://127.0.0.1:880/_upload/ and not http://127.0.0.1/_upload/, because TN itself is exposed to another port

joeschmuck · December 31, 2024, 5:02am

I tried --form-string this morning but I placed it right after ‘curl’ so that explains why it didn’t work for me.

Thanks! I will test in the morning.

joeschmuck · December 31, 2024, 5:04am

Does this mean I need to configure it? I will look in the API tomorrow to see if it is listed so I can make it work without user interaction.

oxyde · December 31, 2024, 2:22pm

Afaik, assuming that using API mean to use a GET curl… Don’t you will have the exact same problem? But i think in the same way that there must be another trick to achieve this without let user bound the used port into the script

neofusion · December 31, 2024, 3:43pm

Unless one can conjure up the used webGUI port using midclt, then yes, I guess that’s going to be something you need to ask for during initial setup. Default to the standard port but ask if it should be changed.

If that is how it goes it’s not going to be a hands-off upgrade for some of the current users. They will need to supplement their current configuration if they use a non-standard port.

neofusion · December 31, 2024, 4:26pm

This gathers the whole configuration on a single line, pipes it to jq to make it more readable and then just spits out the line with the current http-port for the webGUI:
midclt call system.general.config | jq | grep ui_port | sed "s/[^0-9.]*//g"

Is that something you can build on?

I fully expect someone can make that grep/sed concoction prettier. Feel free.

joeschmuck · December 31, 2024, 5:30pm

OMG, That was so long ago but yes I remember it well. I will still fight for the HTML version if I can get there.

@oxyde Thanks for all the effort you have put into this. I of course ran into another problem. I feel it will be this way all the time. Maybe time for a different path.

With the change you presented, that small test file passes without issue. Now give this one a try, rename it to .html to view via web browser to see what it should look like. This is the same file which I can add as an attachment without issue, but who wants to open an attachment when it should be seen in the message body. I have done no processing on the attached file. html_data_full_raw.txt (42.8 KB)

To send a message without an attachment (using mail.send) I need to preprocess it with html_data=$(cat /tmp/logfile_tmp | jq -Rs .) and then it will send. The full set of commands are:

html_data=$(cat /tmp/logfile_tmp | jq -Rs .)
midclt call -job mail.send "{
    \"subject\": \"${subject}\",
    \"html\": $html_data,
    \"to\": [\"${Email}\"]
	}"

As for sending the attached file With Attachments, here is the curl command line, which of course fails with an error (the error changes depending on the preprocessing I do, or don’t do).

curl -k –location-trusted -v -i -u truenas_admin:puppy123 --form-string 'data={"method": "mail.send", "params": [{"subject": "'"${subject}"'", "to": ["'"${Email}"'"], "html": "'"${html_data}"'", "attachments": true}]}' -F "file=@/tmp/attachment.json" http://127.0.0.1/_upload/

But what is the preprocessing that is needed to make it work? I’m starting to dislike this project. Maybe tossing it into a Docker or Jail is what I need to do so I can control more on how things work/change. I will explore that in time. I do not desire to take a step backwards and would rather just present an attachment and a basic body message of “your system has not error messages” or “You got BIG troubles buddy!” or “How are you at replacing a drive?”
That is where I’m at for the moment as much as I would dislike it.

Absolutely! Thanks for saving me the time.

I can see it becoming something like this: local_address=$(midclt call system.general.config | jq | grep ui_port | sed "s/[^0-9.]*//g")

The the curl line changing to curl -k –location-trusted -v -i -u truenas_admin:puppy123 --form-string 'data={"method": "mail.send", "params": [{"subject": "'"${subject}"'", "to": ["'"${Email}"'"], "html": "'"${html_data}"'", "attachments": true}]}' -F "file=@/tmp/attachment.json" http://127.0.0.${local_address}/_upload/

Time to buy some dog food.

EDIT: As I’m thinking about it, most people send attachments once a week, or to me for troubleshooting. If we cannot find a way to make it work like it use to, all is not lost. With the attachment email, the html file would be an attachment. It isn’t a great solution but it would work until a better way appeared.

neofusion · December 31, 2024, 6:05pm

Just a clarification, that midclt call spits out the port number, not the IP.
The intent was to catch cases where someone, like @oxyde, has changed the default port 80 to something else, and give you the ability to automagically adjust the port to curl attachments to.

Cellobita · December 31, 2024, 6:35pm

We’re indebted to you, @joeschmuck. Shame on us if you ever find yourself having to buy a beer whenever you’re in a city where a user of your script lives…

When in Santos, Brazil, drinks and steaks are on me.

TimoJ · December 31, 2024, 7:00pm

I’m also trying to send a report html file but I get this error:
“WebSocket connection closed with code=1009, reason='Max message length is 64 kB”

Any ideas how to change this 64kB limit? Works fine with smaller than 64k files.

joeschmuck · December 31, 2024, 7:22pm

LOL, I knew that, just didn’t put it in the correct spot, ha ha. Should have been http://127.0.0.1:${port_address}/_upload/