ElectricEel-24.10.1 Issue

Update, I got it to work:

[
 {
  "headers": [
   {
    "name": "Content-Transfer-Encoding",
    "value": "base64"
   },
   {
    "name": "Content-Type",
    "value": "application/octet-stream",
    "params": {
     "name": "testfile1.txt"
    }
   }
  ],
  "content": "<base64_encoded_string_from_file1>"
 },
 {
  "headers": [
   {
    "name": "Content-Transfer-Encoding",
    "value": "base64"
   },
   {
    "name": "Content-Type",
    "value": "application/octet-stream",
    "params": {
     "name": "testfile2.txt"
    }
   }
  ],
  "content": "<base64_encoded_string_from_file2>"
 }
]

I had the wrong delimitation, the above results in two files. Obviously you replace the <base64>-comments with the actual base64 strings.

Thanks @oxyde for your suggestion!

5 Likes

Nice job! Hope this can help @joeschmuck

1 Like

Thanks @neofusion @william @Captain_Morgan

That all works! I really appreciate your help. Now to make a few test changes to Multi-Report and see if I can make it send email as I desire it to.

9 Likes

There is still a problem if the user has enabled automatic redirect HTTP requests to HTTPS. The curl fails even if you add the -L flag to follow redirects. It results in:

Not relevant anymore
% curl -L -v -u santa:hohoho -F 'data={"method": "mail.send", "params": [{"subject": "testtest", "to": ["santa@icepole.net"], "html": "<b>This is just a merry test.</b>", "attachments": true}]}' -F "file=@/mnt/icepool/home/santa/attachmentcombined.json" http://127.0.0.1/_upload/
* Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'santa'
> POST /_upload/ HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic merrychristmashash
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 4037
> Content-Type: multipart/form-data; boundary=------------------------25fe85e7cca40086
>
* We are completely uploaded and fine
< HTTP/1.1 307 Temporary Redirect
< Server: nginx
< Date: Thu, 26 Dec 2024 21:31:14 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
* Please rewind output before next send
< Location: https://127.0.0.1:443/_upload/
* Keep sending data to get tossed away
<
* Ignoring the response-body
* Connection #0 to host 127.0.0.1 left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://127.0.0.1:443/_upload/'
* Trying 127.0.0.1:443...
* Connected to 127.0.0.1 (127.0.0.1) port 443 (#1)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=truenas.icepole.net
* start date: Nov 13 15:52:35 2024 GMT
* expire date: Feb 11 15:52:34 2025 GMT
* subjectAltName does not match 127.0.0.1
* SSL: no alternative certificate subject name matches target host name '127.0.0.1'
* Closing connection 1
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (60) SSL: no alternative certificate subject name matches target host name '127.0.0.1'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

If I add the -k flag to ignore the fact that the certificate of my domain name “truenas.icepole.net” doesn’t match “127.0.0.1” it still results in a 401/Unauthorized error.

% curl -L -k -v -u santa:hohoho -F 'data={"method": "mail.send", "params": [{"subject": "testtest", "to": ["santa@icepole.net"], "html": "<b>This is just a merry test.</b>", "attachments": true}]}' -F "file=@/mnt/icepool/home/santa/attachmentcombined.json" http://127.0.0.1/_upload/
* Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'santa'
> POST /_upload/ HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic merrychristmashash
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 4037
> Content-Type: multipart/form-data; boundary=------------------------73533b5cbebfbb8e
>
< HTTP/1.1 307 Temporary Redirect
< Server: nginx
< Date: Thu, 26 Dec 2024 21:33:07 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
< Location: https://127.0.0.1:443/_upload/
<
* Excess found: excess = 164 url = /_upload/ (zero-length body)
* we are done reading and this is set to close, stop send
* Closing connection 0
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://127.0.0.1:443/_upload/'
* Trying 127.0.0.1:443...
* Connected to 127.0.0.1 (127.0.0.1) port 443 (#1)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=truenas.icepole.net
* start date: Nov 13 15:52:35 2024 GMT
* expire date: Feb 11 15:52:34 2025 GMT
* issuer: C=US; O=Let's Encrypt; CN=E6
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/2
* h2h3 [:method: POST]
* h2h3 [:path: /_upload/]
* h2h3 [:scheme: https]
* h2h3 [:authority: 127.0.0.1]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* h2h3 [content-length: 4037]
* h2h3 [content-type: multipart/form-data; boundary=------------------------73533b5cbebfbb8e]
* Using Stream ID: 1 (easy handle 0x556b6dd25ce0)
> POST /_upload/ HTTP/2
> Host: 127.0.0.1
> user-agent: curl/7.88.1
> accept: */*
> content-length: 4037
> content-type: multipart/form-data; boundary=------------------------73533b5cbebfbb8e
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
< HTTP/2 401
< server: nginx
< date: Thu, 26 Dec 2024 21:33:07 GMT
< content-type: text/plain; charset=utf-8
< content-length: 17
< strict-transport-security: max-age=63072000; includeSubDomains; preload
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< permissions-policy: geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()
< referrer-policy: strict-origin
< x-frame-options: SAMEORIGIN
<
* Connection #1 to host 127.0.0.1 left intact
401: Unauthorized

Domain names, file paths, usernames and passwords have been sanitised.
Two elves were harmed in the making of this post, but no one will miss them.

Edit: Fix/workaround in the following post.

I got it to work even when using redirects by doing the following:
% curl -k –location-trusted -v -u santa:hohoho -F ‘data={“method”: “mail.send”, “params”: [{“subject”: “testtest”, “to”: [“santa@icepole.net”], “html”: “<b>This is just a merry test.</b>”, “attachments”: true}]}’ -F “file=@/mnt/icepool/home/santa/attachmentcombined.json” http://127.0.0.1/_upload/

-k sets it to ignore the fact that “127.0.0.1” is not in the certificate (ignoring certificate verification is associated with risks, ymmv).
--location-trusted instructs curl to follow redirects and to reuse the login credentials even on the new https domain.

7 Likes

So this could be a problem, correct? In other words, I should account for this.
I can create a new configuration variable that someone can edit in the multi_report_config.txt file. What I would not want to happen is someone needs to manually edit the script at line 10987 for example. I took a needed break yesterday, now I hope to get started on the script again.

It’ll be a problem if someone, like me, is using the 80->443 web GUI redirect, yes.
If you aren’t you won’t notice it.

As far as I can tell the flags do nothing if the redirect isn’t enabled, so my take is that it doesn’t need to be a toggle, the flags can be enabled for everyone.

Best of luck!

1 Like

That is very good news. I will take that advice and use it.

I think I just finished the drive_selftest.sh script. I sent it to one person who reported a very specific problem. I have pushed it to the github repository. I plan to release it 1 January 2025, but if it works as I expect it will, then it is ready to go now.

3 Likes

I have something working, at least it shows promise. I now have the all the commands and formats written down, however I have not tested them on 24.10.1 yet, I’ve been using 24.10.0.2 for testing because sendmail still works so I can run the script at night. Tomorrow after a good nights sleep, I will roll my system to 24.10.1 and test. I’m hopeful it will work.

Once all that is done, I will post my findings.

Then I need to write a routine so I can utilize the majority of my code which generates the HTML data and to generate the attachment.json file. Unfortunately, while is sounds easy, it never is in practice.

6 Likes

SOLUTION IS HERE:

First I really need to thank everyone who helped out here, I would have been lost for weeks without that help. We have a very good community.

I am presenting two functions, one to send the email, one to generate the attachments.

The first is the send_email function. Consider the variables as global, in my script I really do not have this as a function, it is for simplicity.

Variables are: $subject, $html_data, and $email

If the file /tmp/attachment.json does not exists then we send email without an attachment.
If the file /tmp/attachment.json does exist, then we use curl to send the attachment(s).

send_email() {
if ! test -e "/tmp/attachment.json"; then
	midclt call -job mail.send '{"subject": "'"${subject}"'", "html": "'"${html_data}"'", "to": ["'"${email}"'"]}'
else
    curl -k –location-trusted -v -u truenas_admin:santa -F 'data={"method": "mail.send", "params": [{"subject": "'"${subject}"'", "to": ["'"${email}"'"], "html": "'"${html_data}"'", "attachments": true}]}' -F "file=@/tmp/attachment.json" http://127.0.0.1/_upload/
    rm /tmp/attachment.json    # Cleanup
fi
}

The second is create_attachment function and it will create the attachment.json file.
We call the function like this:

create_attachment_file /mnt/farm/scripts/report_body.txt report.txt

Where the first parameter is the full path and name of the file to be sent.
Where the second parameter is the name of the file as presented in the email.
In the example above we are sending the file /mnt/farm/scripts/report_body_.txt and in the email it will be named report.txt.

If you have multiple files to send, run the create_attachment_file function for each file you want attached, then once you have all your attachments in place, run the send_email function.

create_attachment_file() {

	encoded_content=$(base64 --wrap=0 $1)

	if test -e "/tmp/attachment.json"; then		# If the file exists then we need to add on to the end, but need to delete the last ']'.
	
		# I need to read the file back, remove the last 2 lines, add line ' },', then add new header.
		updating_attachment=$(head -n -2 /tmp/attachment.json)  # This removes the last lines.
		(
		echo "${updating_attachment}"
		echo ' },'
		echo ' {'
		echo '  "headers": ['
		echo '   {'
		echo '    "name": "Content-Transfer-Encoding",'
		echo '    "value": "base64"'
		echo '   },'
		echo '   {'
		echo '    "name": "Content-Type",'
		echo '    "value": "application/octet-stream",'
		echo '    "params": {'
		echo '     "name": "'"$2"'"'
		echo '    }'
		echo '   }'
		echo '  ],'
		echo '  "content": "'"$encoded_content"'"'
		echo ' }'
		echo ']'
		) > /tmp/attachment.json
	else	# Else we generate the initial file.
		(
		echo '['
		echo ' {'
		echo '  "headers": ['
		echo '   {'
		echo '    "name": "Content-Transfer-Encoding",'
		echo '    "value": "base64"'
		echo '   },'
		echo '   {'
		echo '    "name": "Content-Type",'
		echo '    "value": "application/octet-stream",'
		echo '    "params": {'
		echo '     "name": "'"$2"'"'
		echo '    }'
		echo '   }'
		echo '  ],'
		echo '  "content": "'"$encoded_content"'"'
		echo ' }'
		echo ']'
		) > /tmp/attachment.json
	fi
	}

Another piece of advice, while this does run in TrueNAS 24.10.0.2 and 24.10.1, it will not run in CORE so you will need to resort back to sendmail. I test the version of TrueNAS, anything less-than 24.x will use sendmail, anything equal or greater will use the new mail sending commands. I need to figure out how to implement this without changing a lot of HTML code. I think it will be a lot of if statements.

I would like to say that I have not incorporated the $html_data into my script however the static values I provided worked fine.

Hope this helps everyone, it has certainly helped me.

11 Likes

Very nice work Joe. I like the way you approached this, chunked it down and then addressed each element.

I have a question … both sides of the if statement contain lots of identical code.

Can we reduce the duplication by …

create_attachment_file() {
	encoded_content=$(base64 --wrap=0 $1)

	# If the file exists then we need to ...
	# a) delete the last ']'
	# b) append new attachment
	if test -e "/tmp/attachment.json"; then
		# I need to read the file back
		# remove the last 2 lines
		# add line ' },'
		updating_attachment=$(head -n -2 /tmp/attachment.json)  # This removes the last lines.

		(
		echo "${updating_attachment}"
		echo ' },'
		) > /tmp/attachment.json

	else	# Else we have a brand new json file and just need the initial '['
		(
		echo '['
		) > /tmp/attachment.json
	fi
	(

	# add header 'new' information
	echo ' {'
	echo '  "headers": ['
	echo '   {'
	echo '    "name": "Content-Transfer-Encoding",'
	echo '    "value": "base64"'
	echo '   },'
	echo '   {'
	echo '    "name": "Content-Type",'
	echo '    "value": "application/octet-stream",'
	echo '    "params": {'
	echo '     "name": "'"$2"'"'
	echo '    }'
	echo '   }'
	echo '  ],'
	echo '  "content": "'"$encoded_content"'"'
	echo ' }'
	echo ']'
	) >> /tmp/attachment.json

	}

Looks like you can do that. I like the smaller code, maybe I will use that. But then how will I expand my script to beyond 20,000 lines :rofl:.

3 Likes

I have a related issue to which I hope someone can just slap me around when you tell me it was an oversight.

I’ve been battling with getting the HTML to work with the new mail API with no real luck, well bad luck I guess.

It appears that if you are doing very basic text only, you might be safe. But if you toss in a table, it falls on it’s face. I really hope this is a me thing and not a TrueNAS thing.

I am providing a simple html encoded file in which you can open in your web browser without issue. Can anyone figure out how I can use this file with the new email API?

Within the script I had to remove all of the LF or \n characters as the API does not like those. I did not alter the original file.
html_data=$(cat/tmp/logfile_tmp.txt | tr -d '\n')
Now the file has no more line feeds, I’m not sure what that may have done to the overall file.
Next that goes into the script and I get the old [EINVAL] mail_message: A dict was expected error message. So I pick the first 20 lines or so and extract those and try to pass just those through. Failure. I start cutting off the trailing edge of this text, piece by piece. I end up at eliminating all the way to where the first table is defined, then no more error.

I am hoping I am overlooking something. Attached is the original text file. Rename to .html if desired.

I appreciate the help.

-Mark
logfile_tmp.txt (35.0 KB)

It might be that the html_data variable wraps. I can get a very simple table to go if I put it all on one line with no style commands.

happily sent this …


subject="test subject"
html_data='<b>bold1</b><br><br><b>bold2</b><br><br><table><tr><th>H1</th></tr><tr><th>H3</th></tr><tr><th>H2</th></tr><tr><td>1</td></tr><tr><td>2</td></tr><tr><td>3</td></tr></table>'

… but it didn’t come out as a table.

H1
H2
H3
1
2
3

This is going to be an issue. Putting on my thinking cap, one way around it for now is to just create the HTML file, then attach it as a file. The sad part is the email body will just say “Look at the attached chart.html file for a graphical result.” or something like that. I really do not want to go that route but if that gets the user the data, it is better than nothing.

Topic Shift: Why not bring back sendmail, looks like the vulnerability was fixed.

sendmail through 8.17.2 allows SMTP smuggling in certain configurations. Remote attackers can use a published exploitation technique to inject e-mail messages with a spoofed MAIL FROM address, allowing bypass of an SPF protection mechanism. This occurs because sendmail supports . but some other popular e-mail servers do not. This is resolved in 8.18 and later versions with ‘o’ in srv_features.

I’ve got to say, this vulnerability was very interesting to watch the video about.

So is the problem fixed in sendmail 8.18 and we can go back to sendmail ? Or am I missing some other vulnerability?

My research shows that sendmail 8.18.1-6 was ported into Debian on 25 Oct 2024 (pretty damn slow of them considering this vulnerability came out in Dec 2023) as stable. My research may be flawed so maybe there is some other reason sendmail is not being used. This vulnerability affected practically everyone world wide, it was not just sendmail, yes exim had it too according to CVE-2023-51766.

2 Likes

Maybe that is why they decided to remove it rather than wait for a new version to make it into debian? Might be interesting to see whether exim was fixed earlier?

Maybe they fear the potential of another vulnerabitlity the way it is set up or just a repeat of the slow reaction time.

Protection of the data of their enterprise security will be one of their highest priorities. If you do not have it, any vulnerability does not concern you. :confused:

I agree, vulnerabilities need to be addressed. But this was a massive vulnerability which affected practically everyone. It has been reported as fixed. I’m certain more vulnerabilities will pop up over time but in what product? Could be a Debian exploit next. If it took Debian this long to implement the fix, what is to say any other exploit will be fixed faster?

I would like to go back to sendmail if possible, or if we have the correct way to implement the current mail system, that would be fine too, even if it does require a bit of work.

Time to work on a temporary solution.

1 Like

Joe … remember that work you did to move to a html table … instead of …

########## ZPool status report summary for all pools on server NASPROD ##########

+--------------+--------+------+------+------+----+----+--------+------+-----+
|Pool Name     | Status |  Read| Write| Cksum|Used|Frag|   Scrub| Scrub| Last|
|              |        |Errors|Errors|Errors|    |    |Repaired|Errors|Scrub|
|              |        |      |      |      |    |    |   Bytes|      |  Age|
+--------------+--------+------+------+------+----+----+--------+------+-----+
|BankVault     |ONLINE  |     0|     0|     0| 24%|  0%|      0B|     0|   25|
|DuffleBag     |ONLINE  |     0|     0|     0| 23%|  2%|      0B|     0|   28|
|freenas-boot  |ONLINE  |     0|     0|     0| 26%|  1%|      0B|     0|    1|
+--------------+--------+------+------+------+----+----+--------+------+-----+

I just tried a long block of text with line continuations (back or forward slash - I don’t know which is which ) and ‘< br >’ for a line feed / carriage return and it worked nicely.

1 Like

How do I get html code to accept a non proportional font?

This code isn’t working for me.

html_data="<p style=""font-family: 'Courier New', monospace;"">\
<br>\
+--------------+--------+------+------+------+----+----+--------+------+-----+<br>\
|Pool Name     | Status |  Read| Write| Cksum|Used|Frag|   Scrub| Scrub| Last|<br>\
|              |        |Errors|Errors|Errors|    |    |Repaired|Errors|Scrub|<br>\
|              |        |      |      |      |    |    |   Bytes|      |  Age|<br>\
+--------------+--------+------+------+------+----+----+--------+------+-----+<br>\
|BankVault     |ONLINE  |     0|     0|     0| 24%|  0%|      0B|     0|   25|<br>\
|DuffleBag     |ONLINE  |     0|     0|     0| 23%|  2%|      0B|     0|   28|<br>\
|freenas-boot  |ONLINE  |     0|     0|     0| 26%|  1%|      0B|     0|    1|<br>\
+--------------+--------+------+------+------+----+----+--------+------+-----+<br>"

As far i understand, you are putting the content of your file into the html_data variable, right? And the variable Is used to build a JSON…
Watching your file you start use style tag from the table, so you are using "… maybe this Is what break the JSON? I think just JSON encoding the content should do the job, i found some old stack overflow post that point to using jq… Hope this help