How to configure Syslog with (m)TLS?

TrueNAS General
,
1

Hello,
I am trying to forward TrueNAS logs to a central syslog server.
System > Advanced Settings > Syslog:

Syslog Level: Debug
Syslog Server: syslog.mydomain:6514
Syslog Transport: TLS
Syslog TLS Certificate: -
Syslog TLS Certificate Authority: internal_ca

internal_ca is a private CA, and configured under Credentials > Certificate authorities. I confirmed it is valid by testing openssl client on nas.mydomain (TrueNAS SCALE):

openssl s_client \
  -connect syslog.mydomain:6514 \
  -CAfile /etc/certificates/CA/internal_ca.crt

Now when using tcpdump on syslog.mydomain:

tcpdump -vni eth0 port 6514

and triggering a log manually in nas.mydomain via

logger -t test -p 3 'SOME TEST ERROR'

, I don’t get any connection attempt by TrueNAS machine at all. Doesn’t matter how long I wait. It seems, Syslog client is not active with TLS? Invoking openssl s_server or netcat for a “dummy” TLS syslog target lead to same results (no output). When using TCP transport instead of TLS, I see traffic again.

Did I missing something? Unsure what’s going on here.
Thanks for any clarification!

TrueNAS SCALE ElectricEel-24.10.2

Some more infos about my real case (mTLS)

My server syslog.mydomain actually requires mTLS / client authentication.
Adding

-cert /etc/certificates/nas.mydomain.crt \  
-key /etc/certificates/nas.mydomain.key

to openssl connected successfully.
And it seems I can use “Syslog TLS Certificate” to hand over my client certificate to server for validation (but not sure about mTLS support with syslog).

Though underlying problem already seems to be a step earlier, by missing any TLS handshake and connection, hence not bothering for now.

2

Hey Marius,

I noticed a similar issue with TrueNAS SCALE 25.04.2.3.

As you rightly observed, when one launches tcpdump and you are running over TLS no traffic is observed. When I switched back to TCP, this was not the case anymore.

Digging a bit deeper I figured that there was an issue with the generation of the following files, when the TLS option is selected:

"/etc/syslog-ng/conf.d/tndestinations.conf" 
"/etc/syslog-ng/syslog-ng.conf"

More specifically, the function call getvirtconsole() in the first file was failing to resolve so we resolved it ourselves and substituted the relative line with the concerned function call to:

destination d_console_all { file("/dev/tty10"); };

And in the second file, the following part of the configuration related to syslog-ng’s remote logging was not being generated (bug in the function call that’s supposed to generate it). This led to no traffic making it out of TrueNAS:

Missing part of the configuration:

#remote logging
destination loghost { syslog("<domain>" port(<port>) ip-protocol(4) transport("tls") tls( ca-file("/etc/certificates/CA/<imported_ca_file_name>") key-file("/etc/certificates/<imported_key_file_name>") cert-file("/etc/certificates/<imported_cert_file_name>"))); };
log { source(tn_middleware_src); filter(f_tnremote); destination(loghost); };
log { source(tn_auditd_src); filter(f_tnremote); destination(loghost); };
log { source(s_src); filter(f_tnremote); destination(loghost); };

Adding these lines yourself to the aforementioned respective files should solve your problems in the immediate.

One thing to note is that mTLS was breaking the implementation even after fixing the conf files, but since it was not crucial for us, we disabled it for now and did not dig further into it.

A small caveat to this is that if you decide to switch from TLS back to TCP and back to TLS (through the UI), the broken config files will be regenerated, overwriting your changes. So I would recommend either making sure that once you fix the config files you never touch this setting again, or else write a systemd watcher that automatically patches your config files whenever they are overwritten.

I know this sounds a bit of nuisance but if you are patient enough and having Syslog over TLS is not very urgent, a patch for all of this is being proposed in the current release candidate for the 25.10.Beta.1 version. Upgrading to that version when it is out would solve all your problems without having to go through all the previous steps (NAS-137315 / 25.10-RC.1 / Refactor syslog configuration to support multiple servers (by william-gr) by bugclerk · Pull Request #12506 · truenas/webui · GitHub).

Hope that helps :smile:

1 Like
3

Wow, what a thorough answer - you solved all my questions in one go :smile: and also gave this useful hint about v25.10 + mTLS. Thank you very much franklyn!

I definitely will try out your proposed patches. Thanks again for mentioning mTLS quirks as well. This requirement probably is fix, so until 25.10 is out, I’d imagine to have a separate rsyslog instance with TLS only and TrueNAS as sole sender.

Great to know, that fixes are already on the way. Cheers!