Remote logging to Alloy, over TLS, fails with unknown certificate authority

[homelabber123]

Hello good people,

I'm trying to do the following thing:

• Pipe some of the logs, generated by OPNsense
• Remotely, to a Grafana Alloy instance (that then feeds it to Loki, etc.)

Overall, I've arrived at a pretty decent & workable solution, using TCP as transport.

And the final piece that I want to achieve is to actually transport those logs over TLS(4) - effectively doing mTLS between OPNsense (as syslog-provider) and Grafana Alloy (as syslog-receiver)

What I did was the following:

• I generated a CA, to sign the client and server certs.
• And then generated the client and server certs, where OPNsense is the client / provider, and Alloy is the server / receiver

Here is the relevant Grafana Alloy configuration:

Code Select Expand

otelcol.receiver.syslog "remote_syslog" {
protocol = "rfc5424"
allow_skip_pri_header = true

tcp {
listen_address = "0.0.0.0:8094"
tls {
cert_file      = "/etc/alloy/certs/server.pem"
key_file       = "/etc/alloy/certs/server.key"
client_ca_file = "/etc/alloy/certs/ca.pem"
min_version = "1.2"
max_version = "1.2"
}
}

output {
logs = [otelcol.exporter.syslog.syslog_out.input]
}
}

On the OPNsense side, I've done the following:

• Imported the CA in System -> Trust -> Authorities, by simply pasting the PEM string
• Respectively, imported the client certs - pem & key, in System -> Trust -> Certificates
• The CA cert shows "usages" 1 & the client certs correctly display the issuer
• And last but not least, I created a new remote logging target, over TLS(4), that's using the only client certificate from the point above
• And for a good measure, I've restarted (multiple times) the Loggign service.

The error that I'm getting on the Alloy side is the following:

Code Select Expand

remote error: tls: unknown certificate authority

Which kind-of leads me to think that OPNsense is not using the CA to validate the server cert when doing the handshake.

Debugging steps I've done:

• I've validated that the certs are good, using openssl s_client -connect {host}:{port} -CAfile ./ca.pem -cert ./client.pem -key ./client.key -state
• In fact, I've validated this from the OPNsense box itself (sshed, copied the certs in a temp folder and tasted
• I also manually sent a log line, via the openssl s_client invocation from above, using the same certs, and confirmed that it's working.
  
• And from digging around, it kind of looks like the imported CA is not being used by OPNsense at all

I would really appreciate your help & directions.

Please ask for whatever additional information & context you think you'll need.

p.s. the limit to TLS 1.2 was from another initial error that I was getting about a "tls: bad record MAC"