Hello good people,
I'm trying to do the following thing:
- Pipe some of the logs, generated by OPNsense
- Remotely, to a Grafana Alloy instance (that then feeds it to Loki, etc.)
Overall, I've arrived at a pretty decent & workable solution, using TCP as transport.
And the final piece that I want to achieve is to actually transport those logs over TLS(4) - effectively doing mTLS between OPNsense (as syslog-provider) and Grafana Alloy (as syslog-receiver)
What I did was the following:
- I generated a CA, to sign the client and server certs.
- And then generated the client and server certs, where OPNsense is the client / provider, and Alloy is the server / receiver
Here is the relevant Grafana Alloy configuration:
otelcol.receiver.syslog "remote_syslog" {
protocol = "rfc5424"
allow_skip_pri_header = true
tcp {
listen_address = "0.0.0.0:8094"
tls {
cert_file = "/etc/alloy/certs/server.pem"
key_file = "/etc/alloy/certs/server.key"
client_ca_file = "/etc/alloy/certs/ca.pem"
min_version = "1.2"
max_version = "1.2"
}
}
output {
logs = [otelcol.exporter.syslog.syslog_out.input]
}
}
On the OPNsense side, I've done the following:
- Imported the CA in System -> Trust -> Authorities, by simply pasting the PEM string
- Respectively, imported the client certs - pem & key, in System -> Trust -> Certificates
- The CA cert shows "usages" 1 & the client certs correctly display the issuer
- And last but not least, I created a new remote logging target, over TLS(4), that's using the only client certificate from the point above
- And for a good measure, I've restarted (multiple times) the Loggign service.
The error that I'm getting on the Alloy side is the following:
remote error: tls: unknown certificate authority
Which kind-of leads me to think that OPNsense is not using the CA to validate the server cert when doing the handshake.
Debugging steps I've done:
- I've validated that the certs are good, using openssl s_client -connect {host}:{port} -CAfile ./ca.pem -cert ./client.pem -key ./client.key -state
- In fact, I've validated this from the OPNsense box itself (sshed, copied the certs in a temp folder and tasted
- I also manually sent a log line, via the openssl s_client invocation from above, using the same certs, and confirmed that it's working.
- And from digging around, it kind of looks like the imported CA is not being used by OPNsense at all
I would really appreciate your help & directions.
Please ask for whatever additional information & context you think you'll need.
p.s. the limit to TLS 1.2 was from another initial error that I was getting about a "tls: bad record MAC"
Okay, after some more digging, it looks like the CA is actually loaded & hashed properly in `/etc/ssl/certs`
And when I get the issuer hash, from both the client & server pems, I get the same correct hash:
$ openssl x509 -in server.pem -noout -issuer_hash
17d48176
$ openssl x509 -in client.pem -noout -issuer_hash
17d48176
$ ls /etc/ssl/certs/17d48176.0
/etc/ssl/certs/17d48176.0
So it "should" be working :D But it's not.
So I'd love to get some direction, if anyone can spot anything.