Topics

Servus,

ich habe ein schwerwiegendes Problem mit IPv6 & OPNsense's dhcp6c an meinem FTTH Anschluss (M-Net DualStack mit public IPv4). OPNsense 24.7.4-24.7.6 gleiches Problem.

Was läuft:

1. Eingerichtet wie empfohlen: igb0 -> vlan40 interface -> darauf das WAN mit IPv4 per PPPoE und IPv6 per DHCPv6 (Use IPv4 connectivity: CHECK, Configuration Mode: BASIC, Prefix delegation size:56, Request prefix only: CHECK, Send prefix hint: CHECK).
2. Beim Start erhalte ich eine 56er Prefix Delegation und die LAN interfaces erhalten alle eine öffentliche IPv6 per "track interface".
3. Für 2h alles prima. IPv6 läuft.

Dann kommt das Problem:

Nach 2h läuft die PD lease time ab (vltime=7200).   Alle bestehenden leases werden gekündigt.

DHCP6c versucht eine neue PD zu erhalten, bekommt aber keine Antwort. Das sieht dann so aus:

Code Select Expand

2024-10-08T11:58:08 Notice dhcp6c reset a timer on pppoe1, state=SOLICIT, timeo=1, retrans=2125
2024-10-08T11:58:08 Notice dhcp6c send solicit to ff02::1:2%pppoe1
2024-10-08T11:58:08 Notice dhcp6c set IA_PD
2024-10-08T11:58:08 Notice dhcp6c set IA_PD prefix
2024-10-08T11:58:08 Notice dhcp6c set option request (len 4)
2024-10-08T11:58:08 Notice dhcp6c set elapsed time (len 2)
2024-10-08T11:58:08 Notice dhcp6c set client ID (len 14)
2024-10-08T11:58:08 Notice dhcp6c Sending Solicit
2024-10-08T11:58:07 Notice dhcp6c reset a timer on pppoe1, state=SOLICIT, timeo=0, retrans=1052
2024-10-08T11:58:07 Notice dhcp6c send solicit to ff02::1:2%pppoe1
2024-10-08T11:58:07 Notice dhcp6c set IA_PD
2024-10-08T11:58:07 Notice dhcp6c set IA_PD prefix
2024-10-08T11:58:07 Notice dhcp6c set option request (len 4)
2024-10-08T11:58:07 Notice dhcp6c set elapsed time (len 2)
2024-10-08T11:58:07 Notice dhcp6c set client ID (len 14)
2024-10-08T11:58:07 Notice dhcp6c a new XID (6defee) is generated
2024-10-08T11:58:07 Notice dhcp6c Sending Solicit
2024-10-08T11:58:07 Notice dhcp6c reset a timer on pppoe1, state=INIT, timeo=0, retrans=144
2024-10-08T11:58:07 Notice dhcp6c remove an IA: PD-0
2024-10-08T11:58:07 Notice dhcp6c IA PD-0 is invalidated
2024-10-08T11:58:07 Notice dhcp6c remove an address 2001:a61:XXX:XX64:20c:29ff:fe62:7429/64 on vmx0
2024-10-08T11:58:07 Notice dhcp6c remove an address 2001:a61:XXX:XXfa:a236:9fff:fe1d:8b2/64 on vlan05
2024-10-08T11:58:07 Notice dhcp6c remove an address 2001:a61:XXX:XXde:a236:9fff:fe1d:8b2/64 on vlan02
2024-10-08T11:58:07 Notice dhcp6c remove an address 2001:a61:XXX:XXc8:a236:9fff:fe1d:8b2/64 on vlan01
2024-10-08T11:58:07 Notice dhcp6c remove an address 2001:a61:XXX:XX50:a236:9fff:fe1d:8b2/64 on vlan03
2024-10-08T11:58:07 Notice dhcp6c remove an address 2001:a61:XXX:XX29:a236:9fff:fe1d:8b2/64 on vlan02.41
2024-10-08T11:58:07 Notice dhcp6c remove an address 2001:a61:XXX:XX01:a236:9fff:fe1d:8b1/64 on igb1
2024-10-08T11:58:07 Notice dhcp6c remove a site prefix 2001:a61:XXX:XX00::/56
2024-10-08T11:58:07 Notice dhcp6c prefix timeout for 2001:a61:XXX:XX00::/56
Wie bekommt Ihr das denn zum laufen, speziell die M-Net Kollegen (@meyergru)?! Bitte helft :-)

Etwas ausführlicheres log (mit boot + lease) anbei:

Updated to 24.7 r.2 yesterday morning. All went well and working from home was fine.

This morning, DNS resolution was broken - Adguard Home was not resolving (I think - might have also been Unbound as I am running Adguard Home -> Unbound as resolver).

A reboot did NOT help. (Re-)Starting Adguard Home manually from the CLI errores out due to a service "yy" blocking the port (?).

Rolling back (via bectl) to 24.1.10_3 helped instantly - II am up an running without problems.

Apologies for lack of detail - cant dig deeper / replay the erroneous config right now, so just a word of caution and not a full bug report.

Doesn't seem to be the kernel issue, as I am running virtualized in ESXi.

(Edit: fixed title)

I noticed that the standard installation of the chrony plugin may not work as an NTS server. Can't tell for sure as I am not using a clean install. So putting this out here in case (1) it helps someone and/or (2) someone can help me improve / simplify my setup.

my issues:
chrony clients could not connect because of certificate errors.
1. the chrony plugin in server mode needs read access to the certificate and key. Without it, I got the following error:

Code Select Expand

# chronyd -Q -t 3 'server [myserver] iburst nts maxsamples 1'
...
... TLS handshake with [myserver] failed : The TLS connection was non-properly terminated.
...

2. the certificate needs to be the full chain. Without it, I got

Code Select Expand

# chronyd -Q -t 3 'server [myserver] iburst nts maxsamples 1'
...
... TLS handshake with [myserver] failed : Error in the certificate verification. The certificate is NOT trusted. The certificate issuer is unknown....

3. the standard OPNsense certificate & key at /var/etc/cert.pem / ..key.pem are read root only. I did not want to grant chrony access to it or include chrony in the "wheel" group.

What did I do?

1. set up a DNS alias for time.MYDOMAIN.TLD
2. set up a separate letsencrypt certificate for time.MYDOMAIN.TLD
3. set up an automation to have the certificate (full chain + key) be copied over to /var/lib/chrony and chmod those to chronyd and restart chrony:
a) set up the following backend action (https://docs.opnsense.org/development/backend/configd.html):

Code Select Expand

[NTS_renewal]
command:cp /var/etc/acme-client/home/time.XXXXX/fullchain.cer /var/lib/chrony/chrony.fullchain.cer && cp /var/etc/acme-client/home/time.XXXXX/time.XXXXX.key /var/lib/chrony/chrony.key && cp /var/etc/acme-client/home/time.XXXXX/time.XXXXX.cer /var/lib/chrony/chrony.cer && chown chronyd /var/lib/chrony/chrony.key /var/lib/chrony/chrony.cer /var/lib/chrony/chrony.fullchain.cer && /usr/local/etc/rc.d/chronyd stop && /usr/local/etc/rc.d/chronyd start
type:script
message: copy NTS certificates into chrony directory and make them readable
description: Renew NTS certificate & restart chrony (all in one go)
b) enable it:

Code Select Expand

service configd restart
c) setup an automation using this action in the ACME client for the NTS certificate.
4. Point chrony to his certificate and key:

Code Select Expand

ntsservercert /var/lib/chrony/chrony.fullchain.cer
ntsserverkey /var/lib/chrony/chrony.key
(not sure this will persist an update of the plugin though ...)

Now it seems to work. Hooray!  :D

resources:
- https://docs.opnsense.org/development/backend/configd.html
- https://unix.stackexchange.com/questions/712784/chrony-fails-to-sync-with-nts-enabled

pinging @mimugmail in case some of this could be intregrated into the plugin itself (as it purports to create a working NTS server which it _may_ not do).

Especially, it seems that the manual edits to the config file - specifically the "ntsservercert" / "ntsserverkey" part -- seem to be over-written ...

Since [some time], chrony hardly connects to any servers anymore:

Code Select Expand

MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* time.cloudflare.com           3   6    37    11    -36us[ +469us] +/-   17ms
^? sth1.nts.netnod.se            0   8     0     -     +0ns[   +0ns] +/-    0ns
^? sth2.nts.netnod.se            0   8     0     -     +0ns[   +0ns] +/-    0ns
^? ptbtime1.ptb.de               0   8     0     -     +0ns[   +0ns] +/-    0ns
^? ptbtime2.ptb.de               0   8     0     -     +0ns[   +0ns] +/-    0ns
^? ptbtime3.ptb.de               0   8     0     -     +0ns[   +0ns] +/-    0ns
^- nts1.time.nl                  2   6    37    10  -2907us[-2907us] +/-   39ms
^? nts.ntp.se                    0   8     0     -     +0ns[   +0ns] +/-    0ns
^? ntp2.glypnod.com              0   8     0     -     +0ns[   +0ns] +/-    0ns
^? ntpmon.dcs1.biz               0   8     0     -     +0ns[   +0ns] +/-    0ns
^? netmon2.dcs1.biz              0   8     0     -     +0ns[   +0ns] +/-    0ns
^? sth-ts.nts.netnod.se          0   8     0     -     +0ns[   +0ns] +/-    0ns


I can DNS-resolve all and ping most of the above domains

It seems to be an issue with file access rights? System log shows:

Code Select Expand

2021-12-02T14:15:43 chronyd[5971] Selected source 162.159.200.123 (time.cloudflare.com)
2021-12-02T14:15:41 chronyd[5971] Selected source 94.198.159.11 (nts1.time.nl)
2021-12-02T14:15:36 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:35 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:35 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:35 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:35 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:35 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:34 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:34 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:34 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:34 chronyd[5971] Could not set credentials : Error while reading file.
2021-12-02T14:15:34 chronyd[5971] Source 194.58.202.203 changed to 194.58.202.202 (nts.netnod.se)
2021-12-02T14:15:20 configctl[3020] event @ 1638450920.24 exec: system event config_changed
[ Chrony restart ]

It used to run fine; so I am suspecting the latest updates 21.7.5 or 21.7.6 -- or the recent update of my SSL certificate by the new ACME?

Why does OPNsense treat traffic on my WireGuard interface as "in" traffic on WAN?

Code Select Expand

WAN Oct 19 18:08:48 10.10.100.1:5353 224.0.0.251:5353 udp Block private networks from WAN

Code Select Expand

__timestamp__ Oct 19 18:08:48
action [block]
anchorname
datalen 69
dir [in]
dst 224.0.0.251
dstport 5353
ecn
id 46266
interface igb0
interface_name WAN
ipflags none
ipversion 4
label Block private networks from WAN
length 89
offset 0
protoname udp
protonum 17
reason match
rid 1eb94a38e58994641aff378c21d5984f
rulenr 69
src 10.10.100.1
srcport 5353
subrulenr
tos 0x0
ttl 1

This is the mDNS repeater listening on my LAN, VLAN and WireGuard interfaces which all form part of my LocalNet interface group and are generally considered as local interfaces. My OPNsense wireguard interface/endpoint has IP 10.10.100.1.

I would have expected the mDNS repeater repeating DNS traffic emitting from the LAN/VLAN interfaces to the WireGuard interface and vice versa, however this shows as WireGuard mDNS traffic flowing "in WAN" which seems wrong.

Why does this trigger a "WAN in" rule? Seems like a bug to me ....?

https://sslbl.abuse.ch/blacklist/ states:

In addition, SSLBL provides a more performant Suricata ruleset that uses tls_cert_fingerprint instead of tls.fingerprint. Please use either the ruleset above (sslblacklist.rules) OR sslblacklist_tls_cert.rules from below. Do not use both of them at the same time.

...

In order to use the more perfomant Suricata ruleset avilable for download below, you must run Suricata 4.1.0 or newer.

https://sslbl.abuse.ch/blacklist/sslblacklist_tls_cert.rules

Would it be possible to replace the current SSLBL ruleset with the more performant TLS ruleset? Or add it as a custom ruleset?

(the "user defined" tab only seems to permit adding custom rules, not custom rulesets)

[un]

While running Unbound as a local resolver, I had come across three issues:

1. Unbound fails to resolve certain domains. I have no DNSBL in Unbound. Unbound only delivers the CNAME, but no A record. When using Unbound as DoT forwarder, it resolves the hostname normally.

2. Even though I have unchecked "Flush DNS cache during reload", the statistics and cache are cleared with every Unbound reload. This should not happen.

3. Unbound failing to start, see https://github.com/opnsense/core/issues/5150 -  I don't experience this anymore, though.

Has anyone experienced similar issues or could suggest possible (configuration) errors?

[Connection and upload test succeeded.]

My automation for copying the Let's Encrypt certificate to my local ESXI server fails:

1. automation task set up and "test connection" claims everything is fine: Connection and upload test succeeded.

Naming "cert.pem" --> rui.crt
Naming "key.pem" --> rui.key
Naming "ca.pem" --> ca.pem
Naming "fullchain.pem" --> castore.pem

2. Manually SCP'ing the files to ESXi works. ESXi finds and uses the copied certificate:

Code Select Expand

  # scp ./cert.pem root@esxi.XXX.de:/etc/vmware/ssl/rui.crt
  # scp ...

(for the avoidance of doubt, no password needs to be entered when doing scp, i.e. key authentication has been set up and is working)

3. Running automation from the "certificate" submenu fails to copy the certificate and yields the following errors in the logfile:

Code Select Expand

opnsense[82577] /usr/local/opnsense/scripts/OPNsense/AcmeClient/upload_sftp.php: Command execution failed, exit code 2. Last input was: {"host":"esxi.XXX.de","host-key":"","port":"22","identity-type":"rsa","user":"root","remote-path":"/etc/vmware/ssl","chgrp":"","chmod":"","chmod-key":"","cert-name":"rui.crt","key-name":"rui.key","ca-name":"ca.pem","fullchain-name":"castore.pem","certificates":"xxx","automation-id":"xxx"}
opnsense[82577] /usr/local/opnsense/scripts/OPNsense/AcmeClient/upload_sftp.php: Failed on {"source":"/tmp/sftp-upload-3UmGMx","target":"ca.pem","mode":"0440","group":false,"delete_source":true}
opnsense[82577] /usr/local/opnsense/scripts/OPNsense/AcmeClient/upload_sftp.php: Failed uploading file '/tmp/sftp-upload-3UmGMx' to 'ca.pem' ; Cause: {"file_not_found":true,"error":"Couldn't fsetstat: No such file or directory"}
opnsense[69190] AcmeClient: running automation: uploadESXI
opnsense[69190] AcmeClient: running automation: restartGUI
opnsense[69190] AcmeClient: running automations for certificate: *.xxx.de


Strangely, ca.pem is the only file that is actually copied over to ESXi when running this ...!

Is this an error in my setup, or is upload_sftp.php broken?

Hi, strange behaviour that I have come across on 21.1.5, and not sure where to begin to troubleshoot:

I have one(!) laptop that loses connection to the OPNsense gateway intermittently since a few days:

Setup:

WAN -> Cable Modem -> OPNsense -> Unifi switch - LAN clients / wifi clients

Issue:

My work laptop suddenly loses connection to the gateway: It can still ping all LAN devices, as well as the switch, but no longer access the OPNsense IP (nor WAN of course). A reboot of OPNsense fixes this - for a few minutes and then connection _for this client_ fails again.

All other devices (LAN and wifi) continue to be able to access OPNsense and WAN. So the problem seems to be either in OPNsense or the switch.

How could I narrow down the issue? Is it ...
- related to RSTP issues?
- related to routing (I have the FRR routing plugin installed)?
- related to firewall / shaping (why do the other 192.168.1.0/24 clients have access though)?
- ...?
-