Messages

Dear all,

maybe the following issue has been reported previously, but up to now I was not able to find it. Since several weeks, a strange behavior started to happen on my site-to-site High-Availability openVPN connection. I was first looking at some changes on openVPN, but I then noted that sometimes, both master and backup on client side are telling that they are connected to the master server... which is strange, since I thought that when HA is configured, only 1 VPN connection should be active at a time.

It seems that some weeks ago (with an update of OPNsense probably deployed in March 2025), every time the master is getting synchronized with the backup (I use a cron script for that), the openVPN connection on the backup client is not "restarted", which means that it tries to connect, not taking into account the CARP status. This behavior is strange to me, if the initial statements remain true: the backup client takes over the VPN connection only if the master client is in trouble.

Can somebody confirm if this behavior is wanted? I am on Firmware 25.1.5_5, but the behavior existed on previous versions of the 25.1.5 release. Maybe even before, but this is speculation. I thought all the time that the issue was related to ma change or missconfiguration of openVPN, but it seems now related to the management of the connections only after the synchronization in the HA case.

Interesting detail: If I disconnect the WAN on the master client, wait some short time, then connect it again, the master client takes over the VPN connection and the backup releases its connection. But as soon as I synchronize the states again, both are trying to connect to the server, thus interrupting each other, which makes connection conflicts and reset the connection every 1-2 minutes.

Best regards,
Vincent

Thank you franco.

I can give the precision, since I have 2 identical devices, and I have only opened and upgraded one, to ensure that it performs well, before I upgrade the second one.

The temperature on the non-upgraded (i.e., never opened) DEC690 is in the range of 68-70°C. The temperature of the upgraded DEC690 unit was (before I have opened it) between 70-72°C. After having opened, upgraded, and closed it again, it now has 75°C if I use a mechanical tool to press the heat-sing onto the CPU, and it displays 99.9°C if I do not press it... This is what I mean with "Not acceptable".

My question would be: is there somewhere a description how to reclose the enclosure and obtain a good cooling performance? How to proceed? Which kind of thermal adhesive was originally used?

Best regards,
Vincent

Hello,

I have upgraded my DEC690 from 8GB DDR3L to 16GB DDR3L with ECC, and added a second SSD of 1TB to the original 256GB SSD. After reassembling, I encounter cooling issues with the CPU. The design of the thermal contact between the CPU and the heat-sink does not provide a natural mechanical contact pressure, so that thermal paste or even thermal adhesive pads are not providing the required performance. Even with thermal glue, the results are not acceptable.

Does anyone know which material Deciso has used to provide an acceptable thermal conductivity between the CPU and the heat-sink? To my opinion, the thermal design is quite sub-optimal, because there is no controlled pressure applied in this zone (i.e., no screws or anything playing the same role).

Bonus question: if somebody knows how to check that ECC is really used, I would also be very interested. I tried with dmdidecode and with memtest86+, but no information related to memory (first) or ECC (second) could be displayed.

Best regards,
Vincent

Hello,

I have experienced the following strange and unwanted behavior:
I have 2 OPNsense Deciso DEC690 routers configured with CARP. Everything is working well (currently I am on version 22.1). However, I discovered, after the failure of the backup router (thermal issues), that devices which were not statically mapped with an IPv4 IP/MAC couple, have systematically failed to obtain an IPv4 since the backup router has failed. This is working properly when the backup router is on again.

The workaround I found is to delete the IP address filled in "Failover peer IP" under Services -> DHCPv4 -> [LAN1], since it corresponds to the backup router that has failed.

I am wondering if I am the only one experiencing this issue? Maybe I have misconfigured something and therefore obtained this strange behavior? Note that it only occurs with the devices that do not have a couple of IP/MAC mapped.

I do not use any MAC filtering or blacklisting/whitelisting.

Best regards,
Vincent

I found the origin of my problem, why the BACKUP was so slow to check for firmware updates in the GUI (and it also solved all the DNS stuff I explained in my previous post).

The issue is related to the manual Outbound NAT rules I have added to support the VIP on the WAN interface.

The configuration that makes problems is (same on MASTER and BACKUP):
Interface: ETH0_WAN
TCP/IP Version: IPv4
Source Address: any
Translation / target: 192.168.0.250 (Shared WAN CARP Virtual IP)

When I change the configuration to the following:
Interface: ETH0_WAN
TCP/IP Version: IPv4
Source Address: ETH1_LAN net
Translation / target: 192.168.0.250 (Shared WAN CARP Virtual IP)

then I am getting DNS answers on both MASTER and BACKUP.

This means, that I have to generate 1 Outbound NAT rule per LAN or VLAN, and that having only one single rule to manage all the LAN and VLANs does not work properly. Most probably I am missing something, but I am pretty sure that the original automatic rule was a single rule regrouping all the LAN and VLAN interfaces.

I would be curious to hear the explanation.
Thank you.

Hello everyone,

I am facing a very similar issue in an HA environment.

The MASTER Firewall is always working well: NTP, DNS over TLS, Update, Plugin installation, etc...

However, the BACKUP firewall is extremely slow to resolve DNS and if I try to update the firmware, it took around 1h30 from 21.1.6 to 21.1.7!!! The MASTER took less than 10 minutes, including reboot time.

I am using 2 identical DEC690 appliances. I have only DNS over TLS configured (forcing all the traffic through the DNS over TLS, i.e., I am rerouting the TCP/UDP traffic to port 53 of the appliances to the firewall itself to answer these requests. I have no DNS entry under System --> Settings --> General --> DNS servers

I have found out that in some situations (not related to HA and CARP), the firewall can be brought in situations in which I cannot log in again, since the time is wrong (reset to April 2017, for example when the appliance was turned off). It seems to be that NTP is trying to use the DNS servers in these general settings, and not the ones configured in Unbound DNS (i.e., DoT).

Does anybody have an idea, wehre I could search to solve the issue of the extremely slow BACKUP appliance?

Nota: if I disconnect the MASTER and the BACKUP becomes MASTER itself, the problem is transferred. The problem is always on the BACKUP, never on the MASTER (no physical relation).

One more thing: I am on a single WAN (but using CARP VIP on WAN and on LAN side).
And the Network Time always shows "No active peers available " on the BACKUP. Also the proofpoint Telemetry status always shows a cross (i.e., not connected) on the BACKUP.

And if I do a DNS Lookup on the BACKUP (after having disabled the blocking DNS firewall rules and inserted 1.1.1.1 in the general DNS servers (i.e., not Unbound DNS)), I get the following:
Server    Query time
127.0.0.1    475 msec
1.1.1.1    No response

Same on the MASTER:
Server    Query time
127.0.0.1    181 msec
1.1.1.1    51 msec

Thank you.

Some weeks ago, I had the issue that NTP could not access the time servers anymore... because Unbound DNS was not able to resolve the server names... bacause an error in the configuration, that appeared after a reboot. The consequence was, that the date and time was back in 2017 and TOTP was hindering any login.

In such a case, the console saved me (which was not protected with a password).

In consequence, I would prefer to have the root not dependent on the TOTP, but actually, I do not have any ideal solution.

Thanks for sharing yours, Greelan.

Is there any way to let root active, but force the following for SSH login:
- user login over SSH with TOTP
- after successfull user login, switch to root by using standard password (without TOTP)?

Further, could somebody confirm that if no seeds is configured for "root" and only the TOTP server is allowed under System --> Settings --> Administration --> Authentication --> Server ; then if using "sudo", the root password must contain also the TOTP code... but if there is no seeds configured for root, the password will never be valid?

Actually, I would like to reach following situation: user logging with TOTP, switch to root with standard password, after user login was validated successfully. (for SSH).

For WebGUI, I would like that root is not enabled at all.

Maybe there is a more elegant way: in this case, I would be very interested in listening to other strategies.

Best regards
Vincent

I was able to figure out where the problem was.

First of all, as expected, there is no need to enter anything in the "Show advanced option" under "Unbound DNS" --> "General". You can leave this totally empty: it will not hinder you to use Tor in Transparent Mode.

I had an error in the Port Forwarding when redirecting the traffic to the port 9040: I was only redirecting the traffic that did not contain any non-routable Tor IPv4 address as destination. The solution was to redirect all TCP traffic, without filtering out any IP networks.

Now that it is working, I have to figure out more exactly which traffic should be redirected and which one not.

[but]

I am most probably doing something wrong, and it makes me crazy. Here is my config:

OPNsense 21.1.6-amd64
FreeBSD 12.1-RELEASE-p16-HBSD
OpenSSL 1.1.1k 25 Mar 2021

For all the tests, I am using Firefox (lastest version).

Let's consider the following:
- LAN1 (not a VLAN): Tor is configured to listen on this interface, using SOCKS5: when I configure Firefox with SOCKS5, it works perfectly. The check.torproject.org site is showing that Tor is configured successfully, and *.onion domains are resolved successfully.
- VLAN88 (on same physical interface as LAN1): Tor is configured as Transparent Proxy. The UDP trafic on port 53 is redirected to port 9053. All TCP traffic on all ports is redirected to the port transparent port 9040. The check.torproject.org site is showing that Tor is configured successfully, but *.onion domains are not resolved.

I am using Unbound as DNS resolver/forwarder. I have tried to add none, parts or all of the following commands that I found to make the *.onion domain name resolution work, but without success:

domain-insecure: "onion"
private-domain: "onion"
do-not-query-localhost: no
local-zone: "onion." nodefault

forward-zone:
    name: "onion"
    forward-addr: 127.0.0.1@9053
    forward-first: no

In my understanding, the transparent DNS port 9053 should be doing the *.onion resolution itself (i.e., without any interaction with Unbound), so I should actually not need to add anything in Unbound DNS! I am using exclusively DoT in Unbound, and my DNS servers are only added under "Miscellaneous". Maybe I am wrong here in my understanding, but all the rest was apparently running fine in the last months. The unbound.conf file looks fine.

Can somebody help me to figure out where is my understanding problem?

I seems that there are not many ideas, how to solve my issues...

I made further tries and I am having more and more doubts that the problem has a clean solution.

To summarize: I have 2 OPNsense firewalls (DC690-1 and D690-2). The issue in between these 2 OPNsense firewalls and my Fritzbox (the Fritzbox connects to the internet):

DC690 WAN <--> FritzBox LAN <--> Fritzbox Cable Internet

On the DC690 WAN I have the following:
- DC690-1 WAN IP: 178.18.0.101 (assigned by DHCP by the Fritzbox)
- DC690-2 WAN IP: 178.18.0.102 (assigned by DHCP by the Fritzbox)
- CARP Virtual IP on WAN interface: 178.18.0.254

To access the devices in my LAN fron the outside (Internet), I have configured the option "Exposed Host" on my Fritzbox. Without using High Availability (HA), I set the exposed host to be 178.18.0.101 and everything rocks.

However, when I want HA, I have to configure the Fritzbox so that 178.18.0.254 is an Exposed Host. This makes problems. First, I must set a fixed IP that will be applied by the Fritzbox to the OPNsense firewalls... but since it is a Virtual IP that is actually already defined in the OPNsense firewalls, I could imagine that it makes problems. Sometime the Fritzbox assigns 178.18.0.254 to the DC690-1 WAN, and then I do not have access to the internet anymore.

Could somebody help and give me an idea how to configure this specific part of my network?

Many thanks in advance.
Best regards,
Vincent

Hi,

I have 2 issues with my 2 OPNsense routers.

My configuration: OPNsense 21.1.5-amd64 ; FreeBSD 12.1-RELEASE-p16-HBSD ; OpenSSL 1.1.1k 25 Mar 2021
Cable Internet <--> FritzBox 6590 <--> 2x OPNsense DC690 with High Availability <--> LAN Switches and Devices

My cable ISP does not provide IPv6, therefore I am using HE out of the Fritzbox (i.e., the Fritzbox is establishing the IPv4 native connection, and then the HE IPv6 connection).

The DC690 are getting their IP addresses via DHCP. Each DC690 gets an IPv4 via DHCP and an IPv6 via DHCPv6 from the Fritzbox. These IP addresses (IPv4 and IPv6) are defined as static, so they are always the same after reboot or reconnection.

1. The first issue is about the Virtual IP. The IP of the DC690-1 is 172.18.0.101. The IP of the DC690-2 is 172.18.0.102. The Virtual IP (VIP) corresponding to both DC690 IPs in WAN side is 172.18.0.100. Since the 101 and the 102 are attributed by DHCP by the Fritbox and are always the same (because linked to their MAC address), how should I define properly the VIP, to ensure that it will always be 100?
I have tried to set it up in the Fritzbox in the same way as I do it for the two DC690 (i.e., DHCP uses the virtual MAC address and assigns always the same IP which is 100), but if it works quite fine when both DC690 are physically connected to the Fritzbox, it does not work any more when I remove one of the DC690, since the Fritzbox seems to attribute the 100 IP address to the DHCP WAN interface, and I loose then the internet access (not sure why)! Further, I want the 100 address to be declared as "exposed host" in the Fritzbox, so that all the traffic is routed to this VIP. How can I reach this properly? (I do not want to disable DHCP on the Fritzbox, since it would bring other issues).
Any advice how to do this properly? Maybe there is a conflict between the VIP 100 setup in the OPNsense router and the IP 100 assigned by the Fritzbox to the 00:00:05:00:01:01 virtual MAC address?


2. The second issue is quite strange and existed before I have setup the HA configuration and had any VIP (so please consider here that HA and VIP are disabled or not existing and only one DC690 is in the configuration). When I reboot the OPNsense router, in most of the cases, I am unable to loggin over the GUI anymore after the reboot, if I do not disconnect the WAN side of the router during the reboot (important: only during the reboot, since afterwards it seems to have no more effect). Maybe there is something related to DHCPv4 and/or DHCPv6 or Router Advertisement (RA), all provided by the output of the Fritzbox connected to the OPNsense DEC690? How could I investigate this issue and find out to what it is related?
Strangely, if I forget to disconnect the WAN during the reboot, I still have access to the router via SSH, internet is running, the devices on the LAN all get their addresses and gateway address. Any idea where to look to locate the problem?

Many thanks in advance.

Best regards,
Vincent

Thank you marjohn56, for your detailed explanation. Since it is a business laptop on which I do only have very limited rights (no admin), the solution was to add a managed switch and separate it there, properly. Now it is clean, and only a single (untagged) VLAN is set for the port on which my laptop is connected to (no more tagged VLAN in this port anymore). Problem solved "Properly" :-). Thank you.

I am experiencing the exact same behavior as described by blusens.

I am using several VLANs on the opnsense LAN interface. Two of them have IPv6 assigned by RA (in assisted mode) and DHCPv6 (WAN interface is getting DHCPv6 address from WAN, WAN interface is tracked by the VLANs). My Windows 10 computer is connected directly to a D-link DSG-1210-08P managed switch, which is connected to the LAN of the opnsense router.

On the switch port on which my computer is connected, I observe the following behavior:
1. If VLAN 81 is untagged and VLAN 89 tagged, I get 2 IPv6 in Windows (even after blocking all IPv6 traffic in the opnsense firewall)
2. If VLAN 81 is untagged and VLAN 89 is configured as "not member" (instead of "tagged") in the D-link switch, then I get the right and only the expected IPv6 address.

Is this behavior really the expected behavior? I thought that only the untagged VLAN should be "Undersood" by the Windows PC, ignoring all the tagged VLANs. Apparently, my supposition was wrong...

I just figured out where the issue was coming from.

I was using a specific "IP Update Password" consisting in all kind of characters: 0-9, a-z, A-Z and special characters. The issue was created by the special characters: they are wrongly interpreted if trying to use the update URL in a browser, and they are also disturbing the OPNsense Dynamic DNS updater. Once removing all the special characters, the IP update just worked as expected!

Not IPv4 and IPv6 is updating smoothly, with the Result Match parameters set as suggested by the Dynu tutorial page: good|nochg|good %IP%

Be carefull with special characters when used in passwords for UP Update: they can work on one device (this was my case on my Fritzbox) but they will bring you big troubles on other devices (like in OPNsense).