Messages

Alright, thank you very much for all the effort you put into it and even looking into it in the first place.

i was thinking of a ping using only the pfsync interface, but i havent thought about what would happen if only the pfsync interface would fail.

This is solved for me now. Again thank you very much.

Split Brain is exactly what i am experiencing and what i am trying to avoid in any way.
The Interface itself will stay up, but a single VLAN will fail (overload, switch firmware bug, misconfiguration)

Could i check for a heartbeat from backup to master node, maybe via a ping, on a separate network and only initiate a failover when the heartbeat dies.
Alternatively, could i somehow notify the master that the backup is taking over everything.

Now that i am writing it, notifying the master sounds kinda impossible without scripts and overly complicated.

So is there a way to handle a Split Brain Situation or would this be a limitation of HA?

Hey everyone,

I'm currently trying to set up my first CARP HA scenario.
I've installed and licensed the latest OPNsense Business Edition on both nodes.

Both OPNsense instances have two 10G ports configured as a LAG, with all networks configured as VLANs on that interface.
Additionally, each instance has two 1G ports. One is dedicated to pfsync, while the other is used for management access.
All VIPs on the master node have their skew set to 0, while those on the backup node are set to 100. The base is set to 1.
Preemption is enabled on both nodes (the checkbox to disable it is unchecked). I haven't configured anything regarding the demotion level so far.

As far as I understand it, the demotion level can trigger a complete failover if one of the interfaces goes down (assuming it's configured correctly).
However, since all CARP interfaces are VLANs, I assume the demotion mechanism would never be triggered because the VLAN interfaces can't go down individually.
This would mean that CARP failovers could still occur independently for each interface if a particular VLAN experiences connectivity issues or the switch gets misconfigured.

What I'm looking for is a way to prevent the backup node from taking over any CARP VIPs unless the primary node goes offline entirely.
I've found some solutions involving custom scripts that need to be installed via SSH directly in the filesystem. However, I'm a bit concerned that these scripts might break after updates or interfere with the update process itself.

That's why I'd prefer a solution that can be configured entirely through the GUI.
I also came across Monit, but I haven't looked into it in detail yet.

It might sound a bit strange that I'm looking for a solution to this particular scenario, considering that all networks are connected through a single 10G LAG and should theoretically fail together.
However, there are two main reasons for this.

1. Switch firmware bugs
Occasionally, the switches encounter a firmware bug that causes them to stop forwarding traffic for a single VLAN on one switch unit or overload just one particular VLAN. Sometimes even other Devices overload a VLAN which causes an interruption that the switch then can keep isolated in that VLAN.

2. Switch configuration errors
Sometimes, someone makes changes to the switch configuration and somehow manages to remove a VLAN from one of the OPNsense ports.

In both cases, I'd rather have an outage affecting a single network than have the backup node silently take over all CARP IPs, potentially hiding the underlying issue, or taking over only that particular VLAN, sending me in the wrong direction during troubleshooting.

Essentially, I want to avoid individual CARP VIP failovers and ensure that the backup node only takes over when the primary node fails completely.
Does anyone have an idea how to achieve this, preferably using only the OPNsense GUI?

Hey, thank you for your efforts.
Unfortunately, I'm not able to try this right now, and I won't have access to this system for the next few weeks. I was also hoping for a solution that doesn't require modifying configuration files or system files.

I'm looking for a solution that doesn't require creating custom shell scripts or modifying system files, as those kinds of workarounds can easily be forgotten or overwritten by future updates

That said, I will definitely give it a try the next time I have access to this environment.

Hey, thank you for your response.
As I mentioned before, I have already tried the cron job you suggested. The issue is that it only restarts WireGuard if it was previously connected. However, my connection is not online at all, so the cron job never triggers a restart because it doesn't detect any changes.
In other words, the cron job only reacts to state changes, not to initialization failures.

Hi everyone,

I've recently run into an issue with the latest OPNsense version, where WireGuard is included by default.

When I start the VPN tunnel manually, everything works as expected using the domain name. However, after rebooting the device, I noticed that the WireGuard tunnel appears to be enabled (everything is green), but no traffic passes through it. The only way to get it working again is to restart the tunnel manually.
I already tested it with a cron job, but the "stale connections" one only restarts it if there was an existing connection that got updated—like when the IP changes on one side or something similar. When I reboot, though, there was never any connection in the first place, which is why the DNS assumes nothing has changed and never updates the VPN/DNS.

I suspect this is related to the boot order. My guess is that WireGuard tries to establish the tunnel before the DNS service is fully available, so the domain name can't be resolved during startup. This theory seems to be supported by the fact that everything works perfectly if I configure the tunnel to use the public IP address instead of the domain name.

I'm looking for a solution that doesn't require creating custom shell scripts or modifying system files, as those kinds of workarounds can easily be forgotten or overwritten by future updates. I'm also not looking for the suggestion to "just use the IP address."

Has anyone encountered this issue before or found a script-free solution? If you need any additional information, feel free to ask.

Thanks in advance!

I think I found the problem. Every single conf file for the certificates has a value

Code Select Expand

Le_OCSP_Staple='1'
even though the GUI clearly shows it's disabled

When I force a renewal it works, but when I check the file it's still enabled

If I change the value in the file to 0 and then renew it also works, but it remains 0

Hey, we are facing the same issue.

Did you manage to solve this for good?

Sorry for not responding over the past week.

I spent some time testing different identifier settings, which of course caused some connection issues during configuration.
I wasn't able to get this to work using just one DSL line.
Even though the connection only dropped for a few seconds at a time, employees at the remote sites told the manager they were unable to work.

Because of that, I was told to stop changing anything as soon as it's working.

We'll see what happens when the main DSL line goes down – but until then, I'd say this scenario doesn't really work with OPNsense.
So I guess it's kinda solved for now.

Anyway, thanks for your help Monviech!

if i set the PSK values on both to the same, then the OPNsense would still match only based on the ip address and still use only one of the two configs.
Then the proposal would still not match for the second VPN.

I could set both VPNs to use the same proposal, then that would leave me basically with only one config for both remotes.
Can the OPNsense differentiate which ip network needs to go to which remote if anything else is identical?

They use a different Gateway.

I forgot to mention that both VPNs work if they use a different DSL line, but i want them on the same since the second DSL Line is only a backup.

Hello everyone,

I'm working on an existing infrastructure which is a new infrastructure to me. The network previously used three SonicWalls (one per site). Because of a very limited budget, only the main site can be migrated right now.

Current setup:
Main site: OPNsense (running on the latest business edition release) with two DSL lines (static IPs: 1.1.1.1 and 2.2.2.2) in failover mode, plus an LTE connection that is manually activated only if both DSL lines fail.
Two remote sites: each with a SonicWall, a single DSL line, and a dynamic IP (changes almost daily)

I'm trying to establish two site-to-site VPNs (one per remote site). Previously, both SonicWalls connected to 1.1.1.1 using IKEv1 Aggressive Mode. I'm trying to replicate this with OPNsense, but only one VPN connection works at a time.
Right now both VPNs are configured in IKEv1 Main Mode, i tried it with Aggressive Mode but it didnt seem to work at all.

The issue:
Site A: uses AES256-SHA256-ECP256 – connects successfully
Site B: uses AES256-SHA256-MODP2048 (the highest supported by that SonicWall) – fails to connect

Log shows:
Received proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
Configured proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/ECP_256

I've created two separate IPsec configs on OPNsense with the correct proposals. Both use 1.1.1.1 as the local address and 0.0.0.0/0 as remote peer.
Both have different local IKE IDs, remote IKE IDs and Pre Shared Keys which of course match with the corresponding remote site.
However, OPNsense seems to match only by IP and not by IKE ID. As a temporary workaround, I let Site A connect to 1.1.1.1 and Site B to 2.2.2.2. Ultimately, both should prefer 1.1.1.1 and automatically fail over to 2.2.2.2 if the primary DSL fails.

Is there a way to get this to work with the current SonicWalls and the OPNsense?

Thank you very much!
This helped me switching from regular certificates to wildcard certificates!
I now also do score 100% A+ in the SSL test.

So the thing is... (If I understood your setup correctly.)
To solve your issue you would have to create the relevant ACME condition, rule and a frontend listening on port 80 with the acme rule on it. This rule redirects all ACME challenges to localhost:ACMEport of your OPNsense. The Lets Encrypt Plugin is listening on the "ACMEport" of your OPNsense. All of that is created by the Lets Encrypt plugin automatically if you configure it to use HAProxy for SSL offloading.

But instead of the acme rule forwarding acme challenges to the localhost you would have to forward them to your seafile server.
As soon as there is another server that also has a certbot running things get very complicated!
Also I don't know if the conditions for the le-plugin_acme_challenges are identical to the ones of the certbot!

Since you are already reverse proxying, why don't you just let the OPNsense do the SSL offloading?
Just use the Let's Encrypt plugin and configure HAProxy with it.
Then you don't have to run certbots on every single server in your network.

Hello,

since the latest version of HAProxy now supports OCSP stapling I thought it would be a good idea to switch from the "workaround-script" to the official option.

However the option is not working for me. The system log shows the error below.

Code Select Expand

2021-04-06T03:42:01 configd.py[18127] [b2eb554c-4009-4e75-9e58-3f8a1a64e656] Script action stderr returned "b'WARNING: no nonce in response\nResponse verify OK\nWARNING: no nonce in response\nResponse verify OK'"
2021-04-06T03:42:00 configd.py[18127] [b2eb554c-4009-4e75-9e58-3f8a1a64e656] update haproxy ocsp data

I also made some pictures of my config.
https://imgur.com/a/XM9tpZt