Messages

I'm beginning to wonder if there is a potential bug in software.

If I restart the primary firewall or simulate a power loss the VPN drops briefly (observed with 2-3 dropped packets) but then immediately picks up again which is the expected behaviour. Before I was clicking 'Enter Persistent CARP Maintenance Mode'.

Interestingly, when the primary firewall comes back up and becomes the master, it tries to failback but get the before behaviour of the VPN not coming up.

[2020-11-20T12:52:00 charon[27663] 16[IKE] <con1|1> remote address changed from ***.**.165.168 to ***.**.165.168]

Anything in the logs on the other side? Maybe other side tries to keep it open

The otherside is a OPNsense firewall which I have control of.

From checking the logs, nothing obvious stands out.

This is in the entry which is likely when it fails over:
2020-11-20T12:52:00   charon[27663]   16[IKE] <con1|1> remote address changed from ***.**.165.168 to ***.**.165.168

The above proves that the CARP IP / NAT is setup as expected.

Unfortunately all logs after that don't shed much light other than:

2020-11-20T12:52:11   charon[27663]   12[ENC] <con1|1> generating INFORMATIONAL response 2 [ ]
2020-11-20T12:52:11   charon[27663]   12[ENC] <con1|1> parsed INFORMATIONAL request 2 [ ]
2020-11-20T12:52:11   charon[27663]   12[NET] <con1|1> received packet: from ***.**.165.168[14197] to **.***.149.9[4500] (80 bytes)
2020-11-20T12:52:11   charon[27663]   14[NET] <con1|1> sending packet: from **.***.149.9[4500] to ***.**.165.168[14197] (80 bytes)
2020-11-20T12:52:11   charon[27663]   14[ENC] <con1|1> generating INFORMATIONAL request 2 [ ]
2020-11-20T12:52:11   charon[27663]   14[IKE] <con1|1> sending DPD request
2020-11-20T12:52:01   charon[27663]   16[NET] <con1|1> sending packet: from **.***.149.9[4500] to ***.**.165.168[14197] (80 bytes)
2020-11-20T12:52:01   charon[27663]   16[ENC] <con1|1> generating INFORMATIONAL response 1 [ ]

I have a HA setup that I am nearing completion on and putting into production but having an issue with an IPSec site-to-site VPN setup.

The VPN is configured to point to the CARP IP and this works as expected when on the primary.

When I put the primary into CARP maintainence mode, and the firewall fails over the the secondary firewall - The IPsec VPN tunnel takes a good 2+ minutes for traffic to switch over and pings to continue for example.

I have reviewed the policies and disabled MOBIKE but this has not make a difference unfortunately.

Many thanks

I'm also looking for an answer to this and can't find one.

Then you have to check with packet capture whats going wrong

Hi mimugmail

Thank you for your assistance - Having done a packet capture I could see [No response seen]. Which got me thinking.

There is one piece of information which I failed to mention which I should have mentioned.

My firewalls are virtualised on a VMware host (ESXI). Having toggled Promiscuous mode on the port group on the ESXI host it started working as expected and routing via the VIP.

I didn't think this would be required as it was working inbound but not outbound.

I can't see this mentioned in the documentation and feel it would be worth adding to the common issues section of 'Virtual & Cloud based Installation' so will make a suggestion to the team.

Usually this works perfect. Maybe split brain and both are master?

Have checked and master / backup status shows as expected  on both.

Had also tested failover and that worked aa expected l

[Setup outbound NATWhen traffic is going out of the firewall it should also use the virtual IP address to make a seamless migration possible. The default for OPNsense is to use the interfaces IP address, which is in our case the wrong one.Go to Firewall ‣ NAT ‣ Outbound. Choose manual outbound nat on this page and change the rules originating from the 192.168.1.0/24 network to use the CARP virtual interface (172.18.0.100).]

Hi

I have two firewalls setup in a HA, mostly setup now including CARP for each interface including WAN.

I am able to reach the CARP IP externally and have tested a failoverr test using CARP whilst running a ping check and all works as expected.

As per the documentation on HA it says to adjust outbound NAT as per

Setup outbound NAT

When traffic is going out of the firewall it should also use the virtual IP address to make a seamless migration possible. The default for OPNsense is to use the interfaces IP address, which is in our case the wrong one.

Go to Firewall ‣ NAT ‣ Outbound. Choose manual outbound nat on this page and change the rules originating from the 192.168.1.0/24 network to use the CARP virtual interface (172.18.0.100).

However when setting up the above the devices on that given network lose internet access. Screenshot attached from my test environment.

My test environment uses internal IPs but I am having the same issue with a customers environment which uses Public IPs

Setup of customer side is as follows

Primary firewall
LAN IP: 10.0.0.253
LAN VIP: 10.0.0.254
WAN IP: XXX.XX.XX.169
VIP (CARP) IP: XXX.XX.XX.168
Gateway IP: XXX.XX.XX.161

Secondary firewall
LAN IP: 10.0.0.252
LAN VIP: 10.0.0.254
WAN IP: XXX.XX.XX.170
VIP (CARP) IP: XXX.XX.XX.168
Gateway IP: XXX.XX.XX.161

Any help is muchly appreciated

Environment:
Single OPNsense 20.1.7-amd64 firewall (virtualised)
ISP Router > OPnsense Router with 80 & 443 port forwarded

I know my ISP router is not the issue as I can see the traffic hitting the OPNsense firewall logs.

For something that should be simple this is certainly causing a lot of frustration.

I am trying to get NGINX to work and act as a reverse proxy. The long term goal is to have a Wordpress server and Nextcloud server running concurrently on 443 using the one 'WAN IP' which is in effect a private IP as it is sat behind my ISP router.

I have ensured my OPNsense firewall port is not using the default 443 and the direct 80 > 443 setting is disabled.

I have checked over the documentation numerous times as well as the OPNsense forum but with no joy. Documentation around required firewall is quite vague but I had it set to:

Source = *
Port = *
Destination 192.168.10.253
Port 443 (HTTPS)

I am not concerned about Nextcloud for the time being and solely focusing on the Wordpress web server. I know the web server works because when I setup NAT and disable NGINX, I can reach it.

I have now removed all settings so that I am back to a clean slate so do not have a NGINX config to provide at this time.

Is someone able to provide a very quick step 1, step 2, step 3 for me to follow and I can then see if I am being completely blind or it is just isn't working. If it continues to not work, I can then pull the NGINX config to see what is going on for a config that should work.

Having looked through the documentation as well as the forum, I have not been able to find an answer to my problem.

Thank you in advance.

Before proceeding, I would like to point out that if I do not use HAproxy and simply NAT port 443 to my Nextcloud server, I get the Nextcloud splash page as expected which confirms my Nextcloud server is up and reachable.

For sake of troubleshooting, I have disabled HA and I am currently using a standalone firewall with the following details:

OPNsense 20.1.6-amd64
WAN IP = 192.168.10.1 (Sat behind ISP router with port 80 / 443 forwarded to WAN IP of OPnsense firewall).

Steps to setup:
1. Ensure no firewall rules exist for port 443 on WAN interface
2. Install HAproxy plugin
3. Add 'Real Server' with the IP of Nextcloud server (192.168.66.1)
4. Create back-end pool and include the above Nextcloud server
5. Create front-end and listen on 192.168.10.1:443
6. Set Default Backend Pool to one created in step 4
7. Click apply and test syntax
8. Confirm no syntax issues are present
9. Enable HAproxy
10. Setup rule on WAN interface with following settings:
Source = Any
Port = *
Destination = WAN interface (192.168.10.1)
Port = 443 (HTTPS)

Unfortunately the above does not work and the logs simply say backend started. I have referred to the documentation but still not getting anywhere. Either the documentation is wrong or I am missing something blatently obvious.

bump

[Setup:ISP Router > OPNsense VIP (192.168.10.253)FW01 = 192.168.10.1FW02 = 192.168.10.2The firewalls are virtualised and are in HA on the VIP of 192.168.10.253]

Setup:
ISP Router > OPNsense VIP (192.168.10.253)
FW01 = 192.168.10.1
FW02 = 192.168.10.2

The firewalls are virtualised and are in HA on the VIP of 192.168.10.253

I've just spend ages trying to troubleshoot an issue whereby I could not access a test Wordpress website that is behind my OPnsense firewalls in HA.

Initial thoughts were:

1. I hadn't setup a firewall rule or a NAT rule properly
2. Double NAT was interfering

I've now located and pinpointed the exact cause which is the outbound NAT which I have set to manual as per the documentation when using HA. As soon as I set it back to automatic, I am able to access my Wordpress website remotely.

My outbound manual NAT rules consist of the following:

Rule 1:
Interface = WAN
Source = Any
Source Port = *
Destination = *
Destination Port = *
NAT address = WAN VIP
NAT Port = *
Static Port = No

Rule 2:
Interface = WAN
Source = Any
Source Port = *
Destination = *
Destination Port = 500
NAT address = WAN VIP
NAT Port = *
Static Port = Yes

I have attempted to set a manual rule which reflects the automatic rule which essentially sets the NAT address to 'WAN'. This however has now helped.

For the time being I have set the NAT back to automatic but this will not mean full HA.

Has anyone got any ideas on what I could have done wrong?

Thank you in advance.

Setup:
1. Virtualised OPNsense  router
2. Version = 20.1.3-i386
3. Sat behind a router so has a private IP for it's WAN address

Either i'm being absolutely blind or firewall creation is not as intuitive as it could be.

I have a virtualised OPNsense router for my lab at home which is sat behind my ISP router. The ISP router does the routing to the WWW and the OPNsense router has a private IP for it's WAN address.

Clients are able to access the internet when I set the destination as '*' - e.g. on LAN interface. Obviously this also grants 'Any' access to other vlans / interfaces I have setup.

Looking online I see various conflicting advice with most people advising to just use '*' which is wrong and shocking that people think that this is the answer.

Initial thoughts was to use the WAN Net or WAN Address alias's but from looking online this only permits to either the WAN address of the OPNsense or the WAN Network (e.g. WAN Network subnet).

On pretty much all other firewalls, you would create a LAN > WAN > HTTPS > ANY (From Zone > To Zone > Service > Destination) rule. Unfortunately this logic does not seem to apply on OPNsense.

As a temporary measure I have put the (*) rule in place to grant internet access but I would like this locked down.

I have tried playing with the direction setting on both the LAN and the WAN interface firewall rules but can not get the correct behaviour.

I have looked across various forums but have not had much luck but have found that people have done the following:

Create an alias of private networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), set that alias as the destination in the firewall then invert the destination

The above seems clunky and is surely not the solution that the developers intended for us to use.

Does anyone know what the 'best practice' rule should look like to permit  LAN > WAN > HTTPS > ANY (From Zone > To Zone > Service > Destination) without giving access to everything else with the dreaded (*)

Thank you in advance  :D