HA + WAN in private network + Outbound NAT + IPsec and some problems

[voluhar]

Hello all,

We have HA pair of routers and Cisco pair of Switches in front if it.
On Cisco L3 Switches we also run BGP and there is no NAT on switches.

Networks and IPs are not real one.
Lats say that we have public following networks:
<DMZ> Network advertised over BGP: 1.2.0.0/22
<RUT2FW> (Cisco to OPNsense): 10.1.0.0/24
<LAN>: 10.1.1.0/24
<REM>Remote network on IPsec: 10.2.1.0/24

Configuration is like this: Datacenter Net <-> Cisco SW <RUT2FW> OPNsense <LAN>
                                                                          <DMZ>
<RUT2FW> SW1: 10.1.0.2
<RUT2FW> SW2: 10.1.0.3
<RUT2FW> SW HSRP: 10.1.0.1
<RUT2FW> RUT1: 10.1.0.11
<RUT2FW> RUT2: 10.1.0.12
<RUT2FW> RUT CARP: 10.1.0.10
<LAN>RUT1: 10.1.1.2
<LAN>RUT2: 10.1.1.3
<LAN>RUTCARP: 10.1.1.1
<DMZ>RUT1: 1.2.0.2
<DMZ>RUT2: 1.2.0.3
<DMZ>RUTCARP: 1.2.0.1

On switches we have following routes:
ip route 1.2.0.0 255.255.252.0   10.1.0.10
ip route 1.2.0.2 255.255.255.255 10.1.0.11
ip route 1.2.0.3 255.255.255.255 10.1.0.12

We have configured HA as described in documentation: https://docs.opnsense.org/manual/how-tos/carp.html
And default gateway on both OPNsense boxes is 10.1.0.1 (WAN_GW).

But after that we have some real problems.
1. <LAN> to <REM> traffic was flowing to WAN instead through IPsec tunnel.
2. Firmware update timed out.
3. Unbound was not working.

How we resolved first problem:
1. We are running IPsec on DMZ_CARP interface, because WAN is private network without NAT.
2. OPNsense route traffic directly to WAN_GW with Outbound NAT applied to it, before hitting IPsec.
3. Add rule in firewall -> Outbound NAT
   Interface: RUT2FW
   Source: 10.1.1.0/24
   Destination: 10.2.0.0/24
   Do not NAT: checked
   Network traffic
4. Now IPsec traffic flows as expected.

Resolution of second problem:
1. OPNsense itself want to reach internet with its WAN IP.
2. Ok, we add Outbound rule:
   Interface, Source, NAT to
   RUT2FW, 10.1.0.0/24, DMZ CARP address
3. This works on currently active Master, but not on slave. Logicaly.
4. But we change DMZ CARP addres in that rule to DMZ address, and same story.
   After checking slave OPNsense their Outbound rules shows that its NAT to address is actualy DMZ address of master.
   Because of that traffic coming out from slave returns to master.
4. Our solution is that instead of one rule in point 2 we add following rules. Adding rule with no XMLRPC Sync is not possible, because if we add rule on Slave OPNsense it will be overwritten with config Sync from Master. And we also want to avoid double configuration by hand on muster and slave because of bigger possibility of mistakes.
   Interface, Source, NAT to
   RUT2FW, 10.1.0.1, 1.2.0.1
   RUT2FW, 10.1.0.2, 1.2.0.2
   RUT2FW, 10.1.0.3, 1.2.0.3
5. After that change both OPNsense boxes can reach internet, and services like unbound are working normaly.

Yes it will be much easier to take slice of our network for Cisco to OPNsense connection but we did not want to throw away public address space.
I hope that also part of this can find way in documentation for someone else in such not everyday setup.




But there is still some problems that we can not solve and I hope that someone will be able to answer it.

Our actual DMZ interface setup is like this:
RUT1: 1.2.0.2/24, RUT2: 1.2.0.3/24
CARP VHID5: 1.2.0.1/24, CARP VHID5: 1.2.1.1/24, CARP VHID5: 1.2.2.1/24, CARP VHID5: 1.2.3.1/24

One of them is that in IPsec tunnel setting for interface we can not chose CARP all of this CARP IPs.
Actually we can chose only one CARP IP, only first IP that was entered within the same VHID.
Is this normal behavior ?

We are currently running version 19.1.4 and tonight we will update and reboot system and I can see if there will be any difference.

And another one is installation of OPNsense. I installed it yesterday in VM several times and always cut me off web interface after wizard. (also 19.1.4 DVD).
I figured out that I can login in Web interface after installation with my root password but after not putting anything for root password in web wizard it change it to default password.

Also I had some problems with DHCP static leases.
After adding static lease on master node I must restart DHCP service on master and slave otherwise computer get IP from DHCP range.
Can someone else verify this ?


Best regards, Robi

[voluhar]

After update to 19.1.8 we noticed that CARP did not fall back to master firewall and as mentioned in some other thread, reboot of master after "Leave persistent CARP maintenance mode" did help.

However, IPsec still showing only 1 of 4 CARP IPs that we want to be available for same interface for IPsec.

And we noticed problems regarding config synhronisation from master to slave.
DHCP leases and Suricata (Intrusion Prevention) is Synching only with manual sync with pressing perform synhronisation.