Messages

tcpdump does not show any packets on the WAN interface so I do not know the sender address.

Any source address in packets stemming from 127.0.0.0/8 is translated to the CARP VIP on the WAN due to your rule. So it's obvious the you cannot see any IP of this subnet.^^

So I try this:

Code Select Expand

root@opn1:~ # pfctl -s nat
no nat proto carp all
nat on igb1 inet from ! <opn1_igb1_plus_lo_addr> to any -> 192.168.178.2 port 1024:65535
nat on igb1 inet from <opn1_igb1_plus_lo_addr> to any -> <opn1_igb1_address> port 1024:65535 round-robin
root@opn1:~ # pfctl -T show -t opn1_igb1_address
   192.168.178.11
root@opn1:~ # pfctl -T show -t opn1_igb1_plus_lo_addr
   127.0.0.0/8
   192.168.178.11
192.168.178.11 is the interface address and 192.168.178.2 is the VIP.

Any comments ?

Does not resolve the issue:

Code Select Expand

# pfctl -s nat
no nat proto carp all
nat on igb1 inet from ! <opn2_igb1_plus_lo_addr> to any -> 192.168.178.2 port 1024:65535
nat on igb1 inet from <opn2_igb1_plus_lo_addr> to any -> <opn2_igb1_address> port 1024:65535 round-robin
root@opn2:~ # pfctl -T show -t opn2_igb1_address
   192.168.178.12
root@opn2:~ # pfctl -T show -t opn2_igb1_plus_lo_addr
   127.0.0.0/8
   192.168.178.12
root@opn2:~ # ping 192.168.178.1
PING 192.168.178.1 (192.168.178.1): 56 data bytes
ping: sendto: Invalid argument

What am I doing wrong ?

tcpdump does not show any packets on the WAN interface so I do not know the sender address.

Any source address in packets stemming from 127.0.0.0/8 is translated to the CARP VIP on the WAN due to your rule. So it's obvious the you cannot see any IP of this subnet.^^

So I try this:

Code Select Expand

root@opn1:~ # pfctl -s nat
no nat proto carp all
nat on igb1 inet from ! <opn1_igb1_plus_lo_addr> to any -> 192.168.178.2 port 1024:65535
nat on igb1 inet from <opn1_igb1_plus_lo_addr> to any -> <opn1_igb1_address> port 1024:65535 round-robin
root@opn1:~ # pfctl -T show -t opn1_igb1_address
   192.168.178.11
root@opn1:~ # pfctl -T show -t opn1_igb1_plus_lo_addr
   127.0.0.0/8
   192.168.178.11
192.168.178.11 is the interface address and 192.168.178.2 is the VIP.

Any comments ?

author=viragomann link=msg=266681 date=1778350489]
You nat any traffic on WAN to the CARP VIP apart from the interface address. But the CARP VIP is naturally occupied by the master node, and the backup cannot use it.

Exactly that is addressed by the 2 NAT rules.

Traffic from OPNsense itself might rather come from 127.0.0.0/8, however. So you have to nat this subnet to the WAN interface address.

tcpdump does not show any packets on the WAN interface so I do not know the sender address.

This is version 26.1.5
On BACKUP box, all outgoing traffic is blocked.
tcpdump -n -e -ttt -i pflog0 does not show any related violations.
192.168.178.1 is DSL Router.

Seems to be a new problem with 26.x.
Any help appreciated.

ajr

Code Select Expand

root@opn2:~/admin # ping 192.168.178.1
PING 192.168.178.1 (192.168.178.1): 56 data bytes
ping: sendto: Invalid argument
ping: sendto: Invalid argument
root@opn2:~ # ifconfig igb1
igb1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether 00:0d:b9:4f:fd:a1
inet 192.168.178.12 netmask 0xffffffff broadcast 192.168.178.12
inet 192.168.178.2 netmask 0xffffff00 broadcast 192.168.178.255 vhid 1
inet6 fe80::20d:b9ff:fe4f:fda1%igb1 prefixlen 64 scopeid 0x2
inet6 some_DTAG_IP6 prefixlen 64 autoconf pltime 1280 vltime 7200
inet6 fd77:8819:994b:0:20d:b9ff:fe4f:fda1 prefixlen 64 autoconf pltime 3600 vltime 7200
carp: BACKUP vhid 1 advbase 1 advskew 100
      peer 224.0.0.18 peer6 ff02::12
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@opn2:~ # pfctl -s nat
no nat proto carp all
nat on igb1 inet from ! <opn2_igb1_address> to any -> 192.168.178.2 port 1024:65535
nat on igb1 inet from <opn2_igb1_address> to any -> <opn2_igb1_address> port 1024:65535 round-robin
no rdr proto carp all
no rdr on igb0 proto tcp from any to (igb0) port = http
no rdr on igb0 proto tcp from any to (igb0) port = 44221
no rdr on igb0 proto tcp from any to (igb0) port = 44441
root@opn2:~ # tcpdump -nvs 300 -i igb1 not vrrp
tcpdump: listening on igb1, link-type EN10MB (Ethernet), snapshot length 300 bytes
16:20:29.133096 5c:6a:80:f5:84:a0 > ff:ff:ff:ff:ff:ff, Realtek unknown type 0x25
root@opn2:~ # route get 193.99.144.80
   route to: redirector.heise.de
destination: default
       mask: default
    gateway: 192.168.178.1
        fib: 0
  interface: igb1
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0




You have to nat outgoing traffic from OPNsense itself (127.0.0.0/8) to the WAN address.

All other traffic from networks behind has to be natted to the WAN VIP.

Sorry, I did not try your proposal as of August 17 carefully.
Packets from the FW itself have the WAN address as source,
so I used that as filter:

Code Select Expand

root@opn2:~ # pfctl -s nat
no nat proto carp all
nat on igb1 inet from ! <opn2_igb1_address> to any -> 192.168.178.2 port 1024:65535
nat on igb1 inet from <opn2_igb1_address> to any -> <opn2_igb1_address> port 1024:65535 round-robin
no rdr proto carp all
root@opn2:~ # ping heise.de
PING heise.de (193.99.144.80): 56 data bytes
64 bytes from 193.99.144.80: icmp_seq=0 ttl=249 time=7.748 ms
64 bytes from 193.99.144.80: icmp_seq=1 ttl=249 time=5.823 ms

So it's finally resolved.

Thanks a lot viragomann;
ajr

Forget it.
A 2nd default route from another IF address to the some gateway will not be added.

No trick to get the CARP backup device working on the backup system ?

ajr

Adding an alias to the WAN interface and using it as default route to the upstream gateway (FB) seems to work:

Code Select Expand

ifconfig igb1 add alias 192.168.178.110/24
route add default 192.168.178.110 192.168.178.1

ping heise.de
64 bytes from 193.99.144.80: icmp_seq=0 ttl=248

Can this be done in the GUI ?

ajr

Quote from: viragomann on July 24, 2025, 09:23:22 PM
    What do your outbound NAT rules look like?

See atachment.

I seem to miss the attachment (still struggling with the formum GUI).
Here it comes.

ajr

What do your outbound NAT rules look like?

See atachment.

Did override its outbound behavior with a manual rule by any chance?

I'm not aware of any.

Do you want look at my complete rules set (per PM) ?

ajr

Responses to outbound IP4 packets on WAN interface (igb1) of HA backup system are blocked.
Either because all private addresses are blocked.
If I allow private addresses on WAN interface, they are bolcked by state violation rule.

Why is no state created ?

Do I need a 2nd NAT rule, because the WAN VIP is not available at backup firewall ?

There is only one outbound NAT rule:
All source addresses are NATed.
Outbound NAT-address is the WAN VIP

Code Select Expand

igb1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether 00:0d:b9:4f:fd:a1
inet 192.168.178.12 netmask 0xffffff00 broadcast 192.168.178.255
inet 192.168.178.2 netmask 0xffffff00 broadcast 192.168.178.255 vhid 1
inet6 fe80::20d:b9ff:fe4f:fda1%igb1 prefixlen 64 scopeid 0x2
inet6 2003:cb:170c:9700:20d:b9ff:fe4f:fda1 prefixlen 64 autoconf pltime 1799 vltime 7200
inet6 fd77:8819:994b:0:20d:b9ff:fe4f:fda1 prefixlen 64 autoconf pltime 3600 vltime 7200
carp: BACKUP vhid 1 advbase 1 advskew 100
      peer 224.0.0.18 peer6 ff02::12
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


Ping to VDSL-Router (FB) shows:

root@opn2:~ # ping -S 192.168.178.12 192.168.178.1
PING 192.168.178.1 (192.168.178.1) from 192.168.178.12: 56 data bytes
^C

root@opn2:~ # tcpdump -nves 300 -i igb1 icmp
tcpdump: listening on igb1, link-type EN10MB (Ethernet), snapshot length 300 bytes
18:31:25.136468 00:0d:b9:4f:fd:a1 > cc:ce:1e:b3:75:7f, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 19554, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.178.2 > 192.168.178.1: ICMP echo request, id 17147, seq 0, length 64

root@opn2:~ # tcpdump -nves 300 -i pflog0 icmp
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), snapshot length 300 bytes
18:27:33.276553 rule 200/0(match): block in on igb1: (tos 0x0, ttl 63, id 56317, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.178.2 > 192.168.178.12: ICMP echo reply, id 49522, seq 6, length 64



Why seem responses come from the WAN VIP instead from FB ?

Which rules are needed to allow outgoing IP4 traffic of backup system ?

Current rules related to igb1 attached.

Please advice,
ajr

No significant entries in logfile.
Maybe related: My OpenVPN tunnel provides the IPv6 default route.

I'm trying now WireGuard, as OpenVPN legacy client is obsolete ...

Thanks for replying

I upgraded may backup node and then did a failover to (persistent CARP switch) to upgrade the old master.
The OpenVPN client could not create a tunnel.
I switched back to the old master (running 25.1.5_5) and the tunnel came up.

Does the OpenVPN configuration needs a change with 25.1.7, related to (from README):
  openvpn: add port-share as advanced feature
  openvpn: add (push) block-ipv6 option
?

Is this a known bug ?

What else can I do to resolve the issue ?

ajr

I upgraded may backup node and then did a failover to (persistent CARP switch) to upgrade the old master.
The OpenVPN client could not create a tunnel.
I switched back to the master with old master and the tunnel came up.

Does the OpenVPN configuration needs a change with 25.1.7, related to:
openvpn: add port-share as advanced feature
openvpn: add (push) block-ipv6 option

Unfortunately this breaks connectivity of the backup system and
 needs some hack (route through master system) to do firmware update.
 

Unfortunately I can't get selection of backup gateway as default gateway working.
Even if gateway monitoring is on and "Allow default gateway switching" is on in system->settings->general.
It seems that gateway priority always takes precedence. See attached screenshot.

root@opn2:~ # netstat -rnfinet | grep default
default            192.168.178.1      UGS            igb1

How can I fix this ?

This setup works (for me) only if I have deleted the IPv4 address
 of the WAN interface (keeping only the virtual address).

Can anybody please explain, why it works only with this setup ?

ajr