Messages

You have to nat outgoing traffic from OPNsense itself (127.0.0.0/8) to the WAN address.

All other traffic from networks behind has to be natted to the WAN VIP.

Sorry, I did not try your proposal as of August 17 carefully.
Packets from the FW itself have the WAN address as source,
so I used that as filter:

Code Select Expand

root@opn2:~ # pfctl -s nat
no nat proto carp all
nat on igb1 inet from ! <opn2_igb1_address> to any -> 192.168.178.2 port 1024:65535
nat on igb1 inet from <opn2_igb1_address> to any -> <opn2_igb1_address> port 1024:65535 round-robin
no rdr proto carp all
root@opn2:~ # ping heise.de
PING heise.de (193.99.144.80): 56 data bytes
64 bytes from 193.99.144.80: icmp_seq=0 ttl=249 time=7.748 ms
64 bytes from 193.99.144.80: icmp_seq=1 ttl=249 time=5.823 ms

So it's finally resolved.

Thanks a lot viragomann;
ajr

Forget it.
A 2nd default route from another IF address to the some gateway will not be added.

No trick to get the CARP backup device working on the backup system ?

ajr

Adding an alias to the WAN interface and using it as default route to the upstream gateway (FB) seems to work:

Code Select Expand

ifconfig igb1 add alias 192.168.178.110/24
route add default 192.168.178.110 192.168.178.1

ping heise.de
64 bytes from 193.99.144.80: icmp_seq=0 ttl=248

Can this be done in the GUI ?

ajr

Quote from: viragomann on July 24, 2025, 09:23:22 PM
    What do your outbound NAT rules look like?

See atachment.

I seem to miss the attachment (still struggling with the formum GUI).
Here it comes.

ajr

What do your outbound NAT rules look like?

See atachment.

Did override its outbound behavior with a manual rule by any chance?

I'm not aware of any.

Do you want look at my complete rules set (per PM) ?

ajr

Responses to outbound IP4 packets on WAN interface (igb1) of HA backup system are blocked.
Either because all private addresses are blocked.
If I allow private addresses on WAN interface, they are bolcked by state violation rule.

Why is no state created ?

Do I need a 2nd NAT rule, because the WAN VIP is not available at backup firewall ?

There is only one outbound NAT rule:
All source addresses are NATed.
Outbound NAT-address is the WAN VIP

Code Select Expand

igb1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: WAN (wan)
options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
ether 00:0d:b9:4f:fd:a1
inet 192.168.178.12 netmask 0xffffff00 broadcast 192.168.178.255
inet 192.168.178.2 netmask 0xffffff00 broadcast 192.168.178.255 vhid 1
inet6 fe80::20d:b9ff:fe4f:fda1%igb1 prefixlen 64 scopeid 0x2
inet6 2003:cb:170c:9700:20d:b9ff:fe4f:fda1 prefixlen 64 autoconf pltime 1799 vltime 7200
inet6 fd77:8819:994b:0:20d:b9ff:fe4f:fda1 prefixlen 64 autoconf pltime 3600 vltime 7200
carp: BACKUP vhid 1 advbase 1 advskew 100
      peer 224.0.0.18 peer6 ff02::12
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


Ping to VDSL-Router (FB) shows:

root@opn2:~ # ping -S 192.168.178.12 192.168.178.1
PING 192.168.178.1 (192.168.178.1) from 192.168.178.12: 56 data bytes
^C

root@opn2:~ # tcpdump -nves 300 -i igb1 icmp
tcpdump: listening on igb1, link-type EN10MB (Ethernet), snapshot length 300 bytes
18:31:25.136468 00:0d:b9:4f:fd:a1 > cc:ce:1e:b3:75:7f, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 19554, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.178.2 > 192.168.178.1: ICMP echo request, id 17147, seq 0, length 64

root@opn2:~ # tcpdump -nves 300 -i pflog0 icmp
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), snapshot length 300 bytes
18:27:33.276553 rule 200/0(match): block in on igb1: (tos 0x0, ttl 63, id 56317, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.178.2 > 192.168.178.12: ICMP echo reply, id 49522, seq 6, length 64



Why seem responses come from the WAN VIP instead from FB ?

Which rules are needed to allow outgoing IP4 traffic of backup system ?

Current rules related to igb1 attached.

Please advice,
ajr

No significant entries in logfile.
Maybe related: My OpenVPN tunnel provides the IPv6 default route.

I'm trying now WireGuard, as OpenVPN legacy client is obsolete ...

Thanks for replying

I upgraded may backup node and then did a failover to (persistent CARP switch) to upgrade the old master.
The OpenVPN client could not create a tunnel.
I switched back to the old master (running 25.1.5_5) and the tunnel came up.

Does the OpenVPN configuration needs a change with 25.1.7, related to (from README):
  openvpn: add port-share as advanced feature
  openvpn: add (push) block-ipv6 option
?

Is this a known bug ?

What else can I do to resolve the issue ?

ajr

I upgraded may backup node and then did a failover to (persistent CARP switch) to upgrade the old master.
The OpenVPN client could not create a tunnel.
I switched back to the master with old master and the tunnel came up.

Does the OpenVPN configuration needs a change with 25.1.7, related to:
openvpn: add port-share as advanced feature
openvpn: add (push) block-ipv6 option

Unfortunately this breaks connectivity of the backup system and
 needs some hack (route through master system) to do firmware update.
 

Unfortunately I can't get selection of backup gateway as default gateway working.
Even if gateway monitoring is on and "Allow default gateway switching" is on in system->settings->general.
It seems that gateway priority always takes precedence. See attached screenshot.

root@opn2:~ # netstat -rnfinet | grep default
default            192.168.178.1      UGS            igb1

How can I fix this ?

This setup works (for me) only if I have deleted the IPv4 address
 of the WAN interface (keeping only the virtual address).

Can anybody please explain, why it works only with this setup ?

ajr

 I have a setup where WLANs receive (periodically changing)
 DHCPv6 nets from the DSL router und some LANs receive static
 public IPv6 addresses via a OpenVPN tunnel which also provides
 the route to the internet for them.
 
 This setup works (for me) only if I have deleted the IPv4 address
 of the WAN interface (keeping only the virtual address).
 
 Unfortunately this breaks connectivity of the backup system and
 needs some hack (route through master system) to do firmware update.
 
 How can I replace the hack through some automatic gateway config
 change, e.g. gateway monitoring/scripting ?
 
 Is there a better solution for may dual IPv6 WAN setup ?
 
 
 Thanks, ajr
 
 PS: some details:
 
HA configuration (master/backup)
All interfaces have VIPs via CARP
All IPv4 addresses use NAT

LAN nets
 IPv4: static (rfc1918)
 IPv6: static (subnet from VPN)

WLAN nets (via APs)(all have VIPs via CARP)
 IPv4: static (rfc1918)
 IPv6: Track interface (DHCPv6)
 
WAN Interface (transfer net to VDSL router)
 IPv4: none
 IPv6: DHCPv6

Gateways
 IPv4: VDSL router (if master, VIP, Monitor IP router))
  IPv6: DHCPv6

OpenVPN client(legacy)
 Server Mode: Peer to Peer
 Interface: WAN VIP
 IPv6 Remote Network: ::/1,8000::/1
 

Hi Franco,

UPGRADE SUCCEEDED !

/var/cache/opnsense-update/.upgrade.log is 40kB !


Thanks a lot for your time and your patience.

Ajr

Hi Franco,


The check for updates page shows

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.7.12_4 (amd64) at Thu Feb  6 10:21:00 UTC 2025
Fetching changelog information, please wait... done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
OPNsense repository update completed. 869 packages processed.
All repositories are up to date.
Child process pid=2663 terminated abnormally: Segmentation fault
Checking integrity... done (0 conflicting)
Your packages are up to date.
Checking for upgrades (1 candidates): . done
Processing candidates (1 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.

The next page shows current version 25.1 and new version 24.7.12 for both
base and kernel.

So the update script seems to be confused by the combination of OS 14.2 and
opnsense 24.7.3

Shall I try a opnsense-update -kr 24.7.12 ?


After an upgrade attempt (opnsense-update -ur 25.1), the audit log again shows
    pkg-1.21.3 version mismatch, expected 1.19.2_5

Now, reverting the pkg does no longer work:

root@opn2:~ # pkg info | grep pkg
pkg-1.21.3                     Package manager
root@opn2:~ # opnsense-revert -r 24.7.12 pkg
Fetching pkg.pkg: ...... done
Verifying signature with trusted certificate pkg.opnsense.org.20240611... done
pkg-1.21.3: already unlocked

root@opn2:~ # pkg info | grep pkg
pkg-1.21.3                     Package manager
root@opn2:~ # pkg lock pkg
pkg-1.21.3: lock this package? [y/N]: y
Locking pkg-1.21.3
root@opn2:~ # opnsense-revert -r 24.7.12 pkg
Fetching pkg.pkg: .... done
Verifying signature with trusted certificate pkg.opnsense.org.20240611... done
Unlocking pkg-1.21.3
Installing pkg-1.19.2_5...
package pkg is already installed, forced install
Extracting pkg-1.19.2_5: 100%

But now the audit shows:

pkg-1.19.2_5 repository mismatch: unknown-repository

From the log, it seems, unlocking an already unlocked package seems to be fatal.
bash-5.2.37: already unlocked

So I removed bash earlier.

But now:

root@opn2:~ # opnsense-update -G
beep-1.0_2: already unlocked

Any clue ?
Should I lock all packages and retry ?

Any options left to recover my situation ?

Ajr

Also, any chance you might have deleted /var/cache while troubleshooting - before coming here that is - and that is why you had to manually recreate the missing folder ?

I never deleted any log files/directories.