[SOLVED] Strange Gateway hopping

Started by computeralex92, January 13, 2022, 03:15:43 PM

Previous topic - Next topic
Print
Go Down Pages1 2
User actions
January 13, 2022, 03:15:43 PM Last Edit: January 18, 2022, 07:06:35 PM by computeralex92
Hello,

after upgrading to 22.1 R1 from 21.7 I saw a strange "gateway hopping".

Background:
To my firewall there are two internet connections connected:

  • Telekom via PPPoE
  • Vodafone via DHCP (ISP Provider in bridge mode)

In the past the gateways were configured that both gateways from Telekom (PPPoE IPv4 & DHCPv6 IPv6) have the priority of 253, the gateway for Vodafone have 254.
That was working as expected, as normally the Telekom connections were used mainly and only if there was a issue, the vodafone gateway were used.

Now after the switch to 22.1 I noticed that often the connection is switching between Telekom and Vodafone, which  cause issues with VOIP and other services which needs a stable connection.

The only tempfix now was to disable the vodafone interface; which is of course not ideal.
I know the "normal" way to handle a failover connection is to use the gateway group; but out of KISS reasons I tested the above solution and it worked for month without any major issue.

Was there a change in the routing behavior in 22.1?

Thanks,
Alex
January 13, 2022, 03:25:50 PM #1
Is this a gateway monitoring issue or something else? Can you give use some logs to trace the code with?

The gateway code didn't change vs. 21.7.x but much of the interface handling was scrubbed from excess complexity especially during the boot sequence.


Thanks,
Franco
January 13, 2022, 03:45:02 PM #2
Which log files would help to debug this?

I disabled the monitoring of the gateways already; but it still is happening.
January 13, 2022, 03:49:20 PM #3
Relevant entries from System: Log Files: General and System: Gateways: Log File.

Something needs to trigger the change in (default?) gateway... assuming you have default gateway switching turned on.


Cheers,
Franco
January 13, 2022, 04:10:05 PM #4 Last Edit: January 13, 2022, 04:11:40 PM by computeralex92
This is the log after I disabled the gateway monitoring:

Code Select
2022-01-13T15:04:08 Error opnsense /usr/local/etc/rc.newwanipv6: Resyncing OpenVPN instances for interface Telekom.
2022-01-13T15:04:08 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'fe80::9ecc:83ff:fec8:789f%pppoe0'
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway '62.155.242.170'
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: The VODAFONE_DHCP IPv4 gateway address is invalid, skipping.
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: Choose to bind VODAFONE_DHCP on since we could not find a proper match.
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: Removing static route for monitor 1.1.1.1 via 62.155.242.170
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: Choose to bind TELEKOM_PPPOE on 91.9.129.160 since we could not find a proper match.
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: Removing static route for monitor 2606:4700:4700::1111 via fe80::9ecc:83ff:fec8:789f%pppoe0
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway 'fe80::9ecc:83ff:fec8:789f%pppoe0'
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv6 default route to fe80::9ecc:83ff:fec8:789f
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: IPv6 default gateway set to opt2
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: keeping current default gateway '62.155.242.170'
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: setting IPv4 default route to 62.155.242.170
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: IPv4 default gateway set to opt2
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: ROUTING: entering configure using 'opt2'
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: On (IP address: 2003:ef:bfff:e3f:227c:14ff:fea0:644e) (interface: Telekom[opt2]) (real interface: pppoe0).
2022-01-13T15:04:07 Error opnsense /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'
2022-01-13T15:04:06 Error opnsense /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway 'fe80::9ecc:83ff:fec8:789f%pppoe0'
2022-01-13T15:04:06 Error opnsense /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway '62.155.242.170'
2022-01-13T15:04:06 Error opnsense /interfaces.php: The VODAFONE_DHCP IPv4 gateway address is invalid, skipping.
2022-01-13T15:04:06 Error opnsense /interfaces.php: Choose to bind VODAFONE_DHCP on since we could not find a proper match.
2022-01-13T15:04:06 Error opnsense /interfaces.php: Removing static route for monitor 1.1.1.1 via 62.155.242.170
2022-01-13T15:04:06 Error opnsense /interfaces.php: Choose to bind TELEKOM_PPPOE on 91.9.129.160 since we could not find a proper match.
2022-01-13T15:04:06 Error opnsense /interfaces.php: The TELEKOM_DHCP6 IPv6 gateway address could not be found, skipping.
2022-01-13T15:04:06 Error opnsense /interfaces.php: ROUTING: keeping current default gateway 'fe80::9ecc:83ff:fec8:789f%pppoe0'
2022-01-13T15:04:06 Error opnsense /interfaces.php: ROUTING: setting IPv6 default route to fe80::9ecc:83ff:fec8:789f
2022-01-13T15:04:06 Error opnsense /interfaces.php: ROUTING: IPv6 default gateway set to opt2
2022-01-13T15:04:06 Error opnsense /interfaces.php: ROUTING: keeping current default gateway '62.155.242.170'
2022-01-13T15:04:06 Error opnsense /interfaces.php: ROUTING: setting IPv4 default route to 62.155.242.170
2022-01-13T15:04:06 Error opnsense /interfaces.php: ROUTING: IPv4 default gateway set to opt2
2022-01-13T15:04:06 Error opnsense /interfaces.php: ROUTING: entering configure using defaults
2022-01-13T15:04:06 Error opnsense /interfaces.php: Clearing states for stale opt1 route on igb2
2022-01-13T15:04:06 Critical dhclient exiting.
2022-01-13T15:04:06 Error dhclient connection closed
2022-01-13T15:04:06 Error opnsense /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface Telekom.
2022-01-13T15:04:06 Error opnsense /usr/local/etc/rc.newwanip: IP address change detected, killing states of old ip 91.9.129.136
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'fe80::9ecc:83ff:fec8:789f%pppoe0'
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '62.155.242.170'
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: Choose to bind VODAFONE_DHCP on 178.27.64.56 since we could not find a proper match.
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: Removing static route for monitor 1.1.1.1 via 62.155.242.170
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: Choose to bind TELEKOM_PPPOE on 91.9.129.160 since we could not find a proper match.
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: The TELEKOM_DHCP6 IPv6 gateway address could not be found, skipping.
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanipv6: Failed to detect IP for Telekom[opt2]
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanipv6: On (IP address: ) (interface: Telekom[opt2]) (real interface: pppoe0).
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanipv6: IPv6 renewal is starting on 'pppoe0'
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'fe80::9ecc:83ff:fec8:789f%pppoe0'
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to fe80::9ecc:83ff:fec8:789f
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to opt2
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '62.155.242.170'
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 62.155.242.170
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to opt2
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt2'
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: On (IP address: 91.9.129.160) (interface: Telekom[opt2]) (real interface: pppoe0).
2022-01-13T15:04:05 Error opnsense /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'pppoe0'
2022-01-13T15:04:05 Error opnsense /interfaces.php: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on igb0
2022-01-13T15:04:05 Error opnsense /interfaces.php: ROUTING: skipping IPv6 default route
2022-01-13T15:04:05 Error opnsense /interfaces.php: ROUTING: IPv6 default gateway set to opt2
2022-01-13T15:04:05 Error opnsense /interfaces.php: ROUTING: entering configure using 'opt2'
2022-01-13T15:04:05 Error opnsense /interfaces.php: Accept router advertisements on interface pppoe0
2022-01-13T15:04:03 Error dhcp6c transmit failed: Can't assign requested address

And the gateway logfile:

Code Select
2022-01-13T14:50:50 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.0.0.1 bind_addr 178.27.64.56 identifier "VODAFONE_DHCP "
2022-01-13T14:50:50 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 91.9.129.136 identifier "TELEKOM_PPPOE "
2022-01-13T14:50:50 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 2606:4700:4700::1111 bind_addr 2003:ef:bfff:e1b:227c:14ff:fea0:644e identifier "TELEKOM_DHCP6 "
2022-01-13T14:50:49 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.0.0.1 bind_addr 178.27.64.56 identifier "VODAFONE_DHCP "
2022-01-13T14:50:49 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 91.9.129.136 identifier "TELEKOM_PPPOE "
2022-01-13T14:50:48 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.0.0.1 bind_addr 178.27.64.56 identifier "VODAFONE_DHCP "
2022-01-13T14:50:48 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 91.9.129.136 identifier "TELEKOM_PPPOE "
2022-01-13T14:50:48 Warning dpinger TELEKOM_DHCP6 2606:4700:4700::1111: sendto error: 65
2022-01-13T14:50:47 Warning dpinger TELEKOM_PPPOE 1.1.1.1: sendto error: 50
2022-01-13T14:50:47 Warning dpinger TELEKOM_DHCP6 2606:4700:4700::1111: sendto error: 50
2022-01-13T14:50:46 Warning dpinger TELEKOM_PPPOE 1.1.1.1: sendto error: 50
2022-01-13T14:50:46 Warning dpinger TELEKOM_DHCP6 2606:4700:4700::1111: sendto error: 50
2022-01-13T14:50:45 Warning dpinger TELEKOM_PPPOE 1.1.1.1: sendto error: 50
2022-01-13T14:49:09 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.0.0.1 bind_addr 178.27.64.56 identifier "VODAFONE_DHCP "
2022-01-13T14:49:09 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 91.9.129.208 identifier "TELEKOM_PPPOE "
2022-01-13T14:49:09 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 2606:4700:4700::1111 bind_addr 2003:ef:bfff:e2a:227c:14ff:fea0:644e identifier "TELEKOM_DHCP6 "
2022-01-13T14:48:42 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 91.9.129.208 identifier "TELEKOM_PPPOE "
2022-01-13T14:48:42 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 2606:4700:4700::1111 bind_addr 2003:ef:bfff:e2a:227c:14ff:fea0:644e identifier "TELEKOM_DHCP6 "
2022-01-13T14:47:59 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 91.9.129.208 identifier "TELEKOM_PPPOE "
2022-01-13T14:47:59 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 2606:4700:4700::1111 bind_addr 2003:ef:bfff:e2a:227c:14ff:fea0:644e identifier "TELEKOM_DHCP6 "
2022-01-13T14:45:14 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 91.9.129.208 identifier "TELEKOM_PPPOE "
2022-01-13T14:45:14 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 2606:4700:4700::1111 bind_addr 2003:ef:bfff:e2a:227c:14ff:fea0:644e identifier "TELEKOM_DHCP6 "
2022-01-13T14:43:58 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.0.0.1 bind_addr 178.27.64.56 identifier "VODAFONE_DHCP "
2022-01-13T14:43:58 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 91.9.129.208 identifier "TELEKOM_PPPOE "
2022-01-13T14:43:58 Warning dpinger send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 2606:4700:4700::1111 bind_addr 2003:ef:bfff:e2a:227c:14ff:fea0:644e identifier "TELEKOM_DHCP6 "

Since the switching still happen after disabling the monitoring, for me it seems to be not related to the monitoring itself.
On the other side, the system log don't look healthy...

VG;
Alex
January 13, 2022, 04:17:13 PM #5
That's still gateway monitoring code in action:

2022-01-13T15:04:07   Error   opnsense   /usr/local/etc/rc.newwanipv6: Choose to bind VODAFONE_DHCP on since we could not find a proper match.

Not sure why it doesn't return an address. What does ifconfig say to VODAFONE_DHCP interface? Any IP at all? Coming and going?


Cheers,
Franco
January 13, 2022, 04:20:22 PM #6
Quote from: franco on January 13, 2022, 04:17:13 PM
Not sure why it doesn't return an address. What does ifconfig say to VODAFONE_DHCP interface? Any IP at all? Coming and going?

We will see, I disabled the Vodafone interface to have a stable connection but will enable it again.
January 13, 2022, 04:21:31 PM #7
If your MAC address changed, make sure to reboot the Cable modem once.
January 13, 2022, 04:25:41 PM #8
Quote from: athurdent on January 13, 2022, 04:21:31 PM
If your MAC address changed, make sure to reboot the Cable modem once.

Thanks for the hint; but normally the NICs are not changing on the firewall side.

---

After enabling the vodafone interface again, the apply is running for ages (still loading currently), but the interface is up and have a IP.

General log:

Code Select
2022-01-13T16:21:11 Error opnsense /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway 'fe80::9ecc:83ff:fec8:789f%pppoe0'
2022-01-13T16:21:11 Error opnsense /usr/local/etc/rc.filter_configure: ROUTING: keeping current default gateway '62.155.242.170'
2022-01-13T16:21:11 Error opnsense /system_gateways.php: Adding static route for monitor 1.0.0.1 via 178.27.64.254
2022-01-13T16:21:11 Error opnsense /system_gateways.php: Removing static route for monitor 1.0.0.1 via 178.27.64.254
2022-01-13T16:21:11 Error opnsense /system_gateways.php: Adding static route for monitor 1.1.1.1 via 62.155.242.170
2022-01-13T16:21:11 Error opnsense /system_gateways.php: Removing static route for monitor 1.1.1.1 via 62.155.242.170
2022-01-13T16:21:11 Error opnsense /system_gateways.php: Choose to bind TELEKOM_PPPOE on 91.9.129.160 since we could not find a proper match.
2022-01-13T16:21:11 Error opnsense /system_gateways.php: Adding static route for monitor 2606:4700:4700::1111 via fe80::9ecc:83ff:fec8:789f%pppoe0
2022-01-13T16:21:11 Error opnsense /system_gateways.php: Removing static route for monitor 2606:4700:4700::1111 via fe80::9ecc:83ff:fec8:789f%pppoe0
2022-01-13T16:21:11 Error opnsense /system_gateways.php: ROUTING: keeping current default gateway 'fe80::9ecc:83ff:fec8:789f%pppoe0'
2022-01-13T16:21:11 Error opnsense /system_gateways.php: ROUTING: setting IPv6 default route to fe80::9ecc:83ff:fec8:789f
2022-01-13T16:21:11 Error opnsense /system_gateways.php: ROUTING: IPv6 default gateway set to opt2
2022-01-13T16:21:11 Error opnsense /system_gateways.php: ROUTING: keeping current default gateway '62.155.242.170'
2022-01-13T16:21:11 Error opnsense /system_gateways.php: ROUTING: setting IPv4 default route to 62.155.242.170
2022-01-13T16:21:11 Error opnsense /system_gateways.php: ROUTING: IPv4 default gateway set to opt2
2022-01-13T16:21:11 Error opnsense /system_gateways.php: ROUTING: entering configure using defaults

Ifconfig:

Code Select
$ ifconfig
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN
options=4e0272b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,LRO,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 20:7c:14:a0:64:4e
inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
inet6 fe80::227c:14ff:fea0:644e%igb0 prefixlen 64 scopeid 0x1
inet6 2003:ef:bf0e:2a00:227c:14ff:fea0:644e prefixlen 64
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 20:7c:14:a0:64:4f
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Vodafone
options=4e0272b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,LRO,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 20:7c:14:a0:64:50
inet 178.27.64.56 netmask 0xffffff00 broadcast 178.27.64.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: TelekomStatic
options=4e0272b<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,TSO4,TSO6,LRO,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 20:7c:14:a0:64:51
inet 192.168.100.5 netmask 0xffffff00 broadcast 192.168.100.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
groups: enc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
syncpeer: 0.0.0.0 maxupd: 128 defer: off
syncok: 1
groups: pfsync
pppoe0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1492
description: Telekom
inet6 fe80::227c:14ff:fea0:644e%pppoe0 prefixlen 64 scopeid 0x9
inet6 2003:ef:bfff:e3f:227c:14ff:fea0:644e prefixlen 64 autoconf
inet 91.9.129.160 --> 62.155.242.170 netmask 0xffffffff
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
January 14, 2022, 03:32:52 PM #9
OK, small update from my side:
I "solved" this issue for now with a workaround: I setuped the gateway groups for IPv4 (Telekom and Vodafone) and IPv6 (only Telekom) and send every traffic from the LAN net to the GW Groups.
For now it seems that this is working; the traffic goes mostly via Telekom.

Unfortunately, when checking the situation directly on the firewall, the traffic goes nearly the complete time via Vodafone.

One thing I want to add is, that the route table on the FW looks like this:

Code Select
$ netstat -4 -r -n
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            178.27.64.254      UGS        igb2
default            62.155.242.170     UGS      pppoe0
...

For me, with my understanding of routing etc, this routing table can be the cause that everything goes via Vodafone.
But I'm not a expert how this stuff should look like / work under FreeBSD / Opnsense.
January 17, 2022, 10:56:34 AM #10
Certainly odd it has both default routes. Do you have default gateway switching on or off?


Cheers,
Franco
January 17, 2022, 11:00:22 AM #11
PS: FreeBSD 12 and lower did not support multiple default entries. This could definitely end up being an issue for ordering reasons alone.
January 17, 2022, 11:25:15 AM #12
We're going to prevent that from happening for now using:

# sysctl net.route.multipath=0


Cheers,
Franco
January 17, 2022, 08:22:49 PM #13
Quote from: franco on January 17, 2022, 10:56:34 AM
Certainly odd it has both default routes. Do you have default gateway switching on or off?
Default gateway switching is active.

Quote from: franco on January 17, 2022, 11:25:15 AM
We're going to prevent that from happening for now using:

# sysctl net.route.multipath=0


Cheers,
Franco

I will test this in a minute; thank you franco.
January 17, 2022, 08:34:30 PM #14
Looks way better now:

Code Select

# sysctl -a | grep net.route.multipath
net.route.multipath: 0

Code Select

# netstat -4 -r -n
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            62.155.242.170     UGS      pppoe0
1.0.0.1            178.27.68.254      UGHS       igb2
1.1.1.1            62.155.242.170     UGHS     pppoe0
...
Print
Go Up Pages1 2
User actions