OPNsense Forum

Archive => 19.7 Legacy Series => Topic started by: hbc on October 17, 2019, 09:35:31 am

Title: [Solved] Strange gateway behaviour (question to devs)
Post by: hbc on October 17, 2019, 09:35:31 am
Hi!

I have running several networks, connected by various gateways. Some gateways resist in the same subnet, but each routes to other subnets.

Now I have e.g. gateway A (192.168.1.254) in subnet 192.168.1.0/24 used as destination in three routes:

As soon as I add gateway B (192.168.1.10) also located in subnet 192.168.1.0/24, it takes over all routes of gateway A. I did not even add the route for gateway B, but as soon as it gets added, my routing is damaged.

The strange thing is the routing table: netstat -rn is not changed. Gateway A (192.168.1.254) is still shown as gateway. But when doing a traceroute, gateway B is used.
I a next step, I added a linux pc with 192.168.1.100) as 2nd gateway and run tcpdump. And again, the 192.168.1.100 takes over all routes of gateway A and tcpdumps shows the routing traffic getting into the linux pc.

Is there a shadow routing tables that gets overwritten when adding a gateway in the same subnet as an existing one? For me netstat -rn was the only place for routes. Entries in this table are used, but now I have a situation where this routing table is not in sync with the effective used one.

Is this a FreeBSD bug or what commands are issued when adding/deleting/enabling/disabling a gateway?

Usually it should no problem with gateways in same subnet and a routing table like this:

Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 23, 2019, 02:37:40 pm
Rephrased my problem since no feedback yet. I need a solution, since I have to add this route and its gateway without overwriting existing routes.
Title: Re: Strange gateway behaviour (question to devs)
Post by: bootstrap on October 23, 2019, 03:11:38 pm
Hm. Your networks should probably be /24, not /8 (since you specified three octets).
Otherwise one 10.0.0.0/8 should do. Typo?
If you're really adding all networks as /8, you're probably breaking your routing right there.
At least if all your routes on all your devices always use /8...

Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 23, 2019, 04:07:23 pm
Right. Should be /24 in this example. I fixed it. The real networks use public ips and the masks and routing is correct and works ... as long as I do not add a second gateway.
Title: Re: Strange gateway behaviour (question to devs)
Post by: mimugmail on October 23, 2019, 09:31:37 pm
do you have upstream gateway set in WAN interface?
Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 24, 2019, 08:10:51 am
Right. WAN is the only interface with explicitly set gateways. All other interfaces have auto-detect.

But gateway A is not the default/upstream gateway for wan. It is an internal gateway on lan, neither ticked as upstream, nor explicitely assigned as upstream in any interface.
Title: Re: Strange gateway behaviour (question to devs)
Post by: mimugmail on October 24, 2019, 10:28:00 am
Sorry for repeating, again, do you have in ANY interface an upstream gateway or everywhere auto-detect?
With upstreamg gateway set you have some pf magic which can go unexpected, that's why I'm asking :)
Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 24, 2019, 04:21:32 pm
Yes, WAN interface has ipv4 and ipv6 gateway set.

How else does opnsense find my default gateway if not specified at least for wan?
Title: Re: Strange gateway behaviour (question to devs)
Post by: mimugmail on October 24, 2019, 05:32:36 pm
You can have a gateway and mark it as upstream in System : Gateways : Single, or you set upstream gateway in Interface : WAN. Please test/use first option
Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 25, 2019, 09:16:09 am
I tried both settings:

Same result: As soon as adding/activating gateway B, gateway A is overwritten.

But I recognized something about the (active) state.

When WAN gateway is marked as uplink, the text (active) is added behind the gateway name. And this text was only added to my WAN gateway (so I thought this is someting indicating active default gw). The other gateways - even activated and green online state - do not have this (active) text.

Now when I uncheck the uplink option in WAN, my lan gateway A gets this (active) tag - and as soon as I activate/add gateway B, gateway B gets this (active) tag.

What does this (active) behind a gateway name mean? I think this is the key to my problem, since when no gateway is marked as uplink, this (active) tag indicates the used lan gateway (A or B).

There is only one (active) gateway per protocol (IPv4 and IPv6). So I only have a maximum of two gateways with (active) tag. Even the routing tables is not altered, the the (active) LAN gateway gets all routes on Interface LAN_NET_2

WAN Gateways marked as upstream, GATEWAY_B disabled: (Routing as expected)
Code: [Select]
GW_WAN_IPv6 (active) WAN IPv6 255 (upstream) Online
GW_WAN_IPv4 (active) WAN IPv4 255 (upstream) Online
GATEWAY_B LAN_NET_2 IPv4 255 Pending
GW_LAN_IPv6 LAN_NET_2 IPv6 255 Online
GATEWAY_A LAN_NET_2 IPv4 255 Online


WAN Gateways used in WAN interface and no upstream option, GATEWAY_B disabled: (Routing as expected)
Code: [Select]
GW_WAN_IPv6 (active) WAN IPv6 255  Online
GW_WAN_IPv4 WAN IPv4 255  Online
GATEWAY_B LAN_NET_2 IPv4 255 Pending
GW_LAN_IPv6 LAN_NET_2 IPv6 255 Online
GATEWAY_A (active) LAN_NET_2 IPv4 255 Online

WAN Gateways used in WAN interface and no upstream option,  GATEWAY_B enabled: (all routes on interface LAN_NET_2 are taken by GATEWAY_B)
Code: [Select]
GW_WAN_IPv6 (active) WAN IPv6 255 Online
GW_WAN_IPv4 WAN IPv4 255 Online
GATEWAY_B (active) LAN_NET_2 IPv4 255 Online
GW_LAN_IPv6 LAN_NET_2 IPv6 255 Online
GATEWAY_A LAN_NET_2 IPv4 255 Online
Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 25, 2019, 09:21:06 am
[quoted instead of modified]
Title: Re: Strange gateway behaviour (question to devs)
Post by: tong2x on October 25, 2019, 09:35:00 am
I think the active means that it is the default route being used by OPNsense

if wanA is active vs wanB... connection will use wanA.

you can also decrease (lower is prioritized, I believe) the link priority for it to be "active"
Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 25, 2019, 10:24:54 am
I don't think so. Entries made in System:Gateways do not alter my routing table.

Not matter which gateway is shown (active), the default route is still the WAN gateway which is correct.

Altering priorities does not change anything. The most strange thing is the corrrect shown routing table, but the wrong routing done.

My routing tables show gateway A as used, but when activating gateway B, all A routes are routed via B.

So I want to find out what adding a gateway in System:Gatways actually does. For me it was just a place to define a gateway that can be used in interfaces and routes later.

But the definition of a gatway already changes something. I never thought that a pc would use other routes than defined in the kernel routing table.
Title: Re: Strange gateway behaviour (question to devs)
Post by: tong2x on October 25, 2019, 01:43:23 pm
in my case, I have wanA and wanB, whenever I change priorities or which ever is "active". that is the one being used as the Internet connection. but both my WAN have different subnets and IP though.
Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 28, 2019, 11:10:22 am
Well, actually it does not concern WAN nor internet connection. It is just an unpredictable behaviour when I add a new internal gateway which prevents me from using OPNsense as router.

On every other PC, L3 switch, router I can add additional gateways in one subnet without crashing the existing routing logic.
Title: Re: Strange gateway behaviour (question to devs)
Post by: AdSchellevis on October 28, 2019, 05:49:46 pm
TLDR: "Disable force gateway" unchecked in **Firewall ‣ Settings ‣ Advanced** and using the same interface?

(see https://docs.opnsense.org/manual/firewall.html)
Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 29, 2019, 08:20:08 am
AdSchellevis you are the best!

When reading the description for this option, I did not think it would be the reason for my problem since I do not use force gateway anywhere, but then I decided to inspect all my rules and found this autogenerated rule:

IPv4+6 *    lagg0    *    *    *    GW_LAN_IPv4    *    let out anything from firewall host itself (force gw)

And the rule was connected to your mentioned option. So I checked the option and voilà! - Works!
Title: Re: Strange gateway behaviour (question to devs)
Post by: AdSchellevis on October 29, 2019, 08:36:17 am
Which it shouldn't be if you don't want to force gateways....

Usually the outcome for these toggles can be viewed in the firewall rules ("Automatically generated rules" for either the interface or floating rules in case of the force gw toggle).
Title: Re: Strange gateway behaviour (question to devs)
Post by: hbc on October 29, 2019, 08:45:18 am
Yes, you are right. Meanwhile I found this autogenerated rule and modified my previous post.  ;)

So our posts where overlapping  :)