Multi Wan works, but with poor performance

[gsellc]

I'm setting up a Soekris Net6501 with 18.1 to replace an identical piece of hardware running OpenBSD 6.0 or thereabouts. Installation and basic configuration went fine. The device sits on a network with 3 VLANs (plus the unused native VLAN 1 - designated "LAN" on this box) and 3 "WAN" connections. The WAN connections are consumer grade DSL and use a routing modem, so on the OPNSense router I have interfaces as such:

Code: [Select]

em0:
        inet 172.40.1.1 netmask 0xffffff00 broadcast 172.40.1.255
em1:
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
em2:
        inet 192.168.2.2 netmask 0xffffff00 broadcast 192.168.2.255
em3:
        inet 192.168.3.2 netmask 0xffffff00 broadcast 192.168.3.255
em0_vlan10:
        inet 172.40.10.1 netmask 0xffffff00 broadcast 172.40.10.255
        vlan: 10 vlanpcp: 0 parent interface: em0
em0_vlan20:
        inet 172.40.20.1 netmask 0xffffff00 broadcast 172.40.20.255
        vlan: 20 vlanpcp: 0 parent interface: em0
em0_vlan30:
        inet 172.40.30.1 netmask 0xffffff00 broadcast 172.40.30.255
        vlan: 30 vlanpcp: 0 parent interface: em0
Hopefully self explanatory.

The situation is that I followed the multi-wan instructions:

Docs » User Manual » How to’s » Setup Multi WAN

And the setup all seemed to be very straightforward, everything is working as advertised, but performance is abysmal. Often connections fail entirely and when they don't fail they react VERY VERY slowly. It feels a lot like an MTU issue on a PPPoE connection, however there is no PPPoE and the MTU was 1500 on all interfaces on the router this one is replacing with no MSS clamping or other such configs in place.

If I modify my PBRs on the individual VLAN firewall allow rules to use a specific default gateway instead of using the gateway group traffic immediately flows normally. In my mind this eliminates the thought that the problem could be:

NAT related
MTU related
DNS related (actually this seems to be working fine since it's proxied)
Uplink related (tested all 3)

At this point there is no VPN configured, no IPS/IDS, nothing else fancy. I'm not sure what else to look at to troubleshoot this further.

Love the product and look forward to making it work for me in this configuration. Thanks in advance.

[mimugmail]

For me it seems you missed the step regarding DNS in multiwan doc

[gsellc]

No, I don't think that's the problem. This is what I have:

[mimugmail]

What DNS Server do the clients use?

[gsellc]

"system default" - see attachment.

I don't feel like this is a DNS issue. It's very slow to either load, and/or fail. DNS issues in my experience return name resolution errors pretty darn quick.

[mimugmail]

I mean the Computer behind OPN? Just post ipconfig

[gsellc]

Code: [Select]

Wireless LAN adapter Wireless Network Connection:

   Connection-specific DNS Suffix  . : steese.local
   Link-local IPv6 Address . . . . . : fe80::849f:c9a1:ce72:b847%13
   IPv4 Address. . . . . . . . . . . : 172.40.30.76
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 172.40.30.1

Sorry if I wasn't clear in my original posting - the unbound forwarder service is running, I think that's the default right? In any case, DNS resolution is instant even when I am using the multiwan gateway. I don't think the problem is at all DNS related.

[mimugmail]

Again, when you have a multi wan firewall rule for LAN to ANY with gateway GATEWAYGROUP, like multi wan is setup, also your DNS requesdt from the clients to the firewall itself are routed via multi wan which is wrong.

So when you LAN client uses the Unbound of the firewall, you MUST create a rule matching infront without the gateway target.

Please check this first!

https://docs.opnsense.org/manual/how-tos/multiwan.html#step-5-add-allow-rule-for-dns-traffic



Why is this important? When a client only has one DNS server configured (the firewall), internet connectivity doesnt work at all. When a client uses firewall first and e.g. 8.8.8.8 as second DNS, it takes some time to get resolved which would explain your scenario with slow internet or no internet.

[gsellc]

Sorry for the delayed reply...

DNS really I don't think is the problem. Resolution for any domain is consistent and very fast from client PCs, even in the MultiWan setup. I understand the need for the local route, and I did indeed follow the instructions in the wiki, see attachment.

"internalIF" is a firewall group consisting of all my internal "VLAN" layer 3 interfaces.


Thanks again for your attention to the issue.

[namezero111111]

Do you have Sticky NAT under Firewall / Advanced enabled?

[gsellc]

I do - without sticky in the old OpenBSD configuration bank sites and the like blew up.

[namezero111111]

Can you try without for testing? I am troubleshooting a sticky nat issue where packets leave the  interface with an incorrect source ip.
Also, try with Sticky on but "Shared forwarding" disabled.

See https://forum.opnsense.org/index.php?topic=7803.msg35945#msg35945

Maybe you're running into something similar? So far i couldn't find a misconfiguration: still troubleshooting

Please report the results!

[gsellc]

Turning off sticky fixes the problem.

[namezero111111]

Ok, i recommend to workaround this with "poor mans sticky" by using two failover groups for different ip ranges.
Possibly chime in on https://github.com/opnsense/core/issues/2376 and try with sticky on but shared forwarding disabled.
Have you tried that combination?