OPNsense Forum

Archive => 18.1 Legacy Series => Topic started by: gsellc on April 09, 2018, 11:29:00 pm

Title: Multi Wan works, but with poor performance
Post by: gsellc on April 09, 2018, 11:29:00 pm

I'm setting up a Soekris Net6501 with 18.1 to replace an identical piece of hardware running OpenBSD 6.0 or thereabouts. Installation and basic configuration went fine. The device sits on a network with 3 VLANs (plus the unused native VLAN 1 - designated "LAN" on this box) and 3 "WAN" connections. The WAN connections are consumer grade DSL and use a routing modem, so on the OPNSense router I have interfaces as such:

Code: [Select]

em0:
        inet 172.40.1.1 netmask 0xffffff00 broadcast 172.40.1.255
em1:
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
em2:
        inet 192.168.2.2 netmask 0xffffff00 broadcast 192.168.2.255
em3:
        inet 192.168.3.2 netmask 0xffffff00 broadcast 192.168.3.255
em0_vlan10:
        inet 172.40.10.1 netmask 0xffffff00 broadcast 172.40.10.255
        vlan: 10 vlanpcp: 0 parent interface: em0
em0_vlan20:
        inet 172.40.20.1 netmask 0xffffff00 broadcast 172.40.20.255
        vlan: 20 vlanpcp: 0 parent interface: em0
em0_vlan30:
        inet 172.40.30.1 netmask 0xffffff00 broadcast 172.40.30.255
        vlan: 30 vlanpcp: 0 parent interface: em0
Hopefully self explanatory.

The situation is that I followed the multi-wan instructions:

Docs » User Manual » How to’s » Setup Multi WAN

And the setup all seemed to be very straightforward, everything is working as advertised, but performance is abysmal. Often connections fail entirely and when they don't fail they react VERY VERY slowly. It feels a lot like an MTU issue on a PPPoE connection, however there is no PPPoE and the MTU was 1500 on all interfaces on the router this one is replacing with no MSS clamping or other such configs in place.

If I modify my PBRs on the individual VLAN firewall allow rules to use a specific default gateway instead of using the gateway group traffic immediately flows normally. In my mind this eliminates the thought that the problem could be:

NAT related
MTU related
DNS related (actually this seems to be working fine since it's proxied)
Uplink related (tested all 3)

At this point there is no VPN configured, no IPS/IDS, nothing else fancy. I'm not sure what else to look at to troubleshoot this further.

Love the product and look forward to making it work for me in this configuration. Thanks in advance.

Title: Re: Multi Wan works, but with poor performance
Post by: mimugmail on April 10, 2018, 05:39:40 am

For me it seems you missed the step regarding DNS in multiwan doc

Title: Re: Multi Wan works, but with poor performance
Post by: gsellc on April 10, 2018, 11:03:12 am

No, I don't think that's the problem. This is what I have:

Title: Re: Multi Wan works, but with poor performance
Post by: mimugmail on April 10, 2018, 11:55:57 am

What DNS Server do the clients use?

Title: Re: Multi Wan works, but with poor performance
Post by: gsellc on April 11, 2018, 05:21:18 am

"system default" - see attachment.

I don't feel like this is a DNS issue. It's very slow to either load, and/or fail. DNS issues in my experience return name resolution errors pretty darn quick.

Title: Re: Multi Wan works, but with poor performance
Post by: mimugmail on April 11, 2018, 06:06:18 am

I mean the Computer behind OPN? Just post ipconfig

Title: Re: Multi Wan works, but with poor performance
Post by: gsellc on April 12, 2018, 06:19:30 am

Code: [Select]

Wireless LAN adapter Wireless Network Connection:

   Connection-specific DNS Suffix  . : steese.local
   Link-local IPv6 Address . . . . . : fe80::849f:c9a1:ce72:b847%13
   IPv4 Address. . . . . . . . . . . : 172.40.30.76
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 172.40.30.1

Sorry if I wasn't clear in my original posting - the unbound forwarder service is running, I think that's the default right? In any case, DNS resolution is instant even when I am using the multiwan gateway. I don't think the problem is at all DNS related.

Title: Re: Multi Wan works, but with poor performance
Post by: mimugmail on April 12, 2018, 07:24:59 am

Again, when you have a multi wan firewall rule for LAN to ANY with gateway GATEWAYGROUP, like multi wan is setup, also your DNS requesdt from the clients to the firewall itself are routed via multi wan which is wrong.

So when you LAN client uses the Unbound of the firewall, you MUST create a rule matching infront without the gateway target.

Please check this first!

https://docs.opnsense.org/manual/how-tos/multiwan.html#step-5-add-allow-rule-for-dns-traffic



Why is this important? When a client only has one DNS server configured (the firewall), internet connectivity doesnt work at all. When a client uses firewall first and e.g. 8.8.8.8 as second DNS, it takes some time to get resolved which would explain your scenario with slow internet or no internet.

Title: Re: Multi Wan works, but with poor performance
Post by: gsellc on April 22, 2018, 07:33:06 pm

Sorry for the delayed reply...

DNS really I don't think is the problem. Resolution for any domain is consistent and very fast from client PCs, even in the MultiWan setup. I understand the need for the local route, and I did indeed follow the instructions in the wiki, see attachment.

"internalIF" is a firewall group consisting of all my internal "VLAN" layer 3 interfaces.


Thanks again for your attention to the issue.

Title: Re: Multi Wan works, but with poor performance
Post by: namezero111111 on April 23, 2018, 09:37:48 pm

Do you have Sticky NAT under Firewall / Advanced enabled?

Title: Re: Multi Wan works, but with poor performance
Post by: gsellc on April 25, 2018, 04:40:31 am

I do - without sticky in the old OpenBSD configuration bank sites and the like blew up.

Title: Re: Multi Wan works, but with poor performance
Post by: namezero111111 on April 25, 2018, 07:07:19 am

Can you try without for testing? I am troubleshooting a sticky nat issue where packets leave the  interface with an incorrect source ip.
Also, try with Sticky on but "Shared forwarding" disabled.

See https://forum.opnsense.org/index.php?topic=7803.msg35945#msg35945

Maybe you're running into something similar? So far i couldn't find a misconfiguration: still troubleshooting

Please report the results!

Title: Re: Multi Wan works, but with poor performance
Post by: gsellc on April 30, 2018, 10:45:23 pm

Turning off sticky fixes the problem.

Title: Re: Multi Wan works, but with poor performance
Post by: namezero111111 on April 30, 2018, 11:15:16 pm

Ok, i recommend to workaround this with "poor mans sticky" by using two failover groups for different ip ranges.
Possibly chime in on https://github.com/opnsense/core/issues/2376 and try with sticky on but shared forwarding disabled.
Have you tried that combination?