Multiwan

[elektroinside]

Just wanted to let you know it works great for me - so far (dual wan, both dhcp, to be precise).
OPNsense 18.1.4

[mimugmail]

Yep, with 18.1.4 most combinations are possible now  8)
https://github.com/opnsense/core/issues/1821#issuecomment-370360822

Next big thing is the problem with Traffic Shaper on multiwan  ::)

[elektroinside]

Wow, now that's some serious testing, very well done! Thank you!

I'm not using traffic shaping, but good to know bugfixes are scheduled. Fortunately, the internet is so cheap here, even for businesses, and in most cases very good and plenty of bandwidth, that i prefer a second line for exclusive/dedicated stuff.

[AndyX90]

Nice! My only Problem with Multiwan is that you cannot automatically balance the outgoing traffic from the firewall itself(Squid traffic for example).
I tried a floating rule for outgoing traffic with destination Multiwan and it doesn't work. In this case you need to manually change the default gateway or am i wrong?

[elektroinside]

I'm not using load balancing either (yet), just failover, and that works for me. So unfortunately i cannot be much of a help at this time...

[mimugmail]

Nice! My only Problem with Multiwan is that you cannot automatically balance the outgoing traffic from the firewall itself(Squid traffic for example).
I tried a floating rule for outgoing traffic with destination Multiwan and it doesn't work. In this case you need to manually change the default gateway or am i wrong?

Afaik this doesnt work since the Service (Squid) is bound to an interface

[elektroinside]

Ok, so after a few hours of testing, it's not working that good anymore.
So I have my first OPNsense box (box1). Among its physical interfaces, it has a LAN with a subnet with DHCP enabled, and another LAN (for an AP) with another subnet and DHCP enabled.

Built another OPNsense (box2) with 2 WANs for failover (as per https://wiki.opnsense.org/manual/how-tos/multiwan.html). The first WAN will get its IP/GW from Box1Lan1 DHCP, the second WAN will get an IP/GW from Box1Lan2 DHCP via a wireless client.

Let's say:
WAN1 GW: 192.168.1.1 (monitor IP: 8.8.8.8 )
WAN2 GW: 192.168.2.1 (monitor IP: 8.8.4.4 )

1. Physically disconnecting the network cables/wifi client on Box2 (the 2 WANs) works great (so far); tried numerous cases

2. Restarting Box1 will temporarily bring down both LAN interfaces of course; Box2 will sense this, apinger marks both its GWs as down; once the restart of Box1 completes, apinger will sometimes only sense one of the GWs as online again. The other one will remain offline in Box2, even though it is actually online. If I do not restart apinger, it will never recover. Well I didn't wait for never to end, I just waited 15 minutes.

3. Same restart as #2, only this time is worse. Sometimes both GWs will be reported as offline, even though they are of course, online. Same thing, apinger needs to be restarted for this to get fixed and the two GWs to be reported as online.

4. If I bring down the single WAN interface of Box1, on Box2 the monitor IPs will obviously fail. The GWs are up, but the monitor IPs are unreachable. Same thing, most of the times, even though i bring back WAN on Box1, the GWs on Box2 will be reported as offline.

5. LAN clients connected to Box2 will have a working internet connection once the GWs are really up, no matter what apinger reports.

Unfortunately, I don't have time to wait for a fix (if it's a bug). How can i restart apinger from the shell?

"service apinger onerestart" does not work, it will just stop and never start again...

[mimugmail]

Do you have all the options ticked/unticked like in the xls from the github issue?
I never tested both WANs link since this is a really rare case.

In all the tests I had one time an issue where I had zu restart apinger, but after a reboot it wasn't reproduceable anymore.

[elektroinside]

Yep, exactly as in the xls, except that I'm using only one DNS server/WAN (an internal one).
And.. my IPv6 never coming back without a restart on my Box1 PPPoE WAN interface issue is back :( But I have to disconnect a few times to reproduce it.

I'm actually using a LAN client (of Box2) to write this reply with both GWs reported as offline...

[elektroinside]

Logs:

Code Select Expand


Mar 14 18:41:01 apinger: rrdtool respawning too fast, waiting 300s.
Mar 14 18:41:01 apinger: Error while feeding rrdtool: Broken pipe
Mar 14 18:36:01 apinger: rrdtool respawning too fast, waiting 300s.
Mar 14 18:36:01 apinger: Error while feeding rrdtool: Broken pipe
Mar 14 18:31:01 apinger: rrdtool respawning too fast, waiting 300s.
Mar 14 18:31:01 apinger: Error while feeding rrdtool: Broken pipe
Mar 14 18:26:01 apinger: rrdtool respawning too fast, waiting 300s.
Mar 14 18:26:01 apinger: Error while feeding rrdtool: Broken pipe
Mar 14 18:25:48 apinger: ALARM: WAN2_DHCP(8.8.4.4) *** down ***
Mar 14 18:25:48 apinger: ALARM: WAN_DHCP(8.8.8.8) *** down ***


[franco]

Can you try this:

Go to System: Gateways: Single and edit all gateways (save without modification is enough) and then hit apply.


Cheers,
Franco

[elektroinside]

Yeah, that works, I tried it as I found this already in another post. Taught it was fixed with the latest commits.
Anyway, tried something else, I let those GWs as they were, offline. Came back after 10 minutes or so... still offline. Came back after 30 mins or so... got back online.

Is this cosmetic or functionality is also affected (at least indirectly)?

@Franco: I am going to start working on the install

[elektroinside]

I found this on the screen

[franco]

That's expected. Apinger stops crashing because it is being prevented by SEGVGUARD feature to prevent brute forcing exploits.

So it's still crashing? I'd really really like to inspect such a system to see what's going on or easy steps to reproduce.


Cheers,
Franco

[elektroinside]

Skype? I'm online :)