Multi WAN setup

[Wibbling]

Apologies for my tone - I'm struggling.

I've an opnsense box with 4 interfaces. opt0 is LAN, opt1 is my fibre line and opt2 is my 4g modem.

The fibre line has had a couple of outages at their end. Duly I've then connected and configured the 4g modem. Both fibre and 4g (an LM 1200 in bridge mode, for the interested) are simple DHCP ethernet.

I disabled the fibre gateway to send traffic out through the 4g and had internet access.

Later, from the shell I tried ping -S [fibre-ip] 4.2.2.4 and saw the ping return.
 
Setting up a monitor IP for the fibre connection also returns 0 ping failures.

I set about disabling the 4g gateway and enabling the fibre gateway and....

Nothing. No traffic moved, no ping responses despite the fibre gateway showing green in Gateways/Configuration.

Plugging the fibre ISPs router in provided access over the fibre connection, but not through opnsense.

I then reverted, disabling the fibre gateway and enabling the 4g gateway ... and... nothing. No web access, no ping responses, nothing.

I tried rebooting (which always seems a last resort) and no change. Despite the 4g gateway merrily reporting green and working (evidenced by the ISP router), no traffic moved.

To get opnsense to send traffic again I deleted the 4g interface and re-created it, using a timestamp for a name. I honestly don't know why I needed to do this but it did then start forwarding packets.

I would be grateful if someone would kindly explain what I am doing wrong, as - in my mind - this should be straightforward: gateway connection A drops off, switch to B. To see if A has come back, disable B, re-enable A. Rinse, repeat as necessary.

 I'm not trying to set up high availability or failover. I don't want or ned these. I just want to have both connections available and disable one connection while the other is working.

I am fully aware there's a configuration/service restart I'm missing. Would someone kindly tell me where I am going wrong?

Kind regards, Wibbling

[Grossartig]

I am using the Netgear LM1200 in another home, and I configured it to be the decision maker to which WAN to use (SIM vs. wired). You could try to set it up that way and then only have a single WAN configured in OPNsense, which connects to the LM1200, which itself has dual-WAN.

I'm sorry if I'm not able to answer/address your actual questions, but my suggestion would be for letting the LM1200 handle multi-WAN and use OPNsense in single-WAN mode.

Screenshot: https://imgur.com/a/WvVz3mp

[Wibbling]

Hello, thank you for your reply  - I'm not using the LM1200's multi wan options - these are not in use and disabled. I'm just using it as a separate interface/gateway within opnsense.

[Grossartig]

I know -- I'm saying you could use the LM1200 for Multi-WAN instead. You have to also be on the latest firmware for the LM1200 for that to work.

[Thomas Niedermeier]

What I can say is that if you stick to this article it works fine: https://docs.opnsense.org/manual/how-tos/multiwan.html
We also did our own tests and created a wiki article: https://www.thomas-krenn.com/de/wiki/OPNsense_Multi_WAN
I hope that helps.

Best regards,
Thomas

[Wibbling]

Many thanks for your reply Thomas but I have specifically said I do not want to use failover or load balancing.

What I want to do is have both connections configured, with one disabled. Should that fail, enable the other - manually. Not using fail over. I don't need that level of complexity. I simply want to disable one interface and enable another by enabling/disabling the gateway by clicking on it. Surely this is possible?

Why did traffic not flow despite the connection being available? Why did I have to delete the interface completely and re-add it for traffic to flow?

[Thomas Niedermeier]

Oh okay sorry for that, I maybe got it wrong while reading your text initially.
Hmm yes mobile connections are always a bit buggy, because a LTE Modem (for example Quectel EM12-G) is kind of a computer on its own.

I recently tested it again, if you dare to unplug the SIM and insert it afterwards you might guess it would come up again and reconnect... no way you have to unplug and de-energize the whole machine and then boot it up again to get it working again.

The fact that you have to delete it completely... I have no clue...

[Wibbling]

No worries Thomas, I'm sort of trying to do failover but manually!

The modem is fine in itself. I can move that around all day long. The problem is opnsense.

When I  lose gateway A I want to manually enable gateway B.

A different way to explain it:

Given I've the fibre gateway enabled
and I've the 4g gateway disabled
and fibre disconnects/drops packets
when I manually disable the fibre gateway
and manually enable the 4g gateway
Then I want traffic to pass over the 4g gateway.

And it doesn't.

You can re-create this config in a VM (I've tried) with two ethernet interfaces to different networks. It being fibre and 4g modem isn't relevant.

The above process works (forwards traffic) only by deleting and re-creating the interfaces.

Therefore I surmised that a service/routing twiddle needed to be restarted to say 'Oi! Your gateway has changed. Send traffic over the active one!'

[Wibbling]

A minor note here. I created my setup entirely in VM (diagram attached) and fiddled about in that, running traceroutes and browsing, updating the VMs and then went through my settings almost page by page.

I noticed that the Outbound NAT rules were set to automatic and none existed for my 'real world' configuration.
Setting the option to 'Hybrid' I created manual rules copied from the VM for my two WAN interfaces.

Traffic is now moving without issue and I can disable/enable my gateways as I wanted to.

Is this why deleting and re-creating an interface was allowing traffic to flow - as the NAT outbound rules were re-created when the new interface was created?

I've read back and this automatic creation of outbound NAT rules has occurred before with others and has been addressed in this older post: https://github.com/opnsense/core/issues/2914#issuecomment-439904741.

What might have prevented the automatic rules from being created?