18.7.8 traffic for local interfaces routes out gateway instead

[The_Sage]

Here is my issue.
I have WAN and LAN working as normal.
WAN is PPPoE, LAN is 192.168.0.0/24. 4G is 192.168.15.1.
I have 4G as a multi WAN fail over. I have been checked the settings over and over from a system that works, and this one. (Also from OPNSense Wiki Multi WAN doc.)
When WAN goes down, DNS works, as the firewall is the DNS server, but there seems to be no routing of traffic on the LAN network through the 4G network.
The problem seems to be that from the LAN interface, a PC cannot PING the 4G interface,
ping 192.168.15.1 - Request Timed Out
tracert 192.168.15.1 -> out the PPPoE gateway ??
instead the packets go out the "default" gateway. From the firewall itself, I can ping from LAN (firewall IP) to 4G, but NOT from LAN network.

So from the firewalls perspective, the Failover works. But the PC's on the LAN network do not work in a fail over situation.

Can anyone shed any light?

P.S.
I have numerous firewalls setup like this that work. The settings are (seem) to be the same.

[The_Sage]

P.S. I am new to this forums posting.

What more information is required to help out?

[mimugmail]

Screenshots of rules, outbound nat, gateways and gateway groups please :)

[The_Sage]

Thank you.

Here are the requested Files
FW-4G is the Back up / fail over network. It is double natted as can been seen by the IP address. Block Bogan and Private networks are OFF for the interface. Firewall can Ping external addresses.

FW-LAN shows that DNS requests only allowed from firewall, which DHCP sets the DNS server to this IP. Clients can resolve the IP using firewall as DNS, but no reply. Also Clients cant PING the 192.168.15.1 interface. Routing ??

FW-WAN - no extra comments

More Pics next post.

[The_Sage]

... More screen shots

No extra comments for these.

Thanks in advance.

The Sage

[mimugmail]

1. Remove the rule on 4G interface
2. On LAN tab you have ICMP any any to gateway group, then you cant ping firewall.
Just clone the dns rule above for lan to firewall port 53 and make it icmp. Be sure it's above the gateway rule, then it works.

[The_Sage]

Thanks, after looking at the setup after applying your changes makes so much sense. (You know the forest and the trees)

Thanks for your time. I will post back when I have tested.

The Sage

[The_Sage]

I have made the changes recommended, most notibly the 4G rule.
To simplify things, I just disabled the LAN rules and created a ALLOW ALL rule using the failOver Gateway group (as per the wiki). All works with the PPPoE connection, but still NOT working when WAN cable pulled so we SHOULD? get fail over to 4G as per the Gateway group.

Here is what I have found.
If I set the 4G as default gateway, all works good from LAN devices.
Set failOver as Gateway, 4G does NOT work, although from the firewall, PING works for 4G gateway.

Where else can I look for the issue.

If I go to the LIVE firewall rules and filter by client IP address, all I see is successful UDP Ping requests. There is no DROPPED packets (except for some ports for in house software).

[mimugmail]

Firewall : Settings : Advanced .. Default Gateway switching enabled?

[The_Sage]

Yes it is.

I will go over the settings on my test box to see if there is anything different.

Is there any log files or config files to look at? 

[mimugmail]

system.log and routing.log ...

[The_Sage]

Finally go this too work. All settings are the same (except for the 4G rule). All I changed was the DNS rule for the firewall rule to use the FailOver Gateway instead of the Default gateway (defined in the MultiWAN WiKi).

Then it started to work.

This is extremely weird as I have other systems working just fine.

As a side note, I have a test box that now wont even route at all using the default settings of OPNSense. I seems like changes in the WAN Address, from Static to PPPoE or Static yo DHCP causes these issues. Then after checking and recheck, rebooting etc, it finally works. The first time it is setup with static WAN IP fail over works.

Am I on drugs or does this happen to others?

The Sage

[The_Sage]

<SOLVED> NOT on drugs, I just looked through the General logs, and a Firewall rule was failing due to it being associated with another interface. I removed the interface (and thus cleared the rule) and it is working now.