Messages

Bumping this because 26.1.6 still have this bug and I am starting to see people on forums that have same problem like me.

Well, I'm not hiding. Your problem scope fits our support offering but clearly exceeds community support due to the lack of code-bound evidence and not many other people having the issue, which could also mean it is local to you and then it would only be solvable with local analysis which we don't do in community scope.


Cheers,
Franco

Hi there, just to clarify I have found out this bug really by luck. There is the problem if you have almost same speed dual WAN then it looks like its all working fine so nobody cares(many services dont mind assymetric routing). If I would not set up traffic shaper I would never found out that upstream traffic is going elsewhere. You know the bug is kinda hidden so maybe thats why nobody cares because their infrastructure "just works". I am really tring to sell this on forums and convince somebody to test more but I feel like nobody have time to test this even when I have provided reproduction steps...So sorry for my dissapointment this way.

Reply to is not working as intended and devs are playing hide-n-seek. Just look at my issue and how I spend time to reproduce it but nobody gives a sh*t here.

Related: https://github.com/opnsense/core/issues/9806

The two fail over options inside the gateway setup (of both the wan connections) are enabled or not?
Today I observed something similar to your desciption, after disabling them, the issue seems to be disappeared.

Sometimes I had to delete the state and sourcing tables in order for the system to consider the changes to configuration.

At first I had it enabled when I discovered this issue but aftewards I tried to disable it but it wont helped.

Can somebody test with my reproduce steps ?

• Set up some sort of cloud php service (in my case nextcloud AIO) with reverse proxy on WAN2
• Set up DNAT (manual rule) and firewall rule for WAN2 and WAN1 (I set up both because I have failover on OPNsense and Cloudflare DNS)
• Set up PC in remote location
• Prepare 1GB file on cloud
• Set up WAN1 as default gateway, WAN2 as secondary (priority WAN1 - 199, WAN2 - 254)
• Set up traffic shaper on WAN1 and choke it on 1Mbit/s using this - https://docs.opnsense.org/manual/how-tos/shaper_share_evenly.html
• Share link for 1GB file from cloud to remote PC and start download
• Now monitor download speed and choke and unchoke the WAN1 upload speed in shaper and see it does not working properly (upload/download traffic flows through WAN1 not WAN2 as expected)

Nslookup from remote PC to my cloud showing WAN2 tho...

Have you tried sticky connections with a long timeout?

Sticky connections have nothing to do with my problem they are used in load-balancing not in failover (what is my case). Specifically in failover mode you need to drop connections and reestablish them otherwise old connections will hang and wait for timout.

Why dont you sanitise your config.xml of passwords/secrets etc, and upload it to chatgpt and describe your issue.

I'm not an advocate of AI, but it has actually resolved an issue for me in the past with asymmetric routing, and inevitably, it was caused by me....and identified by AI.

I did but looks like it did not find any misconfiguartion issues there. So thats why I made issue on github. My scenario works in pfSense just fine, I suspect guys from OPNsense have bug in core because its behaving naturally wrong.

EDIT: Also there are more issues with routing...I have discovered that I cannot connect my samsung SmartView on first try but always on second try. This does not happen on pfSense and connection is established right on first try. But thats off topic I just wanted to say there are lot of holes here and nobody seems to care about plugging them.

The OPNSense docs state:

For legacy compatibility WAN interfaces set to type DHCP or interfaces with a Gateway Rules selection send reply packets to the corresponding gateway directly, also when the sender is on the same interface. This will break connectivity in some rare scenarios and can be disabled via Firewall->Settings->Advanced->Disable reply-to.

With Multi-WAN you generally want to ensure traffic leaves the same interface it arrives on, hence reply-to is added automatically by default. When using bridging, you must disable this behavior if the WAN gateway IP is different from the gateway IP of the hosts behind the bridged interface.

In my case, I have "Disable reply-to on WAN interface" selected, and my firewall rules have the reply-to explicitly set.
My secondary WAN is DHCP, and my primary is PPPoE, so this felt safest.

That works fine.

EDIT: I should add, I have migrated to the NEW rules....

In this scenario it does not work - https://github.com/opnsense/core/issues/9806#issuecomment-4194715800

EDIT: Also to mention I am using old rules, but nonethless it should work on both because this is core function. Funny is that on pFSense it works correctly and this is project is fork of it...

I can tell you it works fine on 26.1.5 so you must have something misconfigured.

You havent really given us enough information.

Have you checked "Disable Reply-To on WAN rules" on Firewall/Settings/Advanced?
Have you set the "Reply-To" on the actual firewall rule? (advanced mode)

Wait what ??? Disable reply-to and using manual reply-to for each rule ? This is completely nonsense isnt it ?
Reply-to is set as default on all rules(written in docs).
Also I have already tried to setting theese options as you suggested but none of it worked correctly. It behave still the same (wrong)

Hello folks

coming from pfSense and after few hours of happiness to see more advanced development here I struggle to set up symetric routing correctly with current version of OPNsense.

Here is ticket on github - https://github.com/opnsense/core/issues/9806

In short:
I have multiwan setup at home. I am hosting few services on WAN1 and few on WAN2. Everything looks like normal but yesterday I have tested downloading from my cloud from remote location which is hosted on WAN2 and I have wondered why is downloading so slow when I have stronger upload on WAN2 than WAN1. I have found out  that handshake to my cloud is made correctly via WAN2 BUT download stream will start on WAN1.

What I have tested to fix this:
1. Forced rules reply-to WAN2
2. Disable forced gateway in settings
3. Check Bind states to interface
4. Gateway policy routing

Nothing worked.

Everytime OPNsense honors default gateway no matter what you set in rules.

I am curious why this worked out of box on pfSense right after setting up rules and port forward and here it does not work (even after hardsetting rules and state behavior).

Can somebody help me to test this out and maybe make some logs for devs ? This needs to be fixed if its broken here. I really wish to stay here because of more options but this keeping me back.

Thanks in forward.