Help with Policy Based Routing and Multi WAN Setup

[DeltChar]

Greetings everyone,

I have searched the forums for an answer and tried quite a few of the suggestions.  Specifically this post:  https://forum.opnsense.org/index.php?topic=10929.0 and a few others.  I am well aware that I am probably missing something obvious, but I cannot fathom what I am overlooking.  I have two WAN connections (Starlink and Cable) and I also host some of my family's services (Plex, Bitwarden, etc).  Hosting does not work on Starlink due to CG-NAT, so I've kept my unreliable Cable Connection and would like to route my services through the Cable Connection and all other traffic through Starlink.  I have both WANs working and tried to setup Policy Based Routing according to the above link and a few others.  However, no matter how I set the connections up, all services still try to use the default route through Starlink (or don't work at all, misconfiguration and all that) or through the Cable connection when Packet Loss marks Starlink as "down."

It appears that I am limited in the number of image files that can be posted to a single message.  So, I'll post the LAN and WAN Firewall rules I am using to this post.  I can post with additional screenshots of anything requested in further posts.  I am a Linux Administrator with basic knowledge of networking.  If there is something glaringly obvious that I am missing, please, let me know.


I'm well aware of the security risks involved with opening ports in my network.  I did try to setup a Wireguard Tunnel from a VPS back into my network specifically for these services.  However, I was not ever able to get that setup properly either.  In theory, with the WG Tunnel, I could host all of my services through Starlink without issue as well.  But, at this point I'll just settle for things to work again, whichever route is easier to make work.

Things I've tried:

• WAN Group with Failover and Firewall Rules

• Failover worked like a charm, but services refused to operate on anything but primary connection
• Tested Service connections by using `curl icanhazip.com` in Terminal on Machine running services.  IP always came back as default route
• Tried different Tier combinations with mostly the same results.  If on the same Tier (load balancing), services lucky enough to to get assigned the Cable Connection worked for a time, until balanced back over to Starlink

[/list]

• Dual WAN setup without any WAN group and various Firewall Rules

• Similar results as the above attempt

[/list]

• Wireguard tunnel to a VPS running nginx to route different URLs to different Services

• Was never able to get the Wireguard Tunnel Handshake between OPNsense and VPS to connect.  Likely ignorance on my part as the above problems as well.

[/list]


Regards to everyone.

[Alphakilo]

Hey DeltChar!

I think we might be experiencing the same problem.
Try tcpdumping both WAN interfaces to check, whether the responses are actually sent on the interface the packets where received...