Multi-wan with PPPoE not working when get same gateway

[charles]

Hi,

I have 5 PPPoE lines from the same ISP.

After binding them to different interfaces on OPNsense and dialing each up separately, they obtain different IP addresses (all with 32-bit subnets) but the same gateway.

I configured unique monitor IPs for each gateway in the Gateway settings, and now the gateway status (including probe latency and packet loss) shows normal for all.

I also set up individual SNAT rules for each interface—with source/destination addresses set to "any", IPv4 protocol, and translation to the outgoing interface's IP.

However, when I create rules in Firewall -> Rules -> LAN and specify a gateway, only the rule pointing to the first PPPoE gateway works; the others fail to connect.

I've been using this exact setup on pfSense without issues for years. The key was just setting unique monitor IPs. But it seems this doesn't work on OPNsense?

Did I miss something crucial, or is this not supported on OPNsense? Are there any alternative workarounds?

Thanks!

[Monviech (Cedrik)]

https://github.com/opnsense/core/issues/8181#issuecomment-2571634803

https://github.com/opnsense/core/issues/5238#issuecomment-927822469

I doubt it works in pfsense if its unsupported by FreeBSD in general.

If not, whats the configuration magic for that? It would need multiple FIBs (aka virtual routing instances)

[Maurice]

PPP interfaces don't really need a gateway IP address - it's point-to-point, there's no ARP involved.

@Monviech Could we work around this by assigning random (or static) dummy IP addresses to PPP gateways?

@charles As a workaround, maybe creating static gateways with random IP addresses works?

Cheers
Maurice

[Monviech (Cedrik)]

I do not have much knowledge about this, I just wanted to reference these tickets that say its not possible for some reason.

What I assume is that dhclient installs interface routes like this

default via 10.0.0.1 dev pppoe0
default via 10.0.0.1 dev pppoe1
etc...

And these overwrite each other so last pppoe connection that comes up wins or something? I dunno though its all assumptions from my end.