Interfaces: Virtual IPs: Settings > Mode=Other

[lmoore]

This is my setup; I wanted to place OPNSense between 2 routers to do what it is supposed to do: SHAPE TRAFFIC.

Would it be better to say you have a transparent firewall with traffic shaping?

What purpose do the VIP's serve in your configuration?

[notspam]

The truth is that he only want a "layer 2 only / traffic shaping box".

[lmoore]

The truth is that he only want a "layer 2 only / traffic shaping box".

Yes, I understand that. The configuration has VIP's assigned to LAN A1, LAN A2, WAN A1 & WAN A2, the purpose of which is unclear.

[dseven]

The truth is that he only want a "layer 2 only / traffic shaping box".

In my interpretation; they want a traffic shaping box, but they were unable to get (layer 3) routing to work they way that they expected it to, so they cobbled something together with a transparent bridge instead...

[MK4L]

The truth is that he only want a "layer 2 only / traffic shaping box".

In my interpretation; they want a traffic shaping box, but they were unable to get (layer 3) routing to work they way that they expected it to, so they cobbled something together with a transparent bridge instead...

That was exactly the case, 10 years ago!
For the question about what the Other mode VIPs are doing: they allow traffic to be routed *without any static routes*. The system does not even generate dynamic routes for the Other mode VIPs, but routing works exactly like I expect it to.

Not sure what I did back then, but I have just completed the testing *now* and it worked!
No VIPs, only static routing and 2 manual Gateways.


Thank you all so much for your interest to help.

[lmoore]

Other mode VIPs are doing: they allow traffic to be routed *without any static routes*.

I wonder if the routing issue back then was to do with ARP.

Glad it's all working for you!

[MK4L]

The truth is that he only want a "layer 2 only / traffic shaping box".

In my interpretation; they want a traffic shaping box, but they were unable to get (layer 3) routing to work they way that they expected it to, so they cobbled something together with a transparent bridge instead...

I am back to square 1. I recalled all the pain of the past.
I know why I chose to make it transparent!
When you start assigning IP addresses to WAN/LAN interfaces in the first place, 2 ugly things happen:

- UI navigation becomes very much slower and lagging.
- Routing decisions must be taken *on* OPNSense. Routing becomes horribly unreliable.

Like, I only rebooted, and I lost access to the console from OPT1, which I setup for admin access.

This is my test setup...

Interfaces...
LAN Interface: 192.168.10.2/24
WAN Interface: 192.168.11.2/24
OPT1: DHCP IP 172.16.100.6/24, GW 172.16.100.1
Bridge: I should have removed that from prod setup, but it should not really matter, WAN+LAN combined in Bridge0.

Firewall...
- Admin access is allowed only from OPT1 for admin IP 172.16.100.100/24
- IPv4 is allowed IN/OUT using a floating rule
- IPv6 is all dropped

NAT: Disabled IN/OUT. Completely disabled.

Gateways...
LAN-G: 192.168.10.1/24 on LAN (priority 255, connected to completely *open* router Forward/Input)
WAN-G: 192.168.11.1/24 on WAN (priority 255, connected to completely *open* router Forward/Input)
Dynamic OPT1 GW 172.16.100.1/24 on OPT1 (priority 254)

Routing...
Route 1: 10.10.10.0/24 next-hop LAN-G
Route 2: 10.10.11.0/24 next-hop WAN-G

Test...
- Ping from 10.10.10.2 -> 10.10.11.2 (success)
- Ping from 10.10.11.2 -> 10.10.10.2 (success)
- Shaper rules test on file copy (success)
After reboot setup was *still* working but console UI was completely blocked on OPT1


I must have hit a very similar scenario where I realized that OPNSense, if left to its own devices, does horrible routing/blocking decisions on its own, and if not, at least UI navigation becomes very much slower and lagging.


What I have been able to achieve using VIP=Other, for almost 10 years:
- Never get UI console blocked at OPT1.
- Completely secure access to UI console.
- Decision-less, *constant*, and *reliable* layer-3 routing.
- UI navigation is super fast and super reliable.

How I do it (example)...
LAN interface *no* IP assigned, whatsoever
WAN interface *no* IP assigned, whatsoever
Bridge0 combine LAN+WAN
Internal router with IP 192.168.10.1/30
Edge router with IP 192.168.10.2/30
VIP(Other) 192.168.10.1/32 on LAN interface
VIP(Other) 192.168.10.2/32 on WAN interface
Firewall floating rule allow IPv4 from all-to-all in both directions

...and it just works. No surprises. No lag. No blocking. Super reliable.

[Patrick M. Hausen]

You cannot bridge interfaces with IP addresses assigned. In a bridged scenario the IP address must go on the bridge interface. A bridge member (LAN and WAN in your case) must not have an IP address. This is documented for standard FreeBSD and consequently transfers to OPNsense.

Also why are you setting a gateway on your management interface? Are there networks behind that? And should OPNsense route its own Internet access through the management interface instead of WAN?

[MK4L]

You cannot bridge interfaces with IP addresses assigned.

I never used a native BSD setup. Did not know. Nonetheless, my first ever test did not involve bridging; I only used it to make it transparent when I gave up.
And I think it should not affect OPT1, which is not a member?

Also why are you setting a gateway on your management interface?

This is the dynamic gateway created with DHCP client on OPT1. It naturally sits within the admin subnet.
Yes, I prefer to have OPNSense get its own system traffic through the admin subnet.

[Patrick M. Hausen]

I would never use DHCP except on WAN if that's what the ISP delivers.

Yes, OPT will not be affected by your bridge configuration.

All you need should be perfectly doable with routing, though. You can switch off NAT globally. You need a default GW on WAN and none on OPT. You need a static route on your upstream gateway for the LAN network. Done.

[notspam]

Two statements:

1) Bridge:
Yes, like Patrick told you, a network bridge is a layer 2 only and fully transparent device.
Often told "bump in the wire".
Not one with ip addresses on interfaces.
There is only the bridge interface.
And normally a bridge is not a DHCP server or client.
And last thing: A bridge itself is not a routing device. Only the management interface is routed.

2) Management Interface:
Ok, a management interface can have individual routes f.e. to a dedicated out of band management. In the best case as an own routing instance (like virtual router forwarding vrf implementation).

[notspam]

In other words:

Decide if you bridge or route.

What you build is far away from network standards and the reason for the trouble.

[MK4L]

I would never use DHCP except on WAN if that's what the ISP delivers.

Is that consideration specific to OPNSense? Can I flexibly use DHCP on OPT in a native BSD setup?

All you need should be perfectly doable with routing, though. You can switch off NAT globally. You need a default GW on WAN and none on OPT. You need a static route on your upstream gateway for the LAN network. Done.

- I always and consistently disable NAT completely.
- "default GW on WAN and none on OPT": This means I have to reach UI either from LAN, or being directly connected to OPT1?

[Patrick M. Hausen]

You can add a manual static route if your management PC is connected to OPT via some other router. But if you configure OPT via DHCP and the DHCP server offers a default gateway, OPNsense will use it. And since there is no VRF, only a single routing table, that would explain why you had so much trouble. You need an upstream gateway to WAN to reach the Internet. With two upstream gateways, one on WAN, one on OPT, all sorts of weird stuff happens.

As for the "no DHCP" - this is not a BSD nor an OPNsense requirement. Just what I consider best practice. Infrastructure is configured statically.

[MK4L]

no VRF

That explains all! I often use overlapping destination subnets in routes to multiple gateways, for all sorts of things, not specifically in OPNSense.

Would this also mean that the transparent setup I'm using could collapse with any future upgrade?
I would rather still keep it transparent, even with this information in hand.