Does OPNsense use multiple routing tables for Multi WAN?

[whitwye]

The doc refers to Policy Routing, without details on backend implementation. On Linux it's standard to use multiple routing tables for Multi WAN. FreeBSD supports that (https://lists.freebsd.org/pipermail/freebsd-arch/2007-December/007331.html, discussed at https://www.mmacleod.ca/2011/06/source-based-routing-with-freebsd-using-multiple-routing-table/). When I checked the pfSense kernel, it looked like that team left this feature out there. Is OPNsense using multiple tables for its Policy Routing? If so, what's the right way to view the multiple tables? It doesn't look like netstat has an option for that.

[franco]

We don't use multiple routing tables. The routing table, however, is superseded by fine-grained "route-to" directives from the packet filter. This works relatively well and provides proper isolation if configured correctly.

Multiple routing tables would make an interesting addition, but I'm not aware of any efforts to bring OPNsense to this way of doing things, suspecting it may be quite a bit of work.


Cheers,
Franco

[whitwye]

Thanks for the explanation. What is the right command to view the full pf table to see how route-to gets set up? None of the pfctl  flags (including --vvsr and -sa) display that. I'd like to be able to correlate settings through the GUI with back-end results, to resolve some things which are mysteries to me.

[franco]

Almost..

# pfctl -sr | grep -e reply-to -e route-to

You could also see the generated ruleset via /tmp/rules.debug

"route-to" are the rules for outbound policy routes,
"reply-to" are the rules for inbound policy routes in response to the traffic that it applies to in the outbound direction.

[whitwye]

Ah, I was looking for route-to at a time when I just had reply-to rules.

Now that I see them, this leads to another question. When I set up a rule to allow connections on the second WAN interface from a remote IP, this works if I set the gateway to "default," but fails if I set it to the explicit gateway that's the default for that WAN. It puzzles me that the two settings don't mean the same thing for the firewall. Now I see the result is different. Setting it to "default" produces this rule, which does what I want:

pass in  quick on igb2 reply-to ( igb2 <gateway IP> ) ...

Setting it to the specific gateway produces this rule, which at best lets a connection get started, but then fails:

pass in  quick on igb2  route-to ( igb2 <gateway IP> ) ...

Is there a general rule-of-thumb to understand when to specify a gateway as "default" or more specifically when setting up firewall rules, in regards to reply-to and route-to usage on the back end?

[franco]

You set a firewall rules gateway when the routing table (or the default gateway if no routes exist) should be superseded or is not flexible enough.

The normal use case for this is you have a multi-wan and you want to filter directions by source, not destination. The routing table won't do this (it is destination-only).

This can be especially funky with "gateway groups" where you can inject multiple gateways at the same time and alternate between them based on mere chance or round-robin fashion without constantly reloading the routing table with your gateway specification.

Reply-to is special. In our case it is used to ensure upstream gateway communication for incoming traffic that needs to be returned. This is done because otherwise a multi-wan configuration could return the traffic on the wrong outbound interface, creating asymmetric traffic that the client on the other side will reject.

[whitwye]

Thanks for the explanation. As a feature request, there could be either something like a checkbox or radio buttons to specify whether a reply-to or route-to rule is desired, or else text added to explain that choosing "default" results in "reply-to," while choosing the specific IP (which is the default) results in "route-to." Otherwise I don't know how most people will intuit why the results should be different.

Where does a gateway group get represented? If the one I've configured is in the firewall rules, I'm not seeing it.

[franco]

I don't believe throwing in "route-to" and "reply-to" into the help labels helps clarity. A gateway implies route-to. Reply-to can be disabled via the advanced settings and there is a help text.

Gateway groups are immediately available from the drop down list, they would be at the bottom below Null4/Null6.

[whitwye]

Sorry I was imprecise in my question. Where are gateway groups represented within FreeBSD? I'm not asking about the OPNsense GUI, where I have one set. I'm asking about how to see the result on the back end.

Just as I had to look at the real firewall rules to understand how "default" for a gateway setting resulted in a different rule than specifying the IP there -- even though both rules used that IP -- I'd like to know where to look to understand how the gateway group concept is really applied.

[franco]

Gateway groups are an abstraction in OPNsense that are "controlled" by the output of the gateway monitoring daemon (apinger). They do not translate to anything in FreeBSD itself.

[whitwye]

Thanks. I see your name on the apinger man page.

So apinger is triggering a script that sets the route for those firewall rules that have the gateway group defined as their gateway? How do we know, looking at the firewall rules, which have resulted from apinger's actions?

Is there documentation on the back-end implementation somewhere? Hate to keep pestering you with questions. But I don't trust a system unless I know where to look "behind the curtains" to debug it.