Adventures in bridging: US consumer Internet with static IPs

[pfry]

I've had dedicated Internet links with static IPs since 1996 (ISDN, whee!); my latest were cable and FiOS, both presented as bridged services. In the past I've used routing equipment (Cisco/Juniper/etc.) to perform proxy ARP in that scenario, but lately I've used dedicated firewalls (Netscreen, Fortinet) that generally don't handle proxy ARP well (if at all). A bridged config has some advantages, namely in flexibility - I can plug my statically-addressed equipment behind the firewall, in front of the firewall, behind a second firewall, etc. with no need for reconfiguration. Over the years I've moved to an every-port-filtered model, so individual computers attach directly to the firewall and cannot communicate outside of the filtered domain. So when I finally got off my ass and decided to replace my poor old Fortigate, I (eventually, after trying a number of packages) set up an OPNsense box with the following configuration:

Four bridges: EDGE, TRUST, GUEST, and JAIL. One routed interface, eventually/possibly to be a link to a second firewall (which I have not tested yet).
The bridge names are somewhat self-explanatory; Edge has statically-addressed equipment, Trust is trusted, Guest is not, and Jail is for things like Ethernet switches and smart TVs (where I want to limit access). Why bridges? Because I can, and again, flexibility. I have four main rulesets and three DHCP pools instead of bunches, and I can move an interface from one "zone" to another just by assigning it to a different bridge. (Occasionally this makes a bridge unhappy/unresponsive, and I haven't yet solved that outside of a reboot.)

Believe it or not, it all works. There are a few limitations:

- NAT destination address filtering didn't seem to work (specifically negation of multiple addresses within a single rule - not really surprising), so communication from privately-addressed bridges/zones with Internet access (Trust/Guest) to my own equipment on Edge is NAT'd. Edge to Trust/Guest is not, so there's a bit of asymmetry, but that's the only apparent side effect.

- No source-spoof filtering. Since traffic from statically-addressed equipment and the Internet exist on the Edge bridge, and the filters are on the bridge interface (more on that below), there's no way to differentiate inside from outside. Note that I have not run a source-spoof filter in years, and I can't recall why not. Likely I discarded it based on data I'd gathered over the years, but if I have any notes to that effect (from 10 years ago), I can't locate them.

- The net.link.bridge.pfil_* sysctls seem to have little/no effect. When a bridge is configured, pf filters are evaluated on the bridge interface and not on the member interfaces regardless of these settings. I had tested member interface filtering to solve the spoof issue, and it simply did not work.

- My provider's ONT or upstream equipment proxies all ARP. The bloody thing will happily transmit packets to, say, Computer A, using its MAC; when Computer B (e.g. the firewall) sends an ARP request for Computer A, the ONT responds, with its own MAC. For an address it has in its own cache. It's genius. It only manages to hijack "quieter" addresses (chattier sources tend to get resolved pretty quickly), but it can result in loss of connectivity, so it's annoying. I'm considering an ugly proxy ARP hack to limit exposure. If anyone has a better solution, I'd love to hear it.

Another element is a switch I use purely as a port expander. It's a Netgear MS510TX, a web-managed 10-port multi-gig unit - 4 100BASE-T, 2 2.5GBASE-T, 2 5GBASE-T, 1 10GBASE-T, and one SFP+. I've attached it to the firewall via port 10, the SFP+; it is configured with routing disabled (mainly to disable chatty protocols like IGMP) and unique VLANs for ports 1-9, untagged, with port 10 having all assigned and tagged. I have the various VLANs assigned to bridges, and those all work as advertised. This gives me 9 (really 8, as I reserve port/VLAN 1 for switch management) fully-filtered ports some distance from the firewall.
I originally had VLAN 1 untagged, but mixing untagged and tagged traffic on bridges did not go well: DHCP was properly differentiated, but all traffic entered the firewall through the VLAN 1 (untagged) bridge. There were likely other anomalies. At any rate, it was bad. Tagging all VLANs is OK.

Basic limitations of OPNsense:

- The uncontrollable automatically generated ICMPv6 input pass rule. Yes, the types are limited. Yes, I haven't seen an ICMP unreachable attack in a couple of decades. From a security standpoint you never want to configure an open pass rule that cannot be covered by a configurable deny rule. I have always configured explicit pass rules for ICMPv4 unreachables, but I can cover them with a deny rule to limit exposure. For me, in the US, it's not a big deal as I'm not likely to be able to obtain consumer-type IPv6 connectivity. If the rule can in fact be limited and I just haven't figured it out, enlighten me. But frankly, all of the auto generated rules should be selectable.

- pf's separate NAT rules. Not really solvable, but a bit annoying. I prefer filter-based NAT, as it's far more controllable using rule order.

Boredom:
OPNsense machine: http://www.tailbone.net/projects/301/index.html
One of these days I'll update it with the network cards (an x710-DA4 and x550-T1). And add backlinks to http://www.tailbone.net/projects.