Topics

Just opening a thread to observations on the 25.7.11 host/neighbor discovery service.

This release brings the new host discovery service which resolves and remembers
MAC addresses for IPv4 and IPv6 hosts in your connected networks and provides
this data for the firewall MAC aliases and captive portal clients.

GUI: "Interfaces: Neighbors: Automatic Discovery".

I do not use the above features, so I disabled the service. It does not appear to interfere with function, but it does report neighbors/hosts multiple times on bridges (on the bridge and on member interfaces) by default. I thought it might be useful in reporting the member interface to which a particular host is connected, but it does not (limit reporting to that). (Note that I did not exhaustively examine its behavior, e.g. to see if it reported hosts only on member interfaces where it saw a particular MAC as either a source or destination.) Note that this behavior on bridge member interfaces applies because I have them assigned but unconfigured; I assume unassigned interfaces would not be part of the discovery, and discovery scope may be limited under "Settings" in any case.

There have been a few discussions of this in the fora; I didn't see any relevant github requests.

Would anyone be up for FIB/VRF support?

It could be implemented pretty simply. As with many OPNsense features, you could use VRFs/FIBs to really screw yourself up. But I think the feature would be quite usable. The beauty is that default behavior would not change in any meaningful sense, and it could be tested to a considerable extent without (GUI) implementation.

Details:

Possible kernel compile option: "options ROUTETABLES=n". Apparently the standard kernel can be configured (using "net.fibs", as below) for at least n=2. Appropriate setting? I imagine it would depend on impact, if any.

System:

• Settings:
• "net.fibs" in loader.conf. Not sure where to put this setting (General -> Networking, as "FIBs" or "VRFs"?). It would be used as an interlock for most of the settings below. Interlock behavior options: vanish/gray/do nothing/error on setting; zero/ignore fib settings when "net.fibs" is unconfigured.
• "net.add_addr_allfibs" - I would make this a tuneable, default 0. I'm not sure if this setting is still available, or if it will be in future versions.
• Gateways:
• Configuration:
• "fib" setting for the gateway.
• "fib" column (not selected by default) for the display. I would display data from all fibs, using "setfib n [command]", as n=0 should always be valid.
• Routes:
• Configuration:
• "fib" setting for the route.
• "fib" column for the display, as above.
• Status:
• "fib" column for the display, as above.

Interfaces:

• [Interface]:
• "fib": integer

Firewall:

• Automation:
• Filter:
• "fib" column for the display, as above. I would include this as the current display has lots of options.
• Rules:
• [Interface]:
• Interlock: for "Action" = Pass, "Direction" = In: "fib" (pf "rtable") setting: integer.

I've likely missed (quite) a few... e.g. "fib" for ping, trace.

Possible caveat: "route" may be fussy with fib > 0 - it might require an "up" interface in the fib in order to add routes. I'm not sure if this is a non-default behavior, as I haven't tested it.

Recent discussions about the automatic rules, as well as the odd session setup and logging behavior of OPNsense prompted me to take a closer look at the automatic "let anything out from the firewall itself" rule, and outbound rules in general.

Statekeeping behavior in pf is a bit unusual, in that (apparently, from testing - let me know if my logic is flawed):

• For outbound flows originating from the firewall itself, an appropriate (matching, statekeeping) outbound pass rule is required. This is the first purpose of the automatic "let anything out from the firewall itself" rule. A matching inbound pass rule is not required for bidirectional communication.
• For inbound flows (with an appropriate pass rule) terminating on the firewall, no matching outbound pass rule is required for bidirectional communication.
• For inbound flows (with an appropriate pass rule) traversing the firewall, an appropriate outbound pass rule is required to set up (effectively) a second session to allow traversal and bidirctional communication. This is the second purpose of the automatic "let anything out from the firewall itself" rule.

This complicates the task of restricting outbound communication from the firewall a bit. (Not attempting to evaluate usefulness/desirability.)

(Of course, many combinations of stateful and stateless rules can be used to effect bidirectional communication in ways not described above, but it would be tough to come up with a good reason to do so.)

Sample application of outbound rules: Given the very general nature of the automatic "let anything out from the firewall itself" rule, specifically that it does not specify an overlying protocol (IPv4/v6)(something not configurable in the OPNsense UI), I prefer to enable logging on it and configure interface-based "quick" pass rules to override it, that inverse match on interface address (I consider the inversion operator harmful, but since it is not going anywhere...) with logging disabled. These rules show up in the "Firewall: Diagnostics: Sessions"/"States" displays (with custom descriptions/labels), but do not show up in "Firewall: Log Files: Live View", optimizing the available display space. The "let anything out from the firewall itself" rule only appears in the live view on flows originating from the firewall itself (including NAT) or (potentially) unusual situations. I want to see the latter in particular, as they may indicate a problem with my topology and/or ruleset. (Chances are that most non-IP packets will never hit pf, but logging it can't hurt.)

That lack of overlying protocol is interesting, considering that the default drop rules have one:

"Default deny / state violation rule":
block drop in log inet all label "ecd3a310894625657c6591b80daa956a"
block drop in log inet6 all label "ecd3a310894625657c6591b80daa956a"
vs.
"let anything out from the firewall itself":
pass out log all flags S/SA keep state allow-opts label "1232f88e5fac29a32501e3f051020cac"

Could be a quirk of pf, but I haven't checked.

Un-truncated subject: On OPNsense 25.7.6, "Firewall: Log Files: Live View" occasional delayed response when unchecking/checking "Auto-refresh" in Brave browser.

Not much more to it. I don't know what all was changed under the hood, but this seems to be a new behavior. The browser seems unhappy - its UI stalls and stutters a bit, and occasionally times out (with a "page not responding") dialog. FYI, I'm not sure if this will be easily reproducible or considered an issue if it is. It occurs most consistently if I run the full 100 lines, uncheck "Auto-refresh", scroll to the bottom, and check "Auto-refresh".

By the way, the staggered shades are nice, and the default 100 line display is adequate (typically what I would choose).

A search on this topic came up with a number of threads, but no resolution (that I saw offhand). I can induce this behavior myself by adding or deleting rules; this does not seem to cover all of the posted cases (it persists through the life of the session only). I really have one question: Are these dialogs scheduled for rewrite/revision? I don't see anything obvious in the roadmap; I didn't dig through github. I won't bother poking at them if they're going to be obsolete soon (or if I missed a resolution).

Geez, these things are still occurring? I've seen a fair number recently (on Frontier in the US), targeting typical third-party caching servers (e.g. Cloudflare, Google). I'm not affected (no session, as I don't use them), but some might be. I figured RPF would have wiped this sort of thing out by now.

Possibly a minor reason to run your own caching server, and specifically not use the common third-party caching servers?

Edit: As numbers games go, it's a really slim one for anyone reading this forum. But hey.

OPNsense 25.1.1-amd64. I didn't see this mentioned in a quick forum search.

Code Select Expand

Tunable Type Value Default value Description
net.inet.tcp.blackhole runtime default 0 Drop packets to closed TCP ports without returning a RST  
net.inet.udp.blackhole runtime default 0 Do not send ICMP port unreachable message

Looking from a shell, these are not set to 0, nor would I expect them to be by default. Merely two examples - not comprehensive.

If you're trying to configure these and expect the defaults to be accurate, you may end up with extra or incorrect settings. Also, this makes me distrust the "reset to defaults" trash can.

(Aside: Edit: I thought there was a description error from FreeBSD, but it was just me.)

This is off of a firewall sitting in front of a web server. Funny how the web scrapers have business hours. (They seem to have done a little overtime tonight.)

You cannot view this attachment.

I don't have a capture of the CPU figures - it wasn't terrible, but it was unusual. Restarting Kea calmed it down. Recommendations on what to capture if it happens again? The logs don't show anything unusual - just the "Lease File Cleanup" from the prior hour (~40 minutes prior). No related system logs.

Edit: Filling that out a bit, it looked to be eating 100% of one core for >15 minutes. Some sort of race, I imagine.

Searches didn't turn this up, but it's tough to key with such common terms.

Firewall: Diagnostics: Sessions "Pkts" field is listed in units of "Bytes". Also in floating point, to two decimal places. Not the display I would expect. (OPNsense 24.7.11_2-amd64)

After reading "Update to latest OPNsense 24.7.9_1-amd64, now have 2 issues" I checked my firewall logs and saw an unusual number of DNS queries from OPNsense. I enabled the query log on my server, and in addition to (possibly) expected queries (e.g. NAT and Spamhaus) I saw:

Jan 09 17:13:00 [named] 09-Jan-2025 17:13:00.255 queries: info: client @0x39630104c20 47.190.83.202#26515 (bogonsv6): query: bogonsv6 IN AAAA + (47.190.83.191)
Jan 09 17:13:00 [named] 09-Jan-2025 17:13:00.255 queries: info: client @0x39630240a70 47.190.83.202#42637 (bogonsv6): query: bogonsv6 IN A + (47.190.83.191)

Yak! In the log:

2025-01-09T17:38:00-06:00   Notice   firewall   resolving 1 hostnames (0 addresses) for Martians_v6 took 0.00 seconds   
2025-01-09T17:38:00-06:00   Error   firewall   The DNS query name does not exist: bogonsv6. [for Martians_v6]

I had one reference to bogonsv6 in a configured alias, as logged. I removed it, which silenced the queries. I had noted that the alias never loaded... but I can't locate the log messages now, only:

2024-12-30T16:58:38-06:00   Notice   firewall    remove old alias bogonsv6

I would prefer to load the alias - it's not as though I don't have the space. Whether or not I need it.
I would prefer not to have the thing mistake a built-in alias for a DNS name and go nuts on it.

(Referring to the other topic, I did not see "<html" or similar queries. I did see a pile of worthless junk from Windows and Firefox - I need to trash this box.)

I have a bridged Internet service with static IPs. My provider's equipment is an annoying unlimited ARP proxy, so using a simple bridge facing the Internet is unreliable. I see three choices:

1) Live with it.

2) Set up static ARP on all pubicly-addressed equipment. Note that static IPs on consumer-type services generally cost a mountain of money over time (~$20/mo for 5), so I won't have many to configure.
Advantage: I can still re-plug equipment freely, e.g. into firewall, parallel to firewall, etc. This includes firewall interfaces, as edge bridge interfaces are interchangeable (no "inside"/"outside"). Really only useful for firewall bypass/replacement (I'm on such a setup right now).
Disadvantages:
 - Static setup required on multiple (in my case, 2-4) machines;
 - Cannot change interface (MAC) without reconfiguration of other machine(s).

3) Set up proxy ARP on the firewall: On firewall: configure a routed interface with a private address, connected to the Internet; configure proxy ARP (Virtual IPs) appropriately; configure a gateway host (static route) for the Internet gateway; alter NAT rules.
Advantages:
 - All configuration is on the firewall;
 - Allows anti source-spoofing rule on "outside" ruleset.
Disadvantages:
 - Introduces fixed "outside" interface(s) - moving addresses from inside to outside requires reconfiguration of virtual IPs and gateway(s);
 - Privately-addressed external interface introduces potential sourcing issues (could be minor/solveable);
 - Kinda ugly.

Opinions? Alternate solutions?

I've had dedicated Internet links with static IPs since 1996 (ISDN, whee!); my latest were cable and FiOS, both presented as bridged services. In the past I've used routing equipment (Cisco/Juniper/etc.) to perform proxy ARP in that scenario, but lately I've used dedicated firewalls (Netscreen, Fortinet) that generally don't handle proxy ARP well (if at all). A bridged config has some advantages, namely in flexibility - I can plug my statically-addressed equipment behind the firewall, in front of the firewall, behind a second firewall, etc. with no need for reconfiguration. Over the years I've moved to an every-port-filtered model, so individual computers attach directly to the firewall and cannot communicate outside of the filtered domain. So when I finally got off my ass and decided to replace my poor old Fortigate, I (eventually, after trying a number of packages) set up an OPNsense box with the following configuration:

Four bridges: EDGE, TRUST, GUEST, and JAIL. One routed interface, eventually/possibly to be a link to a second firewall (which I have not tested yet).
The bridge names are somewhat self-explanatory; Edge has statically-addressed equipment, Trust is trusted, Guest is not, and Jail is for things like Ethernet switches and smart TVs (where I want to limit access). Why bridges? Because I can, and again, flexibility. I have four main rulesets and three DHCP pools instead of bunches, and I can move an interface from one "zone" to another just by assigning it to a different bridge. (Occasionally this makes a bridge unhappy/unresponsive, and I haven't yet solved that outside of a reboot.)

Believe it or not, it all works. There are a few limitations:

- NAT destination address filtering didn't seem to work (specifically negation of multiple addresses within a single rule - not really surprising), so communication from privately-addressed bridges/zones with Internet access (Trust/Guest) to my own equipment on Edge is NAT'd. Edge to Trust/Guest is not, so there's a bit of asymmetry, but that's the only apparent side effect.

- No source-spoof filtering. Since traffic from statically-addressed equipment and the Internet exist on the Edge bridge, and the filters are on the bridge interface (more on that below), there's no way to differentiate inside from outside. Note that I have not run a source-spoof filter in years, and I can't recall why not. Likely I discarded it based on data I'd gathered over the years, but if I have any notes to that effect (from 10 years ago), I can't locate them.

- The net.link.bridge.pfil_* sysctls seem to have little/no effect. When a bridge is configured, pf filters are evaluated on the bridge interface and not on the member interfaces regardless of these settings. I had tested member interface filtering to solve the spoof issue, and it simply did not work.

- My provider's ONT or upstream equipment proxies all ARP. The bloody thing will happily transmit packets to, say, Computer A, using its MAC; when Computer B (e.g. the firewall) sends an ARP request for Computer A, the ONT responds, with its own MAC. For an address it has in its own cache. It's genius. It only manages to hijack "quieter" addresses (chattier sources tend to get resolved pretty quickly), but it can result in loss of connectivity, so it's annoying. I'm considering an ugly proxy ARP hack to limit exposure. If anyone has a better solution, I'd love to hear it.

Another element is a switch I use purely as a port expander. It's a Netgear MS510TX, a web-managed 10-port multi-gig unit - 4 100BASE-T, 2 2.5GBASE-T, 2 5GBASE-T, 1 10GBASE-T, and one SFP+. I've attached it to the firewall via port 10, the SFP+; it is configured with routing disabled (mainly to disable chatty protocols like IGMP) and unique VLANs for ports 1-9, untagged, with port 10 having all assigned and tagged. I have the various VLANs assigned to bridges, and those all work as advertised. This gives me 9 (really 8, as I reserve port/VLAN 1 for switch management) fully-filtered ports some distance from the firewall.
I originally had VLAN 1 untagged, but mixing untagged and tagged traffic on bridges did not go well: DHCP was properly differentiated, but all traffic entered the firewall through the VLAN 1 (untagged) bridge. There were likely other anomalies. At any rate, it was bad. Tagging all VLANs is OK.

Basic limitations of OPNsense:

- The uncontrollable automatically generated ICMPv6 input pass rule. Yes, the types are limited. Yes, I haven't seen an ICMP unreachable attack in a couple of decades. From a security standpoint you never want to configure an open pass rule that cannot be covered by a configurable deny rule. I have always configured explicit pass rules for ICMPv4 unreachables, but I can cover them with a deny rule to limit exposure. For me, in the US, it's not a big deal as I'm not likely to be able to obtain consumer-type IPv6 connectivity. If the rule can in fact be limited and I just haven't figured it out, enlighten me. But frankly, all of the auto generated rules should be selectable.

- pf's separate NAT rules. Not really solvable, but a bit annoying. I prefer filter-based NAT, as it's far more controllable using rule order.

Boredom:
OPNsense machine: http://www.tailbone.net/projects/301/index.html
One of these days I'll update it with the network cards (an x710-DA4 and x550-T1). And add backlinks to http://www.tailbone.net/projects.

Config: ixl3 had 9 VLANs associated, connected to a 10-port switch with ports 1-9 each assigned a unique VLAN; port 10 assigned all, where 1 was untagged and the rest tagged. On the firewall, the various VLANs and main interface were each assigned to one of three bridges. The odd part: From the various switch ports I could obtain a DHCP IP appropriate to the associated bridge, but only VLAN 1 (port 1) would pass traffic (correctly). All traffic showed up on the firewall as entering the bridge to which the main ixl3 interface was assigned. When I removed ixl3 from its bridge, all of the others (the VLANs) worked.

So I tagged VLAN 1 on the aggregate port on the switch, added it and assigned it on the firewall, and now (after a reboot - the bridge was unhappy after reconfiguration and poking it didn't help, so I just rebooted the firewall) it all works. I recall potential issues with entrance interface assignment regarding bridges and packet filtering, but nothing regarding VLANs. Did I miss it?

I'd poke around a bit and perhaps reply to myself, but with the ready solution, I probably won't.

OPNsense 24.7.11-amd64. Simply put, the net.link.bridge.pfil_bridge and net.link.bridge.pfil_member tunables seem to have no effect. Filters on the bridge interface(s) function, count packets, etc.; member interface filters are never matched (according to behavior and Firewall -> Diagnostics -> Statistics: rules). The default anti-spoofing filters (which would appear to block packets evaluated on the member interfaces) are also never matched. I may not have poked every combination of net.link.bridge.pfil_*, but I've done the obvious ones. From what I've read, the sysctls normally behave as advertised (with ipfw). I have not yet tried passing traffic through the firewall (just to it).

This is likely why the setup guide doesn't lock everyone out, as the "lan" default pass and anti-lockout rules end up on the bridge.

Yes, I'm looking into filtering on the bridge member interfaces directly. In theory it shouldn't be difficult. Am I missing something obvious?