Messages

[VLAN Hardware Filtering]

I just deployed a couple of OPNsense 19.1.4 routers. They're the i386 image, that's why it's an older firmware, I believe it's just about the cutoff line where it became 64-bit exclusively. Anyway, I need them only for DHCP/DHCP6 and maybe route a couple of IPv6 networks. They're perfect for the job.

I need FRR though which requires checking for updates first but it won't update. I get:

Code Select Expand

***GOT REQUEST TO UPGRADE***
Updating FreeBSD repository catalogue...
repository FreeBSD has no meta file, using default settings
Unable to update repository FreeBSD
Updating OPNsense repository catalogue...
repository OPNsense has no meta file, using default settings
Unable to update repository OPNsense
Error updating repositories!

I enabled /usr/local/etc/pkg/repos/FreeBSD.conf but still wouldn't work:

Code Select Expand

root@namemaster1:~ # pkg update
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
pkg: http://pkg.FreeBSD.org/FreeBSD:11:i386/quarterly/meta.txz: No address record
repository FreeBSD has no meta file, using default settings
pkg: http://pkg.FreeBSD.org/FreeBSD:11:i386/quarterly/packagesite.txz: No address record
Unable to update repository FreeBSD
Updating OPNsense repository catalogue...
pkg: Repository OPNsense load error: access repo file(/var/db/pkg/repo-OPNsense.sqlite) failed: No such file or directory
pkg: https://opnsense-update.deciso.com/FreeBSD:11:i386/19.1/latest/meta.txz: No address record
repository OPNsense has no meta file, using default settings
pkg: https://opnsense-update.deciso.com/FreeBSD:11:i386/19.1/latest/packagesite.txz: No address record
Unable to update repository OPNsense
Error updating repositories!

The DNS addresses do resolve:

Code Select Expand

root@namemaster1:/usr/local/etc/pkg/repos # drill opnsense-update.deciso.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 41995
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; opnsense-update.deciso.com. IN A

;; ANSWER SECTION:
opnsense-update.deciso.com. 900 IN A 89.149.211.205

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 639 msec
;; SERVER: 10.11.11.36
;; WHEN: Sat Apr  5 02:23:24 2025
;; MSG SIZE  rcvd: 60
root@namemaster1:/usr/local/etc/pkg/repos # drill pkg.FreeBSD.org
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 24707
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; pkg.FreeBSD.org. IN A

;; ANSWER SECTION:
pkg.FreeBSD.org. 240 IN CNAME pkgmir.geo.FreeBSD.org.
pkgmir.geo.FreeBSD.org. 150 IN A 192.158.252.167

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 476 msec
;; SERVER: 10.11.11.24
;; WHEN: Sat Apr  5 02:23:54 2025
;; MSG SIZE  rcvd: 74

There something about a database that sounds like a local issue, but a little latter something about repo meta, which sounds the opposite. So I just came for help before I start tinkering with it.

Are the repos still live?

Also...since I'm here already :) it's been a while since I last used this version, and I can't seem to make static addressing regardless work.
The firewall/natting is off completely off but addresses don't seem to be configured at all (I ran nmap scans from other systems) unless gotten from DHCP. Even after rebooting. Was there an extra step with this firmware?? :O

In Settings→Interfaces, all the offloads were already disabled (all boxes are checked) out of the box and VLAN Hardware Filtering is set to Leave default. Gateway switching is enabled, but made no difference disabled, if I remember correctly— I might need to double check that.

Thanks!

on the same setup, what was your performance on VyOS.

I'm sorry, I missed the message, I started writing the other one and hadn't refreshed the page until it was posted.

VyOS (and the others) performs at line speed— The ONT's ethernet side line speed, 1gig. pfSense and OPNsense where the slow ones. It seems not anymore though. VyOS does have some weird natting issues though. It drops, I believe, smaller-sized packets or whatever it is it's in a very specific way. For instance, DuckDuckGo (among other domains) stops working. It's resolvable on DNS but the connection just fails. It's an amazing platform otherwise, I think it's my favorite and it's what I was using up until a few hours/days ago.

I lost track of time already.

[SQM]

I think it was bufferbloat. I'm still not a 100% sure, but already I saw a MASSIVE improvement.

I'm losing my mind a little here. I re-enabled hardware acceleration first before doing the bufferbloat settings (which I still have not tuned well) and it's still enabled.

When I enabled acceleration (unchecked the setting, in the screenshots next), I rebooted after each, the first part, I don't remember up to where, resulted in performance gains. The second part took them back.

Since none of the Linux routers needed these settings, and this Smart Queues, or SQM kept popping up here and there always with some notice saying that that's usually needed for speeds up to around 300Mbit/s, I think I thought it was the same, or at least a similar enough thing so I didn't bother at all with it. I had set up FQ-CoDel settings once on years ago on pfSense but it was when I had 4 xDSL with less than 200Mbit/s total. It didn't cross my mind ever again until just now, and even today I thought it was going to be pointless. I'm so happy I was wrong.

The bufferbloat set up on the documentation says to target 85% of the advertised ISP speed first, but I assumed this is because maybe (?) most ISPs don't deliver the advertised speed — I don't want to believe that to be true, but for my own sake I kinda need to — Mine does. It [silently] compensates with extra bandwidth so users get what's advertised, as it turns out, there's a law about it. They must deliver what they advertise, I found about it today, too. Advertising prices without sales tax is illegal too, I should've figured it out sooner. Not the point— I started with that already, the advertised speed which I know I get. 1Gbit/s.

Even if I'm wrong I'm around the 900Mbit/s mark again this time with NAT done on OPNsense. No more symmetric NAT. :D Resource use is low, if it keeps reasonably fast after the full ruleset and services are set up I'm staying put.

I'm lazily documenting in screenshots. I'm not embedding it bc it came kinda huge: https://i.imgur.com/oYSAz4Y.png

I'll check back later when I settle on to something. :)

[VMXNET3]

I'm trying to set up an OPNsense virtual appliance but I'm having a hard time getting good performance, especially when it comes to NAT, that's where it really shines, the issue— that is.

Environment

The VM is on vSphere (type 1 hv) with tons of memory and CPU cores to throw around, compared to what it will replace. Disk on very fast vSAN storage. Towards the end I switched to a RAM disk to eliminate it as a potential bottleneck. I had already moved away from FreeBSD-based firewalls but policy routing is a nightmare on Linux, so here I am.

I found this article somewhere alleging that the issue was FreeBSD drivers in a Linux hypervisor, not the whole Scalar- vs Vector Packet Processing, as I thought it might be. This restarted my efforts to get back on FreeBSD.

QEMU/KVM confirmation

I've read a lot about macvtap being the second coming of passthrough interfaces in the absence of SR-IOV. I tried it and it kind of showed. Throughput was better than before. The last time I did these tests, the [recommended] paravirtual interface driver, VMXNET3, hardly broke past ½Gbit/s.

╭──────────────────────────────────────────────────────────────────────────────╮
  RECAP/PROGRESS SUMMARY
├──────────────────────────────────────────────────────────────────────────────┤
  CHR/OpenWRT/VyOS baseline ☑︎ Excellent.2
  macvtap routing           ☑︎ Good. >900Mbit/s
  macvtap NAT               ☐ (untested)
  SR-IOV VF NIC routing     ☐ (not there yet)
  SR-IOV VF NIC NAT         ☐ (not there yet)
  PCIe NIC routing          ☐ (not there yet)
  PCIe NIC NAT              ☐ (not there yet)
╰──────────────────────────────────────────────────────────────────────────────╯

vSphere

It seemed like there was some true to that article, but I was mostly guessing my way through things on KVM thus I moved back to vSphere. I tried it again with SR-IOV [Virtual Function] NICs: passed the 900Mbit/s mark in Internet speed tests and iperf3. My uplink¹ is only 1Gbits, it's really a bit more than that because my ISP factors in protocol overheads in order to deliver the advertised speed and avoid complaints, I assume.

Almost almost there

The thing is that I can get the full bandwitdh and sustain it with a modest 2-[AMD64]core Linux firewall with just a little over a gig of RAM.

>900Mbit/s is pretty good if you turn a blind eye to knowing that underlying issues are a fact preventing the full throughput of the link, and unfortunately, I was more than willing to do it. This has not been a quick-and-painless journey, exactly. I can't claim sexual abuse if I consent to it, but enough about work (JK).

At this point, another router was handling NAT and the PPPoE session, it was time to hand them over to OPNsense. I just show down a port on a switch and just as fast OPNsense was in control of a public IP address, when I did the tests though it was the worst results I've ever gotten: download throughput didn't reach 300Mbit/s, upload was well over 300Mbit/s on the other hand, well over the max upload speed I get from my ISP last time I checked.  The last part isn't all too surprising bc they keep upgrading the service without warning (though without price hikes either).

NAT rules

My NAT setup is always the same regardless of platform. It's really only outbound (SNAT) on the public interface only and a handful of port forwards to the reverse proxy where the heavy lifting and internal NAT occurs. The forwards weren't set up yet. For the outbound NAT, I undo the rules created by the initial setup wizard, if any, and in its place add two rules:

╭─┬───┬─────┬─────┬─────┬─────────────────┬───────────────┬────────────────────╮
│#│NAT│if   │stack│proto│src:[port/type]  │dst:[port/type]│NAT-to:[port/type]  │
├─┼───┼─────┼─────┼─────┼─────────────────┼───────────────┼────────────────────┤
│1│neg│<wan>│IPv4 │any  │<This Firewall>:*│any:*          │-:-                 │
│2│yes│<wan>│IPv4 │any  │any:*            │any:*          │masquerade-if:static│
╰─┴───┴─────┴─────┴─────┴─────────────────┴───────────────┴────────────────────╯
╭──────────────────────────────────────────────────────────────────────────────╮
  RECAP/PROGRESS SUMMARY
├──────────────────────────────────────────────────────────────────────────────┤
  CHR/OpenWRT/VyOS baseline ☑︎ Excellent.2
  macvtap routing           ☑︎ Good. >900Mbit/s
  macvtap NAT               ☐ (untested)
  SR-IOV VF NIC routing     ☑︎ Good. Slightly better than macvtap's.
  SR-IOV VF NIC NAT         ☒ Bad. ≈300Mbit/s↓ >300Mbit/s↑
  PCIe NIC routing          ☐ (not there yet)
  PCIe NIC NAT              ☐ (not there yet)
╰──────────────────────────────────────────────────────────────────────────────╯

Bare-NIC testing (i.e. like "-metal" but just the tip NIC that goes in the back, a little bare in the back)

The next thing I tried, using the same NIC that I know can handle the traffic because it has done so in Linux firewalls already — easily — I passed it through at the PCIe level to the VM. Full control. I had to shut down everything to do this, you need a career and a psych eval, and security clearance to shutdown a [small] vSAN cluster, I swear.

I did the tests again, (1.) offloaded NAT and PPPoE to the other router, it improved minimally. (2.) Performance tanked again while natting.

╭──────────────────────────────────────────────────────────────────────────────╮
  RECAP/PROGRESS SUMMARY
├──────────────────────────────────────────────────────────────────────────────┤
  CHR/OpenWRT/VyOS baseline ☑︎ Excellent.2
  macvtap routing           ☑︎ Good. >900Mbit/s
  macvtap NAT               ☐ (untested)
  SR-IOV VF NIC routing     ☑︎ Good. Slightly better than macvtap's.
  SR-IOV VF NIC NAT         ☒ Bad. ≈300Mbit/s↓ >300Mbit/s↑
  PCIe NIC routing          ☑︎ Good. No different than SR-IOV's.
  PCIe NIC NAT              ☒ Bad. No different than SR-IOV's.
╰──────────────────────────────────────────────────────────────────────────────╯

Overall, I learned OPNsense can reach pretty close to the full gig of my uplink which is also the speed of the slowest link in my wired network, so it's good enough. It just needs not to use [para]virtualized NICs. Resource-, or hardware-wise it must be able to NAT at that speed too, other systems are natting much faster than that already, case in point: the reverse proxy that hosts a bunch of virtual IP addresses so it NATs what it can't be proxied and might be conflicting in some way, such as TCP port 22.

Notes

 1. i.e. the connection, not the actual bit rate
 2. Bursting briefly past 1Gbit/s (1.1) w/test well underway. No dips below 1G. Results are consistent in iperf3 tested:
 ↳.1 server ←←← <this-router> ←←← client
 ↳.2 <server/this-router> ←←← client
 ↳.3 <server/this-router> →→→ client(-R)
 ↳.4 server ←←← <client/this-router>

Did you sort it out?

I'm more or less in the same spot, except I'm testing with public servers.

My connection is pretty stable and it's only 1Gbit/s. It should be easy to get max it out, but OPNsense is the only firewall that is way below the others.

I even did all the tuning and all that but it improved nothing. :/



For each firewall, I cloned the first VM without disks, then installed a fresh image on it so things are exactly the same.

[In the screenshot] OPNsense was tested tuned (allegedly), without extras and a very basic ruleset. The other test is a loaded Mikrotik CHR, under normal load (at least 15-20K connections just from BitTorrent traffic). Results were the firsts ones I got, not the best. pfSense and OpenWRT were deployed just for the test thus they were in similar conditions than OPNsense; they performed consistently similar to CHR though, actually a bit better, I assume since they had no load.

It's not a scientific as your testing but, y'know, it's real world—to complement yours. Hopefully it gets some expert's attention. :)

Thanks,

Sorry for taking so long, I had been meaning to answer earlier but I had to exhaust all measures first. Y'know, do my part as well. :)

I'll make a list so I don't lose focus (ADHD), I...

- Uninstalled all versions of FRR
  - Including the pythontools package
- Open a text file to start documenting all of this
  - ...but forgot its name and now I can't find it :P
- rm -fR /etc/frr
- edit /etc/rc.conf to remove [watch]frr-related settings
- rm -fR /usr/local/etc/frr
- rm -fR /usr/local/etc/rc.d/{frr,watchfrr}
- found stuff in other directories, like the socket file in a var directory, (not in /var though, just some downlevel var) and some scripts inside an OPNsense directory, left those in place.
- Checked permissions on configuration files and socket files definitely the most tedious since I also had to find the docs for it too, fortunately they're actually quite straightforward :)

I reinstalled the CLI version first, didn't work. I repeated all of the above and reinstalled the GUI version; didn't work.



Running a packet capture on the interface connected to area zero I saw traffic of other routers but not of this one. FRR doesn't seem to be using the network at all because — I think I mentioned in the earlier post — it doesn't even learn RIP routes for which it doesn't have to form adjacencies in order to use them.

Now, I just reinstalled it, if I configure it now it will work for a while until a config change is required, if it doesn't go perfect, at some point it stops accepting, it seems, configuration changes and it's like it "decides to stay put" if you will. But, I have no proof, which is partly why I come for help. :/

Oh well, someone will catch it someday, thanks again, I've would've dropped it had I not seen your answer. :)

I'm attempting to migrate to OPNsense and I need to add a ton of virtual IP addresses.

The UI is a little cumbersome and has it makes me get lost/lose focus of the text some reason so I edited a backup file to add them quicker and sure enough I finished it but without noticing the uuid key until the end.

Can I just omit it? Is it okay for the VIPs not to have a uuid?  Can OPNsense supply it automatically if I omit it, if not, will it crash if the XML schema isn't matched perfectly?

It's been rough moving to OPNsense with so many things broken/undocumented. I'm on day 2 importing aliases as it is. :( And I still have to plan static routes bc FRR broke completely. I have no idea how to do ECMP statically but I'll get there when I get there. :/

But, on my previous router firewall which has been being deconstructed as functions are moved to the new one but I still have enough restore points to get back to a working network, I really want the extra features of OPNsense but I'm still on time to get back to the needed features for the network, just barely.
Are there any other basic features serialized? Or is this it? — Thanks

[no RIP or OSPF exchanged with OPNsense]

Hey all, :)

A while back I set up FRR from the GUI, but the lack of options and this problem where it would stop respoding to config changes (which I can only fix by reinstalling OPNsense and restoring an edited config backup without FRR) drove me FRR proper, v8 on the CLI/console.

However, after setting it up, enabling the daemons ([/usr/local]/etc/frr/daemons), and verifying it (along with watchfrr) would start; vtysh would create the config file and everything but in the end it always starts with the same configuration:

Code Select Expand


Building configuration...

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname f.q.d.n
!
line vty
!
end


It completely ignores the config created on vtysh (written to /usr/local/etc/frr/frr.conf).

I searched for days for the config files or init script or whatever that was persisting that but I don't know much about FreeBSD so I couldn't come up with anything.

I found references to some files under an rc.d directory — I'm not sure which, given the they're a ton of these on repeated over several levels — but those files referenced didn't exist.

I removed FRR 8 and returned to the GUI version (os-frr), set it up but once again it won't send or receive any traffic. I noticed on neighboring routers there is no RIP or OSPF exchanged with OPNsense in their routing tables, only with other routers. And in OPNsense itself, the routing table lists only connected routes; it doesn't receive information either. The firewall is wide open on all affected interfaces.

I need to reset FRR, I think. But the file structure in FreeBSD in infinitely confusing even before getting to OPNsense's own customizations and I'm also trying to make OPNsense my main firewall, so reinstalling — that's full reinstallation or cloning VM template or reverting VM (not system-)snapshot. Otherwise it doesn't fix anything — each time there's a problem won't be an option anymore.

Could you guys give me some pointers how to do this, please? Finding out all related system files used by FRR.

Thanks.

[definition name]

I think I know the answer to this already, so it's more like a feature request than a question.
However, if you have an answer/suggestion/advise, I'm all ears and open to anything. :)

I keep around an OPNsense instance ZeroTier and a few other light tasks. I've a few issues with OSPF in this system so I tried updating before getting down to vtysh.

It didn't but I was just advised to use GUI-less FRR, problem solved. With that done, I explored a little to see what was in the long list of changes and I found out that ASN-based aliases are now supported on OPNsense. If they're the same thing as using pfBlockerNG on pfSense that's amazing news.

There were a few reasons why OPNsense would never fully replace pfSense: ASN filters, HAProxy's GUI, log views, and (somewhat for) the forward proxy and VRF. VRF isn't available of pfSense either, ASNs are done, next was HAProxy's GUI's modularity nightmare.

I get that making it modular could in theory make it more practical, I do. Assembling the final proxy from dropdowns does feel sleek in its own right, however, getting there is so much non-repeatable/-usable/-cyclable disorienting work that's harder to diagnose. The level of modularity is only justifiable by needs that go beyond what an in-firewall reverse proxy should safely have, probably better served by a first-party-supported, dedicated Aloha appliance. HAProxy's rules, matches/fetchers, conditions, monitors are tightly focused in scope and hardly ever reused; all of these need to be defined, each with their own character-constrained definition name which translates into several more components per proxied service to keep track of and hunt around from screen to screen. A giant blank box without any instruction whatsoever (to paste in config files standard well-documented HAProxy syntax) would be much more helpful, easy, and clearer. Even without instructions. Fortunately you can just NAT it or if coming for straightforward-UI-HAProxied pfSense you can just put it inline in a transit network.

That leaves out the forward proxy — I've mixed feelings on this — on one hand OPNsense allows customization of the error pages. I love designing dumb stuff so this is a perfect fit for me. On the other I'm also pragmatic (see: HAProxy above) and good looking error pages, that are errors in the end, aren't nearly as useful as the live log view on pfSense. Additionally neither implementation has usable/maintained block lists which is kind of strange more so now than ever.

With DoH and DoT mainstream, they're readily used to maliciously bypass our network policies. Case in point, I use the Yahoo! website to test network connectivity (because otherwise I'd never visit it, so I know it isn't cached); the other day I found that the website embeds some sort of DoH client that was making requests for yahoodns.net or similar. This can be dealt with a proxy, except that few of them support multi-WAN, the ones that do, have basically no support for de/re-encryption of the traffic. I've been looking non-stop for a good proxy to help with this since I have six DNS server hosts, each has one or two DNS servers (Active Directory, Pi-hole, Dnsmasq, Unbound and BIND) running to filter, proxy, route the traffic to keep tight control over those rogue DoH clients, DNS (filtering not domain-related) is easily 60-70% of my network issues.

But given OPNsense for a long time has done some weird voodoo (shared forwarding) to support splitting traffic among captive portal and other of its otherwise-conflicting components, I'm wondering: has the proxy service been worked on as well? A proxy can be extremely memory hungry to deploy one each upstream.

UNSUBSTANTIATED
I've given it some thought to this not just on OPNsense but using a variety of systems, just using a single OPNsense system though; I have a theory that it could be done if: in transparent proxy mode, use NAT to force-policy-route the traffic running in a first/second instance using the jails subsystem on xBSD. It would still need a lot of memory but they can share kernel and storage and free memory and don't need to send traffic over the network since they'd be in-host. And, were VRF be supported, even NAT could be skipped as well. But, unfortunately none of this is even hinted in the GUI nor mentioned in the documentation other than some dev-level stuff. I wanted to try this but not being as familiar with FreeBSD as am with Linux, macOS; it'll be a while before I find the time to I learn jails to test it out. On paper it seems very plausible though, a good enough solution for cases where a dedicated proxy isn't needed, for beefy multi-WAN/multi-tunnel firewall with a small 100GB-ish disk cache.

[1x switch, 1x OPNsense, 2x pfSense]

Hi everyone :)


So... I have had this set of routers running OSPF: 1x switch, 1x OPNsense, 2x pfSense; they were all fully adjacent. Then I switched one of the pfSenses for OPNsense (new setup: 1x switch, 2x OPNsense, 1x pfSense) which happened to be the only one that wasn't directly connected to area zero and things went south fast. It's across an L2 tunnel, thus it's broadcast and should have no problems, on the other hand there's so little in the UI to set this up so I could never get it to work. About 40 or so hours in I thought "f**k it, I'm going on the CLI" I was already a little familiar with it (vtysh) because it used it to make pfSense form adjacency with the switch-router which doesn't have FRR, just "OSPF", As it turns out, the CLI for these areas is nearly if not identical on each vendor (found it too in Ubiquiti and VyOS proper).


Anyway, first I tried FRR8 so the changes I had already done in the GUI wouldn't overwrite things back but there was some conflict so I went back; I shook it a little, wiggled the thingy, put it against my ear and repeated until it sprung to life, forming full adjacency with the nearest peer.


It's done. It works. However, since pfSense is the closest to OPNsense to where guide myself from, I took a significant^{(just 1% more of 99%)} amount from there, pfSense stores data in /var/etc/frr/ and /usr/local/etc/frr/. The official FRR documentation focuses more on the Linux side only /etc/frr so it's clear as saying "see you at seven", without following it by "hours" or "am/pm" and neither pfSense nor OPNsense have the most extensive documentation on this or at least some that unlike the official FRR documentation is targeted at their respective user bases — y'know, one or two^(thousand) levels below "network engineer", referred to by their tribes as He or She Who Runs Not With Cisco.  ;D


/var/etc/frr wasn't on OPNsense, so I assumed this was a temporary location on pfSense to save configuration, it seems to have a ton of these little pockets.


I modified vtysh.conf (because using vtysh the first time around printed something about it being jsut for show) and frr.conf, followed by service frr restart. It didn't seems to make a difference, it printed a lot of stuff of how misconfigured it was. I copied the files I had created on /var/etc/frrto /usr/local/etc/frr next to several other config files already there. vtysh.conf and frr.conf weren't. service frr restart and this time it only printed one warning. It seems like that was the place except when I tried getting information on vtysh, no interface was actually configured.


FRR docu's does say that daemons need to be started and the daemons file was the one missing from OPNsense. I added it and enabled a few daemons (zebra, ospfd, staticd, bfdd), restarted. Nothing.


But, after retrying vtysh with the modifications I had done, write now worked. I'm sorry I made it this long but I wanted to describe the places (paths) I touched to hopefully get a better answer to: where does OPNsense actually stores the files? (because) From another terminal, I saw the modifications being written by vtysh were where I initially thought the were going to be, but as I mentioned, I had already done this before I repeated them on vtysh and when not done via vtysh they were ignored.


Also, what do I need to do to lock the configuration down? So it's not overwritten by an update or a restart or just checking out its status from the GUI. In pfSense the GUI easily overrules the CLI/configfiles all the time, I'm not sure if OPNsense differs in there yet.


I'll just leav--um.. Thanks.

Hey, sorry nobody replied earlier. This is why I can't switch fully to OPNsense, docu isn't great and you can't get help. :(

Anyway, if you're still looking for the answer; first I'd like to ask you if you added ALL ROUTES at once. But since It's unlikely I log in soon to check, I'll make some assumptions, if that's OK.

For option 121 you add additional routes, that I gather you already know. As I understand (and in my testing) you need to also add the default route at the end, and before the additional routes (at least on Windows' DHCP client and server) add the route for the local subnet, e.g; all your subnets are /24s, you have an L3 switch with an interface on each of your VLANs taking address .1 and spaced roughly /16 apart on the 10/12 range (so [skipped 10.0.0.1] 10.1.0.1, 10.2.0.1...10.15.0.1) and a router on the same subnets on address .2 that goes out to the Internet, your site-to-site tunnels, remote clients, etc. in other words, the default route.

Range:             10/12
(10.0.0.1-10.15.255.254)

Subnets:     10.1.0.0/24,
             10.2.0.0/24,
             10.3.0.0/24,
  (...)       10.15.0.0/24;

L3 switch:      10.1.0.1,
                10.2.0.1,
                10.3.0.1,
  (...)          10.15.0.1;

Router:         10.1.0.2,
                10.2.0.2,
                10.3.0.2,
  (...)          10.15.0.2;

So, for clients on the 9th VLAN, 10.9.0.0/24, you'd need the routes:

10.9.0.0/24   0.0.0.0     #see fig2
10.0.0.0/12  10.9.0.1
0.0.0.0/0    10.9.0.2

If the 9th VLAN has a second or third subnet directly accessible on the broadcast domain, for instance "10.9.1.0/24", you specify it as a local subnet. Clients would still need an address on the subnet, e.g; eth0=10.9.0.44/24,10.9.1.44/24. Option 121 would need to be:

10.9.0.0/24   0.0.0.0
10.9.1.0/24   0.0.0.0
10.0.0.0/12  10.9.0.1
0.0.0.0/0    10.9.0.2

You have to enter all the strings for all subnets concatenated in option 121. I don't know the syntax though. There was this website that did it for you (I did it on pfSense too) but I don't remember which was it. On Microsoft's DHCP server it looks like fig1, and though it looks easier, looks are deceiving: you need to enter the values in order—they can't be rearranged after the fact. It's been a while though, I might be forgetting something.

An alternative to this (if you goal is something like offloading the routing to a beefier, much faster device such as an L3 switch) is setting it up as the default gateway and use a transit network between it and your upstream router, and one or more static routes on the upstream router. So using the same example subnets, let's say you transit network is 10.16.0.0/24..or /30, whatever.

The switch keeps it .1 address, so does the router. On the switch you set up its default route to 10.16.0.2 corresponding to the router on the transit network just outside the 10/12 range. And your done on that side. Now on the router, you add a static route to 10/12 via 10.16.0.1. If it is a firewall, with per-interface ruleset such as pfSense or OPNsense, you'll add them all in a single interface. You may create network aliases for each of your networks, e.g; 10.9.0.0/24 alias "zone9" or "iscsi".

On OPNsense you can (temporarily) add the routes on console, for the example scenario:

Code Select Expand


route add 10.0.0.0/12 10.16.0.1
# test if the internal gateway (the switch in this case) is responding
ping 10.16.0.1
# then test with a host on the remote internal network
ping 10.9.0.77
# or one of the switch's internal interfaces
ping 10.9.0.1


And since your "LAN subnet" preset would not longer match, i.e; it would now match 10.16.0.0/24 while your real LAN is, and must be, on another range; 10/12. So if you're using the default ruleset, "allow anything from the LAN" the firewall will kick in dropping all traffic. For that you need to disable the firewall from the console so you can make your way to the GUI and add (1.) the static routes permanently and (2.) a new firewall rule or edit the existing LAN firewall rule to allow anything from anywhere to buy you some time while you create more targeted rules without being kick out every single change because each time you save something, even if you don't apply it, the firewall re-enables itself. So you need to disable it again, and again, and again...

Code Select Expand


#disable firewall
pfctl -d


Test from the inside out, from a host:

Code Select Expand


# the local gateway
ping 10.9.0.1
# the switch's gateway
ping 10.16.0.2
# the router/firewall's gateway or some public well-known host
ping 9.9.9.9


The transit network approach — and perhaps I should've started with that — will avoid asymmetric routing that can happen when you have more than one path between subnets. Since firewall like OPNsense are stateful, and routers like L3 switches aren't keeping states, data can flow one-way only and get block by the stateful firewall between subnets and by the built-in stateful firewall in most client OSes if NAT or a reverse proxy are in the way.

In regard to option I didn't know about it, or at least I didn't remember about it if I knew, but from Microsoft:
(...) Microsoft Classless Static Route Option (...) only difference is that Option Code 249 SHOULD be used instead of or in addition to Option Code 121.

So if it work when you add 249, it's because you're most likely overriding option 121. Also check out RFC 3442 just before "page 4"; 'Local Subnet Routes'.

I hope this helps OP or anybody in the forum looking for info. Because the documentation... OMG! I've tried setting up IKEv2 with RADIUS, the docs say to select no RADIUS server in the mobile client config and select it in the phase 1 instead, however, that isn't possible. For over a year I keep checking if there's any change with the new releases... none.

fig1:

fig2:

[huge]

COMMA!? I swear the list of characters and escaped characters I tried was much longer and that must've been the one character I didn't try! I'll stick to the spreadsheet, it's where I store most network-related data (MACs, DNS, domain stuff, DHCP, CPE hardcoded passwds...) anyway, it's perfect — thanks!

I got the copy button below some boxes BTW — which didn't make any sense since I couldn't paste — but now if I assume correctly there'd be a paste button if I copy from another of these fields. It makes sense now. Thanks again.



I was going to reply much earlier without checking first. Thankfully I reconsidered because of the new line bit. I had mentioned double new line earlier because I thought it was well-known a single new line didn't cut it. This made me a little dubious about pasting from a spreadsheet so I tried it, in five six browsers:

Firefox, Chromium, Chrome, Vivaldi, Edge and Safari

It didn't work. Fortunately, I'm more stubborn and determined than a Schnauzer mini trying to get a toy and I refused to believe I'd been advised wrong so I pasted that in an IDE and made the lines into a single string, pasted it in the first of six browsers peeking behind the IDE — Firefox — and it took it right away, formatted it nicely! So, in the end, it might take a little more work than not, but it's much more preferable than the alternative of entering each value individually.

I have currently 4 or 5 OPNsense firewalls, (none of then work as firewalls BTW, mostly pre- and post-DNS stuff), some of them require the same settings minus one (self), and then again for IPv6, so it's very easy to make mistakes. This is a huge life saver! Thanks!

I'm trying to get around filling these boxes:


which are kind of a nightmare since they concatenate whatever you paste in.

I tried editing the file directly in the filesystem: in the example in the shot, BIND; but it turns own BIND doesn't read the config. The GUI seems to override the CLI.

Any idea how to filled this correctly to paste is enabled?

So far I've tried pasting the the line with [,],[;],[\],[\n\n](double new line space) and even [<br>]. Nothing seems to work.

Any ideas? :)

I found it!

I SSHed in the router and looked for the BIND configuration files. When I found it, right after I opened it I noticed the error; the IPv6 query source still had the old prefix so naturally it didn't match. The logs specifically mention IPv4 addressing so it kind of threw me off. After I changed it, it immediately came back online! :D

Thanks anyway.

I use BIND as some sort of DNS router between AD domain controllers and in-box Unbound, both listen on :53. The firewall has two interfaces; Unbound works exclusively on one of them, BIND doesn't have the option to choose interface but it set to use the addresses from the other. It had been working great for a long time until I had to change the IPv6 prefix.

That is the only thing different since it stopped working, at least. No other changes have been made. The logs are very... not really useful. The addresses are available, one of each family to each resolver/nameserver, and they are all responding, I'm testing from another subnet; ruling out gateway issues in the process. The "firewall" is not really a firewall, it's in router-only mode, filtering+NAT off.

I composed a little screenshot collage to best illustrate things — you might need to pan around on a laptop display.

Here's the link in case there are issues showing it: https://i.imgur.com/jcyCsUv.png


Any ideas what's wrong?