Messages

Okay, hopefully this will help someone someday.  After more than 10 hours of testing things over 2 days, I think I got it figured out.

First, get your Gateways setup how you want them to be with priority, etc.

Now here's how I got it to work on 25.7_4:

1.  Verified that "Allow default gateway switching" was checked by going to System -> Settings -> General.  Don't forget to Save if you changed the setting. (I did this because I'd rather let everything go out whatever is the "default" *except* my Web Switch Pro.
2.  Go to Firewall -> Settings -> Advanced and check "Skip rules when gateway is down".  Don't forget to Save.
3.  Create 4 rules on your LAN interface.

# Action Interface Protocol Source Destination Gateway Description
1 Pass LAN ICMP (Web Pro Switch IP) 1.1.1.1 Main_ISP_Gateway Web Pro Switch to 1.1.1.1 (Enforce Main ISP)
2 Block LAN ICMP (Web Pro Switch IP) 1.1.1.1 * (default) Block 1.1.1.1 if Main ISP Fails
3 Pass LAN ICMP (Web Pro Switch IP) 8.8.8.8 Secondary_ISP_Gateway Web Pro Switch to 8.8.8.8 (Enforce Secondary ISP)
4 Block LAN ICMP (Web Pro Switch IP) 8.8.8.8 * (default) Block 8.8.8.8 if Secondary ISP Fails


Make sure your 4 rules above are before your "Default LAN to any" rule.  Apply and Save.

Rules 1 and 3 work when the associated Gateway is up.  Rules 2 and 4 prevent the traffic from being redirected to the other gateway on failure of that ISP.  The "default" gateway is basically a "gateway of last resort".  By blocking that traffic, you're blocking the pings from being redirected to the "gateway of last resort".

I have 2 ISPs, once is via cablemodem, the backup is a "wireless fiber".  Basically its to help me avoid downtime if my cable internet goes down.  I have learned that my wireless fiber tends to have issues randomly, and the "fix" is to power cycle the PoE tower on my roof.  I'd like this to "autocorrect itself", so I bought one of these:

https://dlidirect.com/products/new-pro-switch

Seems to work well, and I can manually trip it appropriately.

So my plan was to have the Pro Switch ping 1.1.1.1 via my cable internet, and 8.8.8.8 via my wireless fiber.  If pings fail for 10 minutes, reboot the associated.  So I set it up, and via gateway settings in my LAN interface, I have it direct to the appropriate gateway.  Works great, and the traffic goes where it's supposed to.

The problem, if I make the wireless fiber go offline, then those ICMP packets start going out my cable internet and since those succeed, the Pro Switch never knows that anything is wrong, so it won't power cycle the interfaces.

I believe that my settings are to blame.  Under the gateway configuration I have both ISPs with my cable internet with a priority of 10, and my wireless fiber with a priority of 50.  I already had 2 gateway groups, one with my cable internet as a tier 1 and wireless fiber as tier2, and another with the opposite.  I did even create gateway groups where each of my isps is tier 1 independently and tried setting that in opnsense but I still cannot get the traffic to fail.

After some back and forth with Googling and discussion with ChatGPT, apparently opnsense seems to choose the "good gateway" regardless.  Can someone explain this to me?  I don't think this is the answer despite that being the observed behavior.  I'd like the pings to actually fail if the associated gateway is not working.  This is my first time trying to force basically "all traffic out one ISP", but also allow "some traffic out the other ISP, and for this traffic to fail if the ISP fails".

Found the problem.  Added qat_load=YES to the system years ago.  Apparently it's no longer necessary or somehow is incompatible with FreeBSD 14.  In any case, system is now working fine on the latest version!

Quick little update...

I installed 24.7, updated it to 24.7.12_4 and imported the config file.  No issues.

Updated to 25.1, kernel panic.

So definitely something about 25.1 isn't thrilled with my 24.7 settings.

I have an opnsense box I've been managing for about 5 years.  Supermicro C3758 based system.  I decided that since an upgrade failed long ago, when 25.1 came out I'd do a fresh install and import my config file.  While I was onsite I did a BIOS and IPMI update.  BIOS was factory defaulted and reconfigured appropriately.  System booted up 24.7.14 (whatever the last version was) without issues.

I then decided to install 25.1 which happened without issues.  I then uploaded the config file and first reboot was fine.  However if I reboot again (even if I change nothing) the box kernel panics while booting.

So I reinstalled 25.1 again and updated it to 25.1.3 without problems.  Did a few reboots and no problems.  Then I uploaded my config file and after 2 reboots it kernel paniced again on bootup.

Here's a screenshot of the panic.  Can anyone shed some light on this?  I *really* don't want to have to start over from scratch, it has quite a few VPN links and other things, and if this can give me a direction to go, I'd prefer to go that way before having to start with a clean config.

Note that I hadn't gotten to install any of the plugins yet because I was still in basic testing when I realized this problem.

Okay, so not as easy as doing something from the CLI, but this does work.

First, setup everything in General Settings and Alert Settings.  Under your recipient, make sure the event "Monit instance changed" is checked.  Save and click Apply.  Poof, you'll get an email because that alone will trigger monit to reload itself, and the instance changing will send you an email.

Of course, you can also trigger it anytime now if you keep the "Monit instance changed" setting enabled by simply reloading Monit.

As I'm new to using monit heavily, this took me a long time.  Way too long.

Is there a way to have monit send a test email to make sure everything works properly?  I'm 99% sure monit isn't actually working for me, but I can't figure out how to tell monit to send a test email.

I even tried a service test by creating one with condition 'failed host 127.0.0.1 port 1234', but then after saving it and such, I went to my recipient under alert settings, but I can't even select my test condition.

I was hoping I could just go to the CLI and do something like "monit emailtest" and have an email get sent

Nobody?

Wow.. I didn't think I'd stump the collective sum of knowledge from opnsense!

I'm relatively new to dtrace, but I am convinced this *should* work.  I'm running opnsense 24.7.10_2-amd64 and I'm trying to run this but getting the below error:

Code Select Expand

# dtrace -n 'dtrace:::BEGIN'
dtrace: invalid probe specifier dtrace:::BEGIN: "/usr/lib/dtrace/mbuf.d", line 118: syntax error near "caddr_t"

Running this on a pure FreeBSD system does work.  Am I crazy?  Should this not work?

Ultimately, I'm having performance issues with openvpn, and I'm trying to actually validate that aesni *is* in fact being used.  Also I wanted to see if the kernel module if_ovpn is used.  Just trying to rule out optimizations that aren't being used, etc before I try digging further.

I can verify the modules are loaded, but I'm looking to prove they are actually being used.

Thanks!

And... it's official.  I am stupid.  That worked!  Somehow it never crossed my mind that the checkbox would work that way.  Thank you!

I switch to Kea from ISC for DHCP services.  In ISC it has a very clear field where you can set the DNS server to use.  In my case, I want to use a pihole for one of my networks.  How can I set this?  I looked in the GUI and searched around and either the function doesn't exist in the WebGUI, or I'm stupid and/or blind.

Thanks!

Okay, so I fixed it. I believe the system was all messed up because I had told the interfaces to "prevent interface removal".  Here's what I did:

1.  Uninstalled Zerotier plugin.
2.  Went to the interfaces and unchecked the "prevent interface removal" for each of the zerotier interfaces I had.
3.  Removed the interfaces from the system.
4.  Reinstalled Zerotier plugin.
5.  Reconfigured everything as necessary in the WebGUI.  This included adding the interfaces again, etc.
6.  Went into zerotier's website and added the "new" device to the appropriate network and removed the "old" device from the appropriate network.

I can't remember why I checked 'prevent interface removal' in the first place.  I think there was some odd stuff with zerotier making a new interface on every reboot or every update or something, and instead of having to add my "new" device to the network every time, I simply prevent its removal.

Hello everyone.  So I decided to replace my boot device in opnsense with a mirrored set with ZFS.  I was on the last version of 23.1, and since 23.7 had just come out, I made the choice to go ahead and install 23.7 and then import my config file.  I figured it would be smooth sailing.

Not so much.

Zerotier has been broken since that day.  I haven't had a chance to troubleshoot since it was a low priority, but nwo I need to get it working again.

Looking in the WebGUI of opnsense under the Zerotier section, everything looks normal.

However the zerotier interfaces are not being created.  ifconfig doesn't show them.  I then went to Interfaces -> Assignments and opt7 and opt8 (my two zerotier interfaces) show as "missing" (which I already knew) but I can't get them to be recreated.

I tried uninstalling and reinstall the os-zerotier plugin along with reboots.  I've even installed several of the opnsense updates (along with appropriate reboots) and the interfaces are still not being created.

I also tried creating a new network to see if it would create a new interface.  Still no.  I'm at a loss to know how to either recreate the interfaces that I had or have it create new ones.

I will say that if I log into the Zerotier website to look at device last login, my opnsense box is over 30 days since last login, corresponding with when I did the reinstall.

Any ideas?

Thanks!

@JamesFrisch

Can you provide specs on your opnsense system?  Is it virtualized?

After posting this I noticed the last entry for the OP wasn't a signature but was what he did to fix it.  I tried both US repos, and no change.  I then changed to the DE repo the OP used, and its updating right now.

Sounds like some kind of problem with the repos or something.