Messages

While I second that ZFS is preferable in principle, there are two caveats:

1. The ZFS defaults were different with older versions of OpnSense, such that writes occured more often.
2. There are applications that - if not moved to RAM disk - eat through SSDs quite visibly. Among those are RRD and Netflow.

With my first OpnSense installation (DEC750 on 23.x) and no RAM disk, I consumed half of my enterprise-grade's lifetime in one year.

That's impressive.  I was going to recommend the OP switch to an enterprise ssd for a larger endurance.

Exactly because ive set 853 tls for dns, and blocking outgoing port 53 connections from wan.

Outgoing firewall rules are almost never what you want.  You'd have to explain your setup and rules more for us to know what you're running into.

In the meantime, I wrote about this some time ago so you may find the posts helpful.

https://www.cjross.net/dns-security-and-adblock-with-opnsense-part-1/

https://www.cjross.net/dns-security-and-adblock-with-opnsense-part-2/

You can use the gpu slot for a nic in a Dell.  I'm using a pair of them with 25g mellanox cards for VM hosts.

The problem is that you need to disable pins on the nic.  The hardest part about it is getting the kapton tape just on the pins and not their neighbors.  See here:  https://www.dell.com/community/en/conversations/optiplex-desktops/optiplex-7060-dimm-slot-4-and-pcie-network-card/647f911bf4ccf8a8de2830b8?commentId=67a7ab9967c2593db3dbab73

That said, you're probably better off using a managed switch instead of multiple 10g ports on OPNsense.

What are your LAN speeds?  Often ISPs over provision their plans, so you can get over 1g speeds on a 1g plan if your connection has a 2.5g or faster port.  IIRC, I went from results similar to yours to 1.2g speeds by doing so.

I doubt you'd notice the difference, but it's fun nerd cred. :)

You may wish to have a read of my post from a few years ago.

https://forum.opnsense.org/index.php?topic=31705.msg153860#msg153860

The TLDR is the mlx4en doesnt play nicely with QNAP.
I ended up moving to a Netgear 10G switch instead and the mlx4en's have been solid since....

My upgrade plans got sped up due to the tariffs and I now have a Mikrotik replacing the QNAP.  Initial testing has been positive with no link problems during boot.

OPNsense was plugged into a non-combo port on the QNAP via DAC so I'm not sure why my results were different from yours.  But I appreciate the tip and glad I could finally resolve this issue.

I confirmed that OPNsense is connected not through a combo port.  It is using one of the first 4 via a Cisco DAC.

Updating to 25.1.5_4 caused the card to spam link down messages before switching to spamming an alternating link up/link down.  It did take a few iterations of the link down command to cause it to stop flapping and become stable.

You may wish to have a read of my post from a few years ago.

https://forum.opnsense.org/index.php?topic=31705.msg153860#msg153860

The TLDR is the mlx4en doesnt play nicely with QNAP.
I ended up moving to a Netgear 10G switch instead and the mlx4en's have been solid since....

I'll have to take a look through that thread.  I was going to say that none of my other mlx4en cards have problems but upon further consideration, this might explain the problems on my linux desktop.

What and how does everyone have their Connect-X cards connected to?

I've been running a Connect-X 3 dual port 10g card for several years and multiple versions of OPNsense.  Currently I have one port connected to a QNAP switch and the other connected to a Mikrotik (a recent addition).  Depending on the version and boot circumstances, the ports don't always come up correctly.  But once I go through some variation of link up/down and/or unplugging the DAC, it's completely stable.

Upgrading to 25.1 caused the port connected to the QNAP to spam Link Down messages to the console during the upgrade process.  Despite this, I was able to log in via the console and issue a link down command on the port.  Surprisingly, that not only stopped the Link Down spam but also immediately brought the link up and in a stable state with no further intervention.  The port connected to the Mikrotik did not appear to have any problems.

Upgrading to 25.1.3 saw the same QNAP connected port start to flap but quickly correct itself with no intervention.

I plan on eventually upgrading to a newer Connect-X card as I upgrade to 25/100G.  In the meantime, I will use this thread to document further issues as they occur, especially as the Mikrotik doesn't seem to be affected.

This is a known issue with the way the Unbound blocking is handled.  There have been talks about how to fix it but no one has had the time.

https://github.com/opnsense/core/issues/6722

https://forum.opnsense.org/index.php?topic=35218.msg171068#msg171068

I'm still seeing this issue in 24.7 but it seems that I no longer have to physically unplug the DAC.  Fiddling with the ifconfig up and down commands is enough to eventually get the port to stop flapping.

Oddly, I've enabled the second port as a trunk to a managed switch and I haven't seen any flapping with it.  Looking at the ifconfig results, the only differences I see are in the vendor line.  The working port shows OEM while the problem port shows CISCO-MOLEX.  The OEM listed cable is a 10GTek DAC.

When I have some time I'll try swapping the Cisco DAC for a 10GTek.

Name: modulename_load
Value: YES

You replied while I was busy updating my post with the results of my testing that. :D

Well, this is sort of self-documenting in /boot/loader.conf:

https://github.com/opnsense/core/blob/0adece8d3e165acc0ba3bb2e1d8f0e6593dd8c41/src/etc/rc.loader.d/00-banner#L1-L6

Cheers,
Franco

I do have the appropriate line in /boot/loader.conf.local and up until 24.7 it's always worked.  When I look at /boot/loader.conf I don't see the mlx load line.

I went into system tuneables and added a new tuneable with a tuneable of mlx4en_load and a value of YES.  Upon saving that and applying changes the mlx4en_load line showed up in the tuneables section of /boot/loader.conf  Upon reboot, the module was loaded and my interfaces came up correctly.

It appears that the mechanism to process /boot/loader.conf.local was broken in the changes from 24.1 to 24.7.  I'm guessing that the step importing it into the main file got accidentally removed or commented out, but until I have a chance to dig through the code, I can't know for sure.

I've been running ConnectX cards for a while now and they've worked pretty well once you add the load command to the boot config.

https://www.routerperformance.net/opnsense/mellanox-connecx-management-in-opnsense/

After updating to 24.7 this doesn't seem to work anymore.  I have to manually log into the box and issue the mlx4en load command and then force an interface reload before it starts working.  This works until I reboot at which point I have to repeat the process.

I've checked and I still have the load command set to yes but it's not properly starting the card.  Any suggestions for what to check?

Just experienced my first connection issue on the primary WAN.  The user experience was that everything hung or showed offline for a bit but then reconnected fine.  Faster recovery than waiting for the primary WAN to recover on it's own usually takes but still noticeable.

The logs tell an interesting story.  It only takes five seconds to from packet loss to being marked as down but the routing reconfiguration happens at the same time as them being marked down, so it's hard to confirm if the gateway switching setting worked or not.  I'm not sure why it says that it's keeping IP6 on WAN2 as WAN is higher priority.

A little over two minutes later WAN switches to delay for IPv4 with no change on IPv6, but the routing changes for both back to WAN.  Then ten seconds later WAN goes completely clean and the routing configuration is run again but with no changes.

Code Select Expand

0s
MONITOR: WAN_DHCP (Alarm: none -> loss RTT: 11.4 ms RTTd: 1.9 ms Loss: 12.0 %)
MONITOR: WAN_DHCP6 (Alarm: none -> loss RTT: 35.7 ms RTTd: 54.1 ms Loss: 12.0 %)

5s
ALERT: WAN_DHCP (Alarm: loss -> down RTT: 11.5 ms RTTd: 1.9 ms Loss: 21.0 %)
ALERT: WAN_DHCP6 (Alarm: loss -> down RTT: 34.4 ms RTTd: 53.5 ms Loss: 21.0 %)
reconfiguriging routing due to gateway alarm
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
/usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_DHCP, WAN_DHCP6
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on WAN2
/usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to WAN2_DHCP_GW
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on WAN2
/usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to WAN2_DHCP6_GW

130s
ALERT: WAN_DHCP (Alarm: down -> delay RTT: 485.4 ms RTTd: 1473.6 ms Loss: 0.0 %)
reconfiguriging routing due to gateway alarm
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
/usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to WAN_DHCP_GW
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan
/usr/local/etc/rc.routing_configure: ROUTING: setting inet6 default route to WAN_DHCP6_GW

141s
MONITOR: WAN_DHCP (Alarm: delay -> none RTT: 10.8 ms RTTd: 0.8 ms Loss: 0.0 %)
ALERT: WAN_DHCP6 (Alarm: down -> none RTT: 34.9 ms RTTd: 71.8 ms Loss: 1.0 %)
reconfiguriging routing due to gateway alarm
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on wan
/usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to WAN_DHCP_GW
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on wan
/usr/local/etc/rc.routing_configure: ROUTING: keeping inet6 default route to WAN_DHCP6_GW


Also, there's a typo in the logs.  "reconfiguriging"

Maybe exactly this

WAN Failover
WAN failover automatically switches between WAN connections in case of connectivity loss (or high latency) of your primary ISP. As long as the connection is not good all traffic will be routed of the next available ISP/WAN connection and when connectivity is fully restored so will the routing switch back to the primary ISP.

For me it means you have "preemption" when using GW groups vs when you have not.

This is the part I'm confused about.  In the limited testing I've done, when the original gateway was restored all of the routing switched back to it.  Or are you saying that groups are the only way to have the switch happen due to latency or packet loss and not complete failure?  Because the docs imply that that's determined by the "Allow default gateway switching" setting.  My connection has been remarkably stable since I set up the second link so I can't confirm that it's working correctly.

> and when connectivity is fully restored so will the routing switch back to the primary ISP.

The docs are correct, but lack finesse in the wording WRT established connections and perhaps the use of sticky-address option.

I would assume established connections would continue on the backup gateway but any new connections would be on the primary.  Or do you mean something else?

I'm not too concerned about established connections as they will eventually end and things will migrate to the primary then.