Topics

Just looking over my firewalls in central management, one firewall stands out above all the others with RAM usage at 85%. Every other box is 25-35%. It seems to be ARC, which I believe is part of ZFS.
All firewalls are on 26.4 or 25.10 and have 8GB of RAM. Why would ARC usage be so high on one firewall?

OpnSense 25.10.
I have an unreliable 4G router on one WAN and reliable internet on the other.
I am outside the firewall.
I want a NAT policy that gives me access to 4G router's web interface via the reliable WAN.
For this to work, it needs to translate the source and the destination IP when I make the connection but I don't see a way to do this in the OpnSense web interface.
Can this be done somehow?

OPNsense 25.1.12-amd64
Using IPsec Connections, not IPsec Tunnel Settings.
Route-based VPN tunnel to Azure.
Tunnel mode.
Having the SA enabled seems to result in a new SA being created every 30-60 seconds or so.

[the one MODP_2048 entry below is a working tunnel, not related to this]

Code Select Expand

# ipsec statusall | grep bytes
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
0a06bd58-d311-4be4-b850-e21ee2405c88{45}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{36}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 248 bytes_o, rekeying in 6 hours
dafa61b0-baa3-4120-85bf-b6bf37d00487{24}:  AES_CBC_256/HMAC_SHA2_256_128/MODP_2048, 14657534 bytes_i (142048 pkts, 0s ago), 28693760 bytes_o (127444 pkts, 0s ago), rekeying in 7 minutes
0a06bd58-d311-4be4-b850-e21ee2405c88{62}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{51}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{64}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{63}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{49}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 124 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{42}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{39}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 124 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{44}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{37}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 124 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{68}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 2604 bytes_o, rekeying in 7 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{57}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 248 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{50}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 124 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{33}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{28}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 124 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{58}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 372 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{53}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{38}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{54}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{43}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{35}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 372 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{34}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{55}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{32}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{30}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{48}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 124 bytes_o, rekeying in 7 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{41}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{29}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{59}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 124 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{56}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{52}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{46}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{31}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{47}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 496 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{65}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 0 bytes_o, rekeying in 6 hours
0a06bd58-d311-4be4-b850-e21ee2405c88{61}:  AES_CBC_256/HMAC_SHA2_256_128, 0 bytes_i, 124 bytes_o, rekeying in 6 hours
Some of the alleged SAs have TX bytes on, none have any RX.
If I disable the policy, it stops creating SAs but none actually get deleted.
Behaviour is a bit puzzling, what is going on here? Clearly, my end thinks the SA is up if there are TX bytes, but why does it then bring up another?

Tried my first upgrade with Central yesterday. It seemed to work, only a small step from 25.10 to 25.10_2.
But there doesn't seem to be any useful information in Central about this. There is no log and after clicking Upgrade I wasn't really sure what it was doing. The "Ready to upgrade" and "Upgrade in progress" icons are the same colour which seems like an obvious missed opportunity. We know that new colours are quite cheap nowadays.

What is the meaning of the Expires column? No tooltip, no units, not present in screenshot on current documentation.

Scenario: on TNET interface we have a L3 routed network, one example network is 10.10.45.0/24. Routes to these networks are configured in System: Routes: Configuration.

Outbound NAT policies for these networks have been created manually, I assume they're getting hits because at the far end I can see my test traffic originating from the expected IP.

Here's the problem - the replies are being dropped because of "Default deny / state violation rule". My guess here is that the firewall knows the matching state for these rules, otherwise it wouldn't know the private IP they're for. So what is the real issue?
Devices in the TNET subnet work fine, it's just the routed subnets that don't work.

Code Select Expand

TNET        2025-06-17T08:42:25    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:17    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:09    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:05    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:42:01    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:59    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:57    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:56    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:55    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
TNET        2025-06-17T08:41:54    91.2.119.28:80    10.10.45.49:32975    tcp    Default deny / state violation rule   
WAN        2025-06-17T08:41:54    45.76.130.220:14043    91.2.119.28:80    tcp    let out anything from firewall host itself (force gw)   

Three times in past 4 weeks, disk has filled up on my OpnSense and every time it has been something to do with Zenarmor.
First one was cron jobs that fail to send an error message once per minute. Fixed that.
Second one was mongodb. Sort of fixed that, but haven't actually managed to get the DB engine working again [but at least it's not eating the disk].
This time it is /usr/local/zenarmor/log/active/worker0 logging at about 1MBps. It's mostly lines like this:

2025-03-10T10:01:53.378794 WARN ArrayStream was full, str: osver pos: 8191
2025-03-10T10:01:53.378802 WARN ArrayStream was full, str: ":" pos: 8191
2025-03-10T10:01:53.378811 WARN ArrayStream was full, str: "} pos: 8191
2025-03-10T10:01:53.378820 WARN ArrayStream was full, str: ," pos: 8191
2025-03-10T10:01:53.378828 WARN ArrayStream was full, str: remote_device pos: 8191
2025-03-10T10:01:53.378847 WARN ArrayStream was full, str: ":" pos: 8191
2025-03-10T10:01:53.378855 WARN ArrayStream was full, str: " pos: 8191
2025-03-10T10:01:53.378864 WARN ArrayStream was full, str: ," pos: 8191
2025-03-10T10:01:53.378871 WARN ArrayStream was full, str: community_id pos: 8191
2025-03-10T10:01:53.378879 WARN ArrayStream was full, str: ":" pos: 8191
2025-03-10T10:01:53.378892 WARN ArrayStream was full, str: 1:NLGH3mmPeENTT1aXwYWl5XLicUw= pos: 8191
2025-03-10T10:01:53.378908 WARN ArrayStream was full, str: " pos: 8191
2025-03-10T10:01:53.378916 WARN ArrayStream was full, str: ," pos: 8191
2025-03-10T10:01:53.378924 WARN ArrayStream was full, str: handshake_result pos: 8191
2025-03-10T10:01:53.378931 WARN ArrayStream was full, str: ":" pos: 8191
2025-03-10T10:01:53.378939 WARN ArrayStream was full, str: None pos: 8191
2025-03-10T10:01:53.378964 WARN ArrayStream was full, str: " pos: 8191
2025-03-10T10:01:53.378972 WARN ArrayStream was full, str: }

What is it complaining about here?

How is

Code Select Expand

local host is behind NAT, sending keep alives
determined? Is it due to what the far-end says ["you are behind NAT"], or is it some other heuristic? I am seeing it in a scenario where there is definitely no NAT at my end ["local host"] and almost certainly not at the far end.

Firewall with 2x WAN interfaces with public IPs. 1 LAN with private IP. Only routes to far end of tunnel are using WAN interfaces.
I see that "local host is behind NAT, sending keep alives" even though there is no NAT involved. How is the firewall determining that NAT is in use?

Two routers set up in HA, two WANS /29 and /30. Standby router cannot use gateway on /30 as there aren't enough IPs, so Monit constantly complains about it being down.
Can I get Monit to ignore the /30 WAN? So long as at least one WAN works, that's "good enough".

I have two WANs. Each WAN has a gateway configured.
I want to be able to manage the firewall remotely to either public IP address.
I have spent quite a bit of time fiddling with this.
Ping works fine to both WANs. I could only get HTTPS working to one at a time.
I have found that the key to fixing this was explicitly setting a reply-to on the management access rule.
The rule consists of a source alias [a list of remote management public IPs], destination of "This Firewall" and a service of Any.
If I leave "reply to:" as "default", it doesn't work. The help for this option says:

Determines how packets route back in the opposite direction (replies), when set to default, packets on WAN type interfaces reply to their connected gateway on the interface (unless globally disabled). A specific gateway may be chosen as well here. This setting is only relevant in the context of a state, for stateless rules there is no defined opposite direction.

This makes sense - ICMP is stateless and works regardless of this setting. But HTTPS does not work unless I explicitly set reply-to to the interface's default gateway here. This contradicts what the documentation says.
There is a hint about "unless globally disabled", but what is that setting called and where is it?
Even specifying the gateway explicitly with the Gateway setting does not make this work.

/var/db/mongodb/mongod.log is full of this:

Code Select Expand


2024-07-05T11:08:19.312+0000 I NETWORK  [conn742973] end connection 127.0.0.1:9158 (6 connections now open)
2024-07-05T11:08:20.414+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:9165 #742975 (7 connections now open)
2024-07-05T11:08:20.414+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:9166 #742976 (8 connections now open)
2024-07-05T11:08:20.414+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:9167 #742977 (9 connections now open)
2024-07-05T11:08:20.414+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:9168 #742978 (10 connections now open)
2024-07-05T11:08:20.415+0000 I NETWORK  [conn742975] received client metadata from 127.0.0.1:9165 conn742975: { driver: { name: "mongo-go-driver", version: "v1.7.0" }, os: { type: "freebsd", architecture: "amd64" }, platform: "go1.19.3" }
2024-07-05T11:08:20.415+0000 I NETWORK  [conn742976] received client metadata from 127.0.0.1:9166 conn742976: { driver: { name: "mongo-go-driver", version: "v1.7.0" }, os: { type: "freebsd", architecture: "amd64" }, platform: "go1.19.3" }
2024-07-05T11:08:20.415+0000 I NETWORK  [conn742977] received client metadata from 127.0.0.1:9167 conn742977: { driver: { name: "mongo-go-driver", version: "v1.7.0" }, os: { type: "freebsd", architecture: "amd64" }, platform: "go1.19.3" }
2024-07-05T11:08:20.415+0000 I NETWORK  [conn742978] received client metadata from 127.0.0.1:9168 conn742978: { driver: { name: "mongo-go-driver", version: "v1.7.0" }, os: { type: "freebsd", architecture: "amd64" }, platform: "go1.19.3" }
2024-07-05T11:08:20.716+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:9171 #742979 (11 connections now open)
2024-07-05T11:08:20.716+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:9172 #742980 (12 connections now open)
2024-07-05T11:08:20.716+0000 I NETWORK  [conn742979] received client metadata from 127.0.0.1:9171 conn742979: { driver: { name: "mongo-go-driver", version: "v1.7.0" }, os: { type: "freebsd", architecture: "amd64" }, platform: "go1.19.3" }
2024-07-05T11:08:20.716+0000 I NETWORK  [conn742980] received client metadata from 127.0.0.1:9172 conn742980: { driver: { name: "mongo-go-driver", version: "v1.7.0" }, os: { type: "freebsd", architecture: "amd64" }, platform: "go1.19.3" }
2024-07-05T11:08:20.720+0000 I NETWORK  [conn742980] end connection 127.0.0.1:9172 (11 connections now open)
2024-07-05T11:08:20.720+0000 I NETWORK  [conn742978] end connection 127.0.0.1:9168 (10 connections now open)
2024-07-05T11:08:20.720+0000 I NETWORK  [conn742975] end connection 127.0.0.1:9165 (9 connections now open)
2024-07-05T11:08:20.721+0000 I NETWORK  [conn742976] end connection 127.0.0.1:9166 (8 connections now open)
2024-07-05T11:08:20.721+0000 I NETWORK  [conn742977] end connection 127.0.0.1:9167 (7 connections now open)
2024-07-05T11:08:20.721+0000 I NETWORK  [conn742979] end connection 127.0.0.1:9171 (6 connections now open)
2024-07-05T11:08:21.831+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:9177 #742981 (7 connections now open)



You can see that there were 18 events in 1 second here. I have tried restarting mongodb but it didn't make any difference. Additionally, the log file does not get rotated. I can see that it was rotated once but then never again.

Code Select Expand


root@router:/var/db/mongodb # ls -alh mongod.log*
-rw-------  1 mongodb  mongodb   336M Jul  5 11:08 mongod.log
-rw-------  1 mongodb  mongodb   4.8K May 10 11:15 mongod.log.2024-05-10T11-15-15



My /var/spool/clientmqueue had literally hundreds of thousands of files in it. They are all just a UUID. The message info indicates that they are caused by this cron job:

Code Select Expand


# crontab -u nobody -l
# or /usr/local/etc/cron.d and follow the same format as
# /etc/crontab, see the crontab(5) manual page.
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
#minute hour    mday    month   wday    command
# Origin/Description: Zenarmor/Zenarmor periodicals
*       *       *       *       *       /usr/local/sbin/configctl -d 'zenarmor periodicals' '>' /dev/null '2>&1'



Note the quotes around the >.
This does not work as intended:

Code Select Expand


# /usr/local/sbin/configctl -d 'zenarmor periodicals' '>' /dev/null '2>&1'
3fc7e3fa-5fbb-46f6-9069-939c2b154e05



This does:

Code Select Expand


# /usr/local/sbin/configctl -d 'zenarmor periodicals' > /dev/null '2>&1'
#


I don't really know configctl but it also has a -q option to suppress stdout, perhaps that would make sense here.

Reporting > Health > Traffic
Why are the interface graphs in bytes per second not bits per second? Nobody measures network throughput in Bps. Doing it in Bps is almost perverse!

Was "Sticky connections" renamed to "Sticky outbound NAT" at some point?

I see it referenced here:

https://github.com/opnsense/core/issues/2170

but can't find that setting.

Trying to interoperate with Sonicwall firewalls. They allow to create an IPsec tunnel interface and create route policies on it, without assigning IPs to the tunnel, or specifying any local/remote subnets. I am not sure about the terminology of this, but I think this would be an unnumbered VTI.
I don't see a way to do this with the OpnSense UI, but it might just be that I am not familiar with how the UI works. If I choose "Route based" then I have to put IP addresses in, otherwise:

"A valid local network IP address must be specified."

Is is it possible to do this?

Can the interface descriptions seen with SNMP be synced with what is in the web interface?
Eg every interface is seen as the device name:

Code Select Expand


RFC1213-MIB::ifDescr.1 = STRING: "igb0"
RFC1213-MIB::ifDescr.2 = STRING: "igb1"
RFC1213-MIB::ifDescr.3 = STRING: "igb2"
RFC1213-MIB::ifDescr.4 = STRING: "igb3"
RFC1213-MIB::ifDescr.5 = STRING: "igb4"


but in UI we have the Identifier and Description values which would be more useful than the device name. Running OpnSense on a variety of hardware means that different systems show different names, even though all systems may have LAN, WAN1, WAN2, etc.

Have been experimenting with using a ZT tunnel as a default route for internet traffic.
It works OK once enabled with

Code Select Expand

zerotier-cli set <networkId> allowDefault=1

but then after a reboot, it's broken - Zerotier cannot establish a connection at all and no traffic is passed. Flip it back with allowDefault=0 , reboot and internet access is restored [albeit no longer over the ZT tunnel].
It is as if Zerotier is trying to use its own default route to establish connectivity for its own traffic, which seems like a silly defect.

We have had some success with this deployment scenario using Teltonika RutOS devices, but they simply don't have the horsepower to handle the throughput we need, hence looking at doing this on Opnsense [and I must say I am pretty damn impressed with Opnsense so far, other than this specific issue].