Messages

Hello OPNsense,

I have two DEC2752 units configured in HA that are being used for a new remote office network build. These two units were purchased less than 1-year ago and the configuration on them is quite basic. No IPv6. No unbound/dnsmasq configuration. I've got an IPSEC vpn connection from the VIP addresses to my primary site. DHCP/DNS (at this time) is handled at my primary DC.

So the OPNsense firewall cluster is just acting as a secure gateway with a site to site tunnel. Nothing fancy.

Last week, I tried to connect to both units. Master (10.103.0.1) responds fine. Secondary (10.103.0.2) is sluggish. The web gui fails to load properly.

The IPSec VPN tunnel is functional.

When I started to investigate what is going on with the secondary unit, I see a ton of errors via CLI:

Code Select Expand

root@FW02:~ # swap_pager: out of swap space
swp_pager_getswapspace(10): failed
swp_pager_getswapspace(3): failed
swap_pager: out of swap space
swp_pager_getswapspace(4): failed
swap_pager: out of swap space
swp_pager_getswapspace(1): failed
swp_pager_getswapspace(4): failed
swp_pager_getswapspace(6): failed
swap_pager: out of swap space
swp_pager_getswapspace(1): failed
swap_pager: out of swap space
swp_pager_getswapspace(2): failed
swap_pager: out of swap space
swp_pager_getswapspace(22): failed
swp_pager_getswapspace(20): failed

If I reboot the secondary firewall, the OS loading process seems slow. Once I get in and press 8 for CLI, I don't have much time before it starts to bog down.

running df -h I get:

Code Select Expand

root@FW02:~ # df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default    222G     10G    212G     5%    /
devfs                 1.0K      0B    1.0K     0%    /dev
/dev/gpt/efifs        256M    645K    255M     0%    /boot/efi
zroot/tmp             212G    224K    212G     0%    /tmp
zroot                 212G     96K    212G     0%    /zroot
zroot/var/log         212G    164M    212G     0%    /var/log
zroot/var/audit       212G     96K    212G     0%    /var/audit
zroot/usr/home        212G     96K    212G     0%    /usr/home
zroot/usr/ports       212G     96K    212G     0%    /usr/ports
zroot/usr/src         212G     96K    212G     0%    /usr/src
zroot/var/crash       212G     96K    212G     0%    /var/crash
zroot/var/mail        212G    144K    212G     0%    /var/mail
zroot/var/tmp         212G     96K    212G     0%    /var/tmp
devfs                 1.0K      0B    1.0K     0%    /var/dhcpd/dev


When I take a look at top -o res, I see high swap:

Code Select Expand

root@FW02:~ # top -o res
last pid: 13953;  load averages:  4.39,  2.72,  1.50    up 0+00:10:36  15:54:20
63 processes:  25 running, 38 sleeping
CPU: 53.4% user,  0.0% nice, 44.4% system,  2.2% interrupt,  0.0% idle
Mem: 5721M Active, 702M Inact, 195M Laundry, 754M Wired, 2056K Buf, 493M Free
ARC: 257M Total, 184M MFU, 66M MRU, 610K Anon, 1329K Header, 5222K Other
     224M Compressed, 324M Uncompressed, 1.44:1 Ratio
Swap: 8418M Total, 6101M Used, 2317M Free, 72% Inuse, 314M In
swap_pager: out of swap spaceiled
swp_pager_getswapspace(10): failedIZE    RES STATE    C   TIME    WCPU COMMAND
85082ager_getswapspace(48: faile 824M   555M RUN      2   0:04  32.18% php
 7616pager: out of swap42paceile1069M   490M RUN      2   0:14  30.59% php-cgi
 5271ager_getswapspace(48): faile488M   393M CPU0     0   0:02  24.03% php-cgi
28393 root          1  21    0   548M   392M select   2   0:15   0.01% php-cgi
 5177 root          1  48    0   494M   391M RUN      1   0:02  31.91% php
 9260 root          1  24    0   584M   386M select   2   0:06   0.00% php-cgi
91350 root          1  50    0   516M   368M RUN      0   0:03   6.38% php
 2079 root          1  42    0   440M   338M RUN      2   0:03  43.99% php-cgi
 6068 root          1  24    0   538M   291M RUN      3   0:02   9.70% php-cgi
 2260 root          1  44    0   751M   270M RUN      3   0:09   9.23% php-cgi
63351 root          1  24    0   726M   266M RUN      3   0:03   7.69% php
28926 root          1  20    0   634M   236M select   3   0:16   0.00% php-cgi
 8504 root          1  20    0   584M   236M select   2   0:06   0.00% php-cgi
 1583 root          1  24    0   792M   224M CPU1     1   0:07  23.80% php-cgi


I have tried to clean up some logs that I had in /var/log and reboot but that didn't help.

These are the only packages I have installed:

Code Select Expand

root@FW02:~ # pkg info | grep os-
os-OPNBEcore-1.7_3             OPNsense Business Edition add-ons
os-OPNcentral-1.12_2           OPNsense central management
os-dmidecode-1.2               Display hardware information on the dashboard
os-etpro-telemetry-1.8         ET Pro Telemetry Edition



What I'm struggling to understand is why would my primary unit be working just fine and my secondary having this issue. I have been evaluating OPNsense as a use case for our remote site(s) but my configurations seem a bit light.

I do enable logging on my firewall rules but I don't have many rules at all.
I have a total of 11 VHIDs and on my primary unit at this time, my swap is 0.0%, memory used is 1047mb/arc 1103mb and my disk utilization is 1%.

When checking my snapshot, I see that bectl list shows default as 10.4G.

Code Select Expand

root@FW02:~ # bectl list
BE      Active Mountpoint Space Created
default NR     /          10.4G 2025-04-17 09:23

When I compare that to my primary/active unit, it shows 1.29G

My concern here is that this is some kind of hardware failure but I'm not sure how to confirm that or check.


The web interface is unresponsive that I can't even go in and create a recent backup. The page for backups won't load. I should have a latest backup but I'm just pointing out as to how locked up the interface is.

I can't recall what was done in the past 1-3 weeks but it wouldn't be much. These firewalls are waiting for me to rebuild a new IPSEC vpn connection from my primary location so I haven't performed any recent configuration changes to them to my knowledge.

I've captured screenshots and I can get further logs from the POST sequence and OS bootup if it helps.


Thank you,

You configure a CARP address on the Internet facing (WAN) interface and use that as the endpoint for your IPsec tunnel(s). Connectivity will move with the CARP address in case the primary node fails.

Did you setup your HA cluster following the documentation? So you have a HA/CARP address on all interfaces?

Yes I've got my HA cluster configured as per the documentation. My concern is not the OPNsense node failing but the other end. Be it a hardware failure or ISP being down, I am trying to get my OPNsense cluster to have a secondary IPSEC connection going to the opposite site, to their secondary connection.

Hello everyone,

I have two Deciso DEC2752 units in a HA configuration that I am soon about to deploy.

At this moment I am nearly ready except I need to figure out how to configure my OPNsense deployment so that if my primary IPSEC VPN connection goes down, the secondary IPSEC VPN connection will establish.

The remote end are two Versa-SDWAN appliances. Versa #1 has one ISP connection and Versa #2 has the other IPS connection. Both ISP connections are for separate ISPs for redundancy.

Right now, my OPNsense cluster is configured for IPSEC VPN to Versa #1 Public IP. I can power off one of my OPNsense units and the other kicks in as expected.

But for whatever reason I cannot seem to figure out how to apply some kind of metric/weight to keep the primary IPSEC tunnel active and failover to the other IPSEC tunnel if my primary versa is down.

Would anybody be able to point me into the direction on what to read or how to accomplish this?


Thank you!

Hello everyone,

Our organization uses Cisco Umbrella for web filtering. Our our primary site (Home Office) I have two Cisco Umbrella Virtual Forwarders that are used for DNS resolution.

I am working on configuring and testing two DEC2752 units in a HA configuration for a remote office. The remote office will connect to Home Office via IPSEC site to site VPN connection.

This remote office is small enough that there is not and won't be any server onsite. Due to this, I want our web traffic from the remote site to traverse the VPN tunnel back to the home office.

Now, in the event that the VPN tunnel is down, I want to use Cisco Umberella public DNS IPs.

The remote office staff get their IP addressing/DNS information VIA AD/DHCP. This of course won't work when the tunnel is down.

I was thinking that I might be able to configure the public DNS IP addresses in the OPNsense System/General settings but I am not sure if that would help.

Within OPNsense, I have not configured Unbound/DNSMasq.

Any suggestions with my current configuration on what I can do to keep web traffic flowing if IPSEC is down?


Thank you,

Well I guess I'll document this myself.

Frusterated by this, I factory reset both devices, logged in, deleted the LAN interface and setup my core VLAN. Setup my public IP addresses for each device, the gateways and left them connected to the internet.

Not configured in HA, both devices show regular activity for their WAN gateways. Again, not going to disable gateway monitoring as I want to see the status/functionality of my gateway.

FW #1:


FW #2:



So this looks normal to me. I know my ISP (Bell Canada) and my service is stable since I have other services (VPN) using the same static IP range connected to a different firewall.

I will leave this connected over the weekend and check in and see how it is going. I suspect something was mis-configured but I'm a bit surprised that following the guides I did had me experience this.

I do think the OPNsense team needs to work on their community building more. Lack of support/interaction on many posts (not just mine) doesn't drive users to donate or buy official hardware.

[FW#1:]

Hello team,

I have two DEC2752 units I'm working on for our remote office. Both are configured as HA. For some reason, the primary/master unit is often showing my WAN Gateway down, when it is not.

I've got a /27 subnet with 3+ available public IPs that I am using. No IPv6.

Between my ISP router and the HA cluster, connectivity is to a basic HPE wan switch. I have my ISP connection on a VLAN and the OPNsense WAN connections are untagged/access to that VLAN.

I've reviewed my HA settings and everything looks fine but I'm not sure why this is occurring. I have enabled gateway monitoring (why would I want it off?) and only monitoring my ISP gateway.

Here is what I see in Reporting --> Health --> Quality = WAN

FW#1:







FW#2:









I have confirmed that my ethernet cabling is tested good and I don't have any funky configuration on the wan switch (it acts as a dumb switch to forward traffic, nothing special configured on it).

I'm really not sure where to go from here as I know my WAN connection is 100% fine since we have another connecting using an available IP from my /27 block, providing a critical service and it doesn't go down.


Thank you

["Always create Carp VIPs with the same subnet mask as it's parent interface. If the parent interface is /24, your Carp VIP should also be /24. Even though some sources claim that /32 will work, services like DHCP Failover will fail with peer holds all free leases."]

Hello everyone,

I was looking to get some clarification on OPNsense HA/CARP settings as I've got two units I'm building for a remote office.

Connectivity from this remote office will be via IPSEC VPN back to HQ. No local DHCP will be used. DHCP will be provided by my AD DHCP servers for the time being.

1) "Always create Carp VIPs with the same subnet mask as it's parent interface. If the parent interface is /24, your Carp VIP should also be /24. Even though some sources claim that /32 will work, services like DHCP Failover will fail with peer holds all free leases."
https://docs.opnsense.org/manual/how-tos/carp.html#adding-multiple-carp-ips

My Core VLAN is a /28 and I assign .1 and .2 to be my OPNsense devices. .3 will be my Core VLAN Virtual IP.

The rest of my VLANs at this location are /24. So with the above documentation stating to use the same CARP Subnet Mask as the parent interface (Core VLAN = /28), is this a real issue if I have CARP IPs set for the rest of my VLANs as /24?


2) "When designing a high-availability CARP setup, the underlying switch infrastructure plays a critical role in ensuring proper failover and performance. Both firewall nodes should ideally reside in the same Layer 2 broadcast domain and preferably within a unified switching fabric."

I just wanted to confirm this. My two OPNsense firewalls will reside on VLAN 1100 (CORE VLAN) while my switch IP is part of my VLAN 32 (MGMT VLAN). The Default Gateway for VLAN 32 is the CARP IP of VLAN 32 (10.103.32.3) and the ports from the switch to each OPNsense FW are tagged with the required VLANs.

This should be fine, just want to clarify this as my broadcast domain for VLAN 1100 (Core VLAN) will have both OPNsense firewalls part of it.



3) I'm not clear on the impact on both firewalls and connectivity if one VLAN/interface doesn't have a rule to allow CARP packets.

Commenting to see how this works out as I have nothing to help with.

[Insight]

Hello OPNsense team,

As an OPNsense user at home and slowly working to deploy it out for some of our smaller sites, a nice to have feature would be to have a dashboard widget, similar to the Firewall Live View but allow it to show representation of maybe 24 hours/x days.

I realize that maybe this data can be sent off to a data collector but it would be nice and convenient if there was something on the dashboard that showed not just live but statistics from a period of time.

While working on a new OPNsense deployment and testing/reviewing what is being blocked, its much easier to see this from the dashboard widget currently versus the actual live log. It just makes it difficult when the dashboard widget resets every time the page is reloaded.

I realize there is the Insight setting under Reporting but it would be nice to have this type of data on the dashboard for quick overview.

Is there a reason that the team hasn't deployed this? I bet there are higher priority items but I would like to see what others think regarding this.

Edit: I realize I can go into Firewall --> Log Files --> Overview but this doesn't show time periods (1h, 2h, 12h, 24h, etc...).

It would be nice to have this data easily visible from the dashboard. My old simple Cisco ASA5505 had this ability and that thing has 128mb of memory!

Edit 2: If the dashboard Firewall widget would stop refreshing every time the page refreshed, that would be nice. What is annoying about this widget is that the colors of the firewall rules changes each time the page is refreshed.

So if my eyes are quickly focusing on Red for Block - WAN to LAN Emerging Threats, when I refresh the page, it will change each time.

Its a nice widget but I personally think it needs to be reworked to provide better and consistent data.

Thanks

Add another user that is having this issue.

[100723] <Error> -- Just ran out of space in the queue. Please file a bug report on this   

[Edit:]

Hello everyone,

I'm working on building out the configuration for our new Deciso DEC2752 that will be going into a remote office.

Between HQ and remote office, I would really want to use DHCP relay inside the IPSec VPN tunnel so that the workstations and users at the remote office get IP addressing from our DHCP servers at HQ.

I came across past posts about some issues with this type of configuration but I'm not clear if it was a configuration issue or a problem within FreeBSD/OPNsense.

As I have the business license with our unit, I'm sure we will be using ISC DHCP.

Does anybody have any recent experience with the latest versions of OPNsense and DHCP relay inside IPSec VPN?


Thanks


Edit:
From reading this post: https://forum.opnsense.org/index.php?topic=39555.0

"We're moving to OpenBSD's dhcrelay (the development version migration is done) which can theoretically handle layer 2 and layer 3 relay, but there are no plans to start dashing out layer 3 relay support in OPNsense. We did always require a layer 2 device to relay from."

My remote site would be:

OPNsense FW --> Aruba L2 2530 Switch.

The aruba switch allows me to set ip helper-addresses on each vlan.

So if I set my DHCP server IP for each VLAN ip helper-address entry per vlan on the switch, does this sound like it would work? This way I don't have to configure DHCP relay in OPNsense and use my L2 switch for that instead.

The remote site is about 2 hours away so I'm trying to understand how to make this work, test it and deploy it.

For anyone else looking for this uncheck the AUTOCOLLECT box and you will be able to set gateway and dns.

Thanks for posting this!

Hey intelliIT,

Did you get anywhere with this? I'm in a similar situation and came across your thread just now.

Now, nda1 (the new drive) may have been formatted and used in another random PC so I'm unsure if its an issue with formatting?

Can you try wiping the new drive first (sure you do it on the right device! Backup is always a good idea, of course):

As mentioned in OPNsense Forum: Formatting whole disk before installation, try destroy the disk and delete the first 1M of the drive:

Code Select Expand

gpart destroy -F nda1
dd if=/dev/zero of=/dev/nda1 bs=1M count=1

Thank you,

That doesn't seem to work.

Code Select Expand

root@OPNsense:~ # gpart destroy -F nda1
gpart: arg0 'nda1': Invalid argument


I did a bit more searching around and I came across a similar issue on a different system-forum:
https://www.truenas.com/community/threads/create-new-pool-ends-with-error-command-gpart-create-s-gpt-dev-ada0-returned-non-zero-exit-status-1.77775/

I did set

Code Select Expand

sysctl kern.geom.debugflags=16

and then I ran

Code Select Expand

gpart create -s gpt /dev/nda1
After that, I then was able to proceed with the instructions for adding the second NVME disk to Opnsense.

Now it looks like everything is complete.

When I go into zpool status, I can see both nda0p4 and nda1p4 listed.

Code Select Expand

root@OPNsense:~ # zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: resilvered 2.33G in 00:00:04 with 0 errors on Sat Mar 22 10:59:11 2025
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            nda0p4  ONLINE       0     0     0
            nda1p4  ONLINE       0     0     0

errors: No known data errors


Thank you,

Hello everyone,

I have the CWWK N100 and I'm looking to add another NVME drive into it just for redundancy as I have the spare drives.

I have been looking at these instructions (https://forum.opnsense.org/index.php?topic=32650.0) which seem simple enough but I'm getting hung up on the steps to copy the partition table.

Code Select Expand

"gpart backup ada0 | gpart restore -F ada1"
When I run geom list disk, I see both of my drives showing:

Code Select Expand

root@OPNsense:~ # geom disk list
Geom name: nda0
Providers:
1. Name: nda0
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Mode: r3w3e6
   descr: KINGSTON SNV2S500G
   lunid: 00000000000000000026b76865cfcd85
   ident: 50026B76865CFCD8
   rotationrate: 0
   fwsectors: 0
   fwheads: 0

Geom name: nda1
Providers:
1. Name: nda1
   Mediasize: 500107862016 (466G)
   Sectorsize: 512
   Mode: r1w1e1
   descr: KINGSTON SNV2S500G
   lunid: 00000000000000000026b7686c0a0ee5
   ident: 50026B7686C0A0EE
   rotationrate: 0
   fwsectors: 0
   fwheads: 0

Code Select Expand

root@OPNsense:~ # gpart show
=>       40  976773088  nda0  GPT  (466G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528   16777216     3  freebsd-swap  (8.0G)
   17311744  959461376     4  freebsd-zfs  (458G)
  976773120          8        - free -  (4.0K)
My opnsense is installed on nda0 but when I run the command to start copying the partition table, I get an error stating:

Code Select Expand

root@OPNsense:~ # gpart backup nda0 | gpart restore -F nda1
gpart: geom 'nda1': Operation not permitted
Now, nda1 (the new drive) may have been formatted and used in another random PC so I'm unsure if its an issue with formatting?

I don't think that it is related to formatting but I'm not entirely sure.

Has anybody else ran into this issue?