Messages

[top -aSH]

I've been able to get into SSH on the secondary unit and running top -aSH shows very high CPU usage on [idle{idle: cpu0}] to [idle{idle: cpu3}]

It will fluctuate between 60-90% on idle CPU.

Code Select Expand

339 threads:   6 running, 308 sleeping, 25 waiting
CPU:  1.0% user,  0.0% nice,  2.6% system,  0.5% interrupt, 95.9% idle
Mem: 5466M Active, 4188K Inact, 1599M Laundry, 753M Wired, 2056K Buf, 45M Free
ARC: 248M Total, 191M MFU, 41M MRU, 5741K Anon, 1550K Header, 9298K Other
     199M Compressed, 350M Uncompressed, 1.76:1 Ratio
Swap: 8418M Total, 5057M Used, 3361M Free, 60% Inuse, 3160K In, 16M Out

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B    64K CPU0     0 212:59  98.46% [idle{idle: cpu0}]
   11 root        187 ki31     0B    64K RUN      1 213:02  97.30% [idle{idle: cpu1}]
   11 root        187 ki31     0B    64K RUN      2 213:18  97.11% [idle{idle: cpu2}]
   11 root        187 ki31     0B    64K CPU3     3 213:02  96.48% [idle{idle: cpu3}]
    2 root        -60    -     0B    64K WAIT     2   1:59   2.19% [clock{clock (0)}]
   16 root        -16    -     0B    16K psleep   3   0:53   2.08% [vmdaemon]
    9 root        -16    -     0B    48K swbufa   0   1:11   1.93% [pagedaemon{laundry: dom0}]
26862 root         20    0    15M  2748K CPU2     2   0:00   0.66% top -aSH
    9 root        -16    -     0B    48K CPU1     1   3:31   0.53% [pagedaemon{dom0}]
   12 root        -64    -     0B   336K WAIT     1   0:10   0.40% [intr{irq61: nvme0:io1}]
64127 root         24    0   984M   249M swread   3   0:17   0.33% /usr/local/bin/php-cgi
65172 root         20    0   746M    81M swread   0   0:14   0.28% /usr/local/bin/php-cgi
95245 root         20    0   538M   115M swread   1   0:04   0.27% /usr/local/bin/php-cgi
69176 root         21    0   634M   168M swread   3   0:19   0.27% /usr/local/bin/php-cgi
   12 root        -64    -     0B   336K WAIT     3   0:10   0.25% [intr{irq63: nvme0:io3}]



Here is what I had when I ran top -o size

Code Select Expand

root@FW02:~ # top -o size
last pid:  9419;  load averages:  4.04,  4.38,  4.72                                                                                                                                        up 0+04:14:12  12:56:00
76 processes:  1 running, 71 sleeping, 1 zombie, 3 waiting
CPU: 38.8% user,  0.0% nice,  6.3% system,  0.7% interrupt, 54.1% idle
Mem: 4471M Active, 1060K Inact, 2614M Laundry, 740M Wired, 2056K Buf, 40M Free
ARC: 248M Total, 193M MFU, 44M MRU, 45K Anon, 1576K Header, 9401K Other
     205M Compressed, 364M Uncompressed, 1.78:1 Ratio
Swap: 8418M Total, 8407M Used, 11M Free, 99% Inuse, 15M In, 49M Out

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
64127 root          1  20    0   874M    67M swread   2   0:34   0.14% php-cgi
90300 root          1  20    0   824M   110M pfault   2   0:05   0.22% php
84811 root          1  20    0   824M    81M pfault   1   0:05   0.08% php
 9889 root          1  20    0   824M    55M pfault   2   0:07   0.24% php
 8548 root          1  20    0   824M    49M swread   3   0:07   0.27% php
10724 root          1  20    0   824M    25M swread   3   0:06   0.21% php
67711 root          1  20    0   824M    13M swread   0   0:07   0.40% php
17207 root          1  20    0   746M    76M select   0   0:34   0.00% php-cgi
65172 root          1  20    0   746M    69M swread   2   0:24   0.14% php-cgi
70166 root          1  23    0   746M  4096B lockf    1   0:33   0.00% <php-cgi>
70079 root          1  24    0   746M  4096B lockf    1   0:31   0.00% <php-cgi>
  395 root         19  21    0   741M  4096B WAIT     2   0:51   0.00% <python3.11>
54338 root          1  20    0   726M   110M pfault   0   0:05   0.22% php
95464 root          1  20    0   726M    75M swread   0   0:04   0.14% php
 6824 root          1  20    0   726M    64M swread   1   0:05   0.14% php
49842 root          1  20    0   726M    46M swread   0   0:04   0.19% php
67322 root          1  28    0   635M   296M lockf    0   0:37   2.03% php-cgi
68272 root          1  28    0   634M   236M lockf    1   0:43   0.00% php-cgi
69176 root          1  21    0   634M   219M select   3   0:33   0.00% php-cgi
68381 root          1  20    0   634M    84M select   3   0:31   0.01% php-cgi
64196 root          1  20    0   634M   129M pfault   2   0:27   0.21% php-cgi
66839 root          1  20    0   634M    32M select   1   0:34   0.00% php-cgi
18065 root          1  20    0   538M   380M select   1   0:34   0.00% php-cgi
69630 root          1  20    0   538M  5988K select   3   0:29   0.00% php-cgi
68121 root          1  21    0   538M  4096B lockf    3   0:29   0.00% <php-cgi>
59949 root          1  20    0   538M   356M select   3   0:25   0.00% php-cgi
 2236 root          1  20    0   538M   317M select   3   0:23   0.00% php-cgi
 8262 root          1  20    0   538M   198M select   3   0:18   0.00% php-cgi
71237 root          1  20    0   538M   169M select   1   0:27   0.00% php-cgi
71329 root          1  20    0   538M    26M select   1   0:26   0.00% php-cgi
95245 root          1  24    0   538M  4096B WAIT     1   0:19   0.00% <php-cgi>
 4833 root          1  36    0   510M   407M pfault   3   0:02  16.10% php
98709 root          1  34    0   490M   396M pfault   0   0:02  17.35% php
 4317 root          1  36    0   490M   395M pfault   1   0:02  18.56% php
 7701 root          1  40    0   450M   369M pfault   3   0:02  33.57% php
 8680 root          1  34    0   446M   356M pfault   0   0:01  12.93% php
 8837 root          1  36    0   440M   359M pfault   3   0:01  16.98% php
 7792 root          1  36    0   438M   357M pfault   1   0:01  19.34% php
 7077 root          1  36    0   438M   356M pfault   1   0:01  18.72% php
52571 root         17  68    0    97M  3236K sigwai   3   0:00   0.00% charon
 3712 root          1  29    0    70M    39M pfault   2   0:01   4.29% python3.11
16540 root          1  20    0    53M  4096B wait     2   0:00   0.00% <php-cgi>
62847 root          1  20    0    53M  4096B wait     0   0:00   0.00% <php-cgi>
62672 root          1  68    0    53M  4096B wait     2   0:00   0.00% <php-cgi>
63410 root          1  68    0    53M  4096B wait     1   0:00   0.00% <php-cgi>
31731 root          3  20    0    49M  2564K kqread   2   0:05   0.03% syslog-ng
17415 root          1  20    0    41M  3612K nanslp   3   3:28   0.01% python3.11
  393 root          1  68    0    35M  4096B wait     2   0:00   0.00% <python3.11>
53436 root          1  20    0    28M  4408K select   1   0:01   0.01% python3.11
53217 root          1  20    0    27M  3788K select   3   0:00   0.02% python3.11
20171 root          4  68    0    26M  1360K uwait    2   0:02   0.02% dpinger
31675 root          1  68    0    24M  4096B wait     3   0:00   0.00% <syslog-ng>
57565 root          2  20    0    24M  3144K select   1   0:02   0.01% ntpd
61730 root          1  20    0    23M  3228K kqread   0   0:02   0.00% lighttpd
35568 root          1  20    0    20M  2064K select   3   0:00   0.02% sshd-session
  451 root          1  20    0    20M  1072K select   3   0:00   0.00% sshd-session
86829 root          1  20    0    20M  2536K select   0   0:00   0.00% sshd
88249 root          1  20    0    17M  4096B pause    3   0:00   0.00% <csh>
56438 root          1  20    0    15M  2040K CPU3     3   0:00   0.10% top
  757 root          1  20    0    15M   408K select   1   0:00   0.00% devd
65025 root          1  68    0    14M  1000K ttyin    2   0:00   0.00% sh
36343 root          1  20    0    14M  4096B wait     2   0:00   0.00% <sh>
64828 root          1  56    0    14M  4096B wait     0   0:00   0.00% <login>
  197 root          1  20    0    14M  1416K piperd   3   0:00   0.11% cron
16433 root          1  20    0    14M  1300K bpf      3   0:00   0.03% filterlog
 2237 root          1  20    0    14M  4096B wait     2   0:00   0.00% <flock>
72462 root          1  20    0    14M  1304K kqread   2   0:00   0.00% tail
74613 root          1  20    0    14M  1296K select   0   0:00   0.00% tail
15949 root          1  20    0    14M  4096B WAIT     3   0:00   0.00% <cron>
52141 root          1  68    0    13M  4096B kqread   0   0:00   0.00% <daemon>
88865 root          1  68    0    13M  4096B kqread   2   0:00   0.00% <daemon>
61055 root          1  20    0    13M  1072K select   3   0:01   0.01% powerd
40897 _flowd        1  20    0    13M   980K select   2   0:00   0.00% flowd
40882 root          1  68    0    13M  4096B sbwait   1   0:00   0.00% <flowd>


Hello OPNsense,

I have two DEC2752 units configured in HA that are being used for a new remote office network build. These two units were purchased less than 1-year ago and the configuration on them is quite basic. No IPv6. No unbound/dnsmasq configuration. I've got an IPSEC vpn connection from the VIP addresses to my primary site. DHCP/DNS (at this time) is handled at my primary DC.

So the OPNsense firewall cluster is just acting as a secure gateway with a site to site tunnel. Nothing fancy.

Last week, I tried to connect to both units. Master (10.103.0.1) responds fine. Secondary (10.103.0.2) is sluggish. The web gui fails to load properly.

The IPSec VPN tunnel is functional.

When I started to investigate what is going on with the secondary unit, I see a ton of errors via CLI:

Code Select Expand

root@FW02:~ # swap_pager: out of swap space
swp_pager_getswapspace(10): failed
swp_pager_getswapspace(3): failed
swap_pager: out of swap space
swp_pager_getswapspace(4): failed
swap_pager: out of swap space
swp_pager_getswapspace(1): failed
swp_pager_getswapspace(4): failed
swp_pager_getswapspace(6): failed
swap_pager: out of swap space
swp_pager_getswapspace(1): failed
swap_pager: out of swap space
swp_pager_getswapspace(2): failed
swap_pager: out of swap space
swp_pager_getswapspace(22): failed
swp_pager_getswapspace(20): failed

If I reboot the secondary firewall, the OS loading process seems slow. Once I get in and press 8 for CLI, I don't have much time before it starts to bog down.

running df -h I get:

Code Select Expand

root@FW02:~ # df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default    222G     10G    212G     5%    /
devfs                 1.0K      0B    1.0K     0%    /dev
/dev/gpt/efifs        256M    645K    255M     0%    /boot/efi
zroot/tmp             212G    224K    212G     0%    /tmp
zroot                 212G     96K    212G     0%    /zroot
zroot/var/log         212G    164M    212G     0%    /var/log
zroot/var/audit       212G     96K    212G     0%    /var/audit
zroot/usr/home        212G     96K    212G     0%    /usr/home
zroot/usr/ports       212G     96K    212G     0%    /usr/ports
zroot/usr/src         212G     96K    212G     0%    /usr/src
zroot/var/crash       212G     96K    212G     0%    /var/crash
zroot/var/mail        212G    144K    212G     0%    /var/mail
zroot/var/tmp         212G     96K    212G     0%    /var/tmp
devfs                 1.0K      0B    1.0K     0%    /var/dhcpd/dev


When I take a look at top -o res, I see high swap:

Code Select Expand

root@FW02:~ # top -o res
last pid: 13953;  load averages:  4.39,  2.72,  1.50    up 0+00:10:36  15:54:20
63 processes:  25 running, 38 sleeping
CPU: 53.4% user,  0.0% nice, 44.4% system,  2.2% interrupt,  0.0% idle
Mem: 5721M Active, 702M Inact, 195M Laundry, 754M Wired, 2056K Buf, 493M Free
ARC: 257M Total, 184M MFU, 66M MRU, 610K Anon, 1329K Header, 5222K Other
     224M Compressed, 324M Uncompressed, 1.44:1 Ratio
Swap: 8418M Total, 6101M Used, 2317M Free, 72% Inuse, 314M In
swap_pager: out of swap spaceiled
swp_pager_getswapspace(10): failedIZE    RES STATE    C   TIME    WCPU COMMAND
85082ager_getswapspace(48: faile 824M   555M RUN      2   0:04  32.18% php
 7616pager: out of swap42paceile1069M   490M RUN      2   0:14  30.59% php-cgi
 5271ager_getswapspace(48): faile488M   393M CPU0     0   0:02  24.03% php-cgi
28393 root          1  21    0   548M   392M select   2   0:15   0.01% php-cgi
 5177 root          1  48    0   494M   391M RUN      1   0:02  31.91% php
 9260 root          1  24    0   584M   386M select   2   0:06   0.00% php-cgi
91350 root          1  50    0   516M   368M RUN      0   0:03   6.38% php
 2079 root          1  42    0   440M   338M RUN      2   0:03  43.99% php-cgi
 6068 root          1  24    0   538M   291M RUN      3   0:02   9.70% php-cgi
 2260 root          1  44    0   751M   270M RUN      3   0:09   9.23% php-cgi
63351 root          1  24    0   726M   266M RUN      3   0:03   7.69% php
28926 root          1  20    0   634M   236M select   3   0:16   0.00% php-cgi
 8504 root          1  20    0   584M   236M select   2   0:06   0.00% php-cgi
 1583 root          1  24    0   792M   224M CPU1     1   0:07  23.80% php-cgi


I have tried to clean up some logs that I had in /var/log and reboot but that didn't help.

These are the only packages I have installed:

Code Select Expand

root@FW02:~ # pkg info | grep os-
os-OPNBEcore-1.7_3             OPNsense Business Edition add-ons
os-OPNcentral-1.12_2           OPNsense central management
os-dmidecode-1.2               Display hardware information on the dashboard
os-etpro-telemetry-1.8         ET Pro Telemetry Edition



What I'm struggling to understand is why would my primary unit be working just fine and my secondary having this issue. I have been evaluating OPNsense as a use case for our remote site(s) but my configurations seem a bit light.

I do enable logging on my firewall rules but I don't have many rules at all.
I have a total of 11 VHIDs and on my primary unit at this time, my swap is 0.0%, memory used is 1047mb/arc 1103mb and my disk utilization is 1%.

When checking my snapshot, I see that bectl list shows default as 10.4G.

Code Select Expand

root@FW02:~ # bectl list
BE      Active Mountpoint Space Created
default NR     /          10.4G 2025-04-17 09:23

When I compare that to my primary/active unit, it shows 1.29G

My concern here is that this is some kind of hardware failure but I'm not sure how to confirm that or check.


The web interface is unresponsive that I can't even go in and create a recent backup. The page for backups won't load. I should have a latest backup but I'm just pointing out as to how locked up the interface is.

I can't recall what was done in the past 1-3 weeks but it wouldn't be much. These firewalls are waiting for me to rebuild a new IPSEC vpn connection from my primary location so I haven't performed any recent configuration changes to them to my knowledge.

I've captured screenshots and I can get further logs from the POST sequence and OS bootup if it helps.


Thank you,

You configure a CARP address on the Internet facing (WAN) interface and use that as the endpoint for your IPsec tunnel(s). Connectivity will move with the CARP address in case the primary node fails.

Did you setup your HA cluster following the documentation? So you have a HA/CARP address on all interfaces?

Yes I've got my HA cluster configured as per the documentation. My concern is not the OPNsense node failing but the other end. Be it a hardware failure or ISP being down, I am trying to get my OPNsense cluster to have a secondary IPSEC connection going to the opposite site, to their secondary connection.

Hello everyone,

I have two Deciso DEC2752 units in a HA configuration that I am soon about to deploy.

At this moment I am nearly ready except I need to figure out how to configure my OPNsense deployment so that if my primary IPSEC VPN connection goes down, the secondary IPSEC VPN connection will establish.

The remote end are two Versa-SDWAN appliances. Versa #1 has one ISP connection and Versa #2 has the other IPS connection. Both ISP connections are for separate ISPs for redundancy.

Right now, my OPNsense cluster is configured for IPSEC VPN to Versa #1 Public IP. I can power off one of my OPNsense units and the other kicks in as expected.

But for whatever reason I cannot seem to figure out how to apply some kind of metric/weight to keep the primary IPSEC tunnel active and failover to the other IPSEC tunnel if my primary versa is down.

Would anybody be able to point me into the direction on what to read or how to accomplish this?


Thank you!

Hello everyone,

Our organization uses Cisco Umbrella for web filtering. Our our primary site (Home Office) I have two Cisco Umbrella Virtual Forwarders that are used for DNS resolution.

I am working on configuring and testing two DEC2752 units in a HA configuration for a remote office. The remote office will connect to Home Office via IPSEC site to site VPN connection.

This remote office is small enough that there is not and won't be any server onsite. Due to this, I want our web traffic from the remote site to traverse the VPN tunnel back to the home office.

Now, in the event that the VPN tunnel is down, I want to use Cisco Umberella public DNS IPs.

The remote office staff get their IP addressing/DNS information VIA AD/DHCP. This of course won't work when the tunnel is down.

I was thinking that I might be able to configure the public DNS IP addresses in the OPNsense System/General settings but I am not sure if that would help.

Within OPNsense, I have not configured Unbound/DNSMasq.

Any suggestions with my current configuration on what I can do to keep web traffic flowing if IPSEC is down?


Thank you,

Well I guess I'll document this myself.

Frusterated by this, I factory reset both devices, logged in, deleted the LAN interface and setup my core VLAN. Setup my public IP addresses for each device, the gateways and left them connected to the internet.

Not configured in HA, both devices show regular activity for their WAN gateways. Again, not going to disable gateway monitoring as I want to see the status/functionality of my gateway.

FW #1:


FW #2:



So this looks normal to me. I know my ISP (Bell Canada) and my service is stable since I have other services (VPN) using the same static IP range connected to a different firewall.

I will leave this connected over the weekend and check in and see how it is going. I suspect something was mis-configured but I'm a bit surprised that following the guides I did had me experience this.

I do think the OPNsense team needs to work on their community building more. Lack of support/interaction on many posts (not just mine) doesn't drive users to donate or buy official hardware.

[FW#1:]

Hello team,

I have two DEC2752 units I'm working on for our remote office. Both are configured as HA. For some reason, the primary/master unit is often showing my WAN Gateway down, when it is not.

I've got a /27 subnet with 3+ available public IPs that I am using. No IPv6.

Between my ISP router and the HA cluster, connectivity is to a basic HPE wan switch. I have my ISP connection on a VLAN and the OPNsense WAN connections are untagged/access to that VLAN.

I've reviewed my HA settings and everything looks fine but I'm not sure why this is occurring. I have enabled gateway monitoring (why would I want it off?) and only monitoring my ISP gateway.

Here is what I see in Reporting --> Health --> Quality = WAN

FW#1:







FW#2:









I have confirmed that my ethernet cabling is tested good and I don't have any funky configuration on the wan switch (it acts as a dumb switch to forward traffic, nothing special configured on it).

I'm really not sure where to go from here as I know my WAN connection is 100% fine since we have another connecting using an available IP from my /27 block, providing a critical service and it doesn't go down.


Thank you

["Always create Carp VIPs with the same subnet mask as it's parent interface. If the parent interface is /24, your Carp VIP should also be /24. Even though some sources claim that /32 will work, services like DHCP Failover will fail with peer holds all free leases."]

Hello everyone,

I was looking to get some clarification on OPNsense HA/CARP settings as I've got two units I'm building for a remote office.

Connectivity from this remote office will be via IPSEC VPN back to HQ. No local DHCP will be used. DHCP will be provided by my AD DHCP servers for the time being.

1) "Always create Carp VIPs with the same subnet mask as it's parent interface. If the parent interface is /24, your Carp VIP should also be /24. Even though some sources claim that /32 will work, services like DHCP Failover will fail with peer holds all free leases."
https://docs.opnsense.org/manual/how-tos/carp.html#adding-multiple-carp-ips

My Core VLAN is a /28 and I assign .1 and .2 to be my OPNsense devices. .3 will be my Core VLAN Virtual IP.

The rest of my VLANs at this location are /24. So with the above documentation stating to use the same CARP Subnet Mask as the parent interface (Core VLAN = /28), is this a real issue if I have CARP IPs set for the rest of my VLANs as /24?


2) "When designing a high-availability CARP setup, the underlying switch infrastructure plays a critical role in ensuring proper failover and performance. Both firewall nodes should ideally reside in the same Layer 2 broadcast domain and preferably within a unified switching fabric."

I just wanted to confirm this. My two OPNsense firewalls will reside on VLAN 1100 (CORE VLAN) while my switch IP is part of my VLAN 32 (MGMT VLAN). The Default Gateway for VLAN 32 is the CARP IP of VLAN 32 (10.103.32.3) and the ports from the switch to each OPNsense FW are tagged with the required VLANs.

This should be fine, just want to clarify this as my broadcast domain for VLAN 1100 (Core VLAN) will have both OPNsense firewalls part of it.



3) I'm not clear on the impact on both firewalls and connectivity if one VLAN/interface doesn't have a rule to allow CARP packets.

Commenting to see how this works out as I have nothing to help with.

[Insight]

Hello OPNsense team,

As an OPNsense user at home and slowly working to deploy it out for some of our smaller sites, a nice to have feature would be to have a dashboard widget, similar to the Firewall Live View but allow it to show representation of maybe 24 hours/x days.

I realize that maybe this data can be sent off to a data collector but it would be nice and convenient if there was something on the dashboard that showed not just live but statistics from a period of time.

While working on a new OPNsense deployment and testing/reviewing what is being blocked, its much easier to see this from the dashboard widget currently versus the actual live log. It just makes it difficult when the dashboard widget resets every time the page is reloaded.

I realize there is the Insight setting under Reporting but it would be nice to have this type of data on the dashboard for quick overview.

Is there a reason that the team hasn't deployed this? I bet there are higher priority items but I would like to see what others think regarding this.

Edit: I realize I can go into Firewall --> Log Files --> Overview but this doesn't show time periods (1h, 2h, 12h, 24h, etc...).

It would be nice to have this data easily visible from the dashboard. My old simple Cisco ASA5505 had this ability and that thing has 128mb of memory!

Edit 2: If the dashboard Firewall widget would stop refreshing every time the page refreshed, that would be nice. What is annoying about this widget is that the colors of the firewall rules changes each time the page is refreshed.

So if my eyes are quickly focusing on Red for Block - WAN to LAN Emerging Threats, when I refresh the page, it will change each time.

Its a nice widget but I personally think it needs to be reworked to provide better and consistent data.

Thanks

Add another user that is having this issue.

[100723] <Error> -- Just ran out of space in the queue. Please file a bug report on this   

[Edit:]

Hello everyone,

I'm working on building out the configuration for our new Deciso DEC2752 that will be going into a remote office.

Between HQ and remote office, I would really want to use DHCP relay inside the IPSec VPN tunnel so that the workstations and users at the remote office get IP addressing from our DHCP servers at HQ.

I came across past posts about some issues with this type of configuration but I'm not clear if it was a configuration issue or a problem within FreeBSD/OPNsense.

As I have the business license with our unit, I'm sure we will be using ISC DHCP.

Does anybody have any recent experience with the latest versions of OPNsense and DHCP relay inside IPSec VPN?


Thanks


Edit:
From reading this post: https://forum.opnsense.org/index.php?topic=39555.0

"We're moving to OpenBSD's dhcrelay (the development version migration is done) which can theoretically handle layer 2 and layer 3 relay, but there are no plans to start dashing out layer 3 relay support in OPNsense. We did always require a layer 2 device to relay from."

My remote site would be:

OPNsense FW --> Aruba L2 2530 Switch.

The aruba switch allows me to set ip helper-addresses on each vlan.

So if I set my DHCP server IP for each VLAN ip helper-address entry per vlan on the switch, does this sound like it would work? This way I don't have to configure DHCP relay in OPNsense and use my L2 switch for that instead.

The remote site is about 2 hours away so I'm trying to understand how to make this work, test it and deploy it.

For anyone else looking for this uncheck the AUTOCOLLECT box and you will be able to set gateway and dns.

Thanks for posting this!

Hey intelliIT,

Did you get anywhere with this? I'm in a similar situation and came across your thread just now.

Now, nda1 (the new drive) may have been formatted and used in another random PC so I'm unsure if its an issue with formatting?

Can you try wiping the new drive first (sure you do it on the right device! Backup is always a good idea, of course):

As mentioned in OPNsense Forum: Formatting whole disk before installation, try destroy the disk and delete the first 1M of the drive:

Code Select Expand

gpart destroy -F nda1
dd if=/dev/zero of=/dev/nda1 bs=1M count=1

Thank you,

That doesn't seem to work.

Code Select Expand

root@OPNsense:~ # gpart destroy -F nda1
gpart: arg0 'nda1': Invalid argument


I did a bit more searching around and I came across a similar issue on a different system-forum:
https://www.truenas.com/community/threads/create-new-pool-ends-with-error-command-gpart-create-s-gpt-dev-ada0-returned-non-zero-exit-status-1.77775/

I did set

Code Select Expand

sysctl kern.geom.debugflags=16

and then I ran

Code Select Expand

gpart create -s gpt /dev/nda1
After that, I then was able to proceed with the instructions for adding the second NVME disk to Opnsense.

Now it looks like everything is complete.

When I go into zpool status, I can see both nda0p4 and nda1p4 listed.

Code Select Expand

root@OPNsense:~ # zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: resilvered 2.33G in 00:00:04 with 0 errors on Sat Mar 22 10:59:11 2025
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            nda0p4  ONLINE       0     0     0
            nda1p4  ONLINE       0     0     0

errors: No known data errors


Thank you,