Topics

Last week I setup a new ESXi VM template to move from UFS to ZFS and upgrade to 26.1.
I ran several ZFS unmap tests inflating thin VMDK with large ISO and deleting.

zpool set autotrim=on zroot

System didn't appear to auto trim / unmap within ~30 min.
But running - zpool trim zroot - manual trim worked.  VMDK shrunk to very close to exact used space  So was happy and proceeded to swap over to new ZFS VM template.

Only been weekend since deployed and now reviewing today VMDK is 47GB yet Opnsense reports only 5GB used?
I've since ran manual trim again but it only shrunk VMDK ~1GB.  There is no way this much data has ever been written other than some internal ZFS function.

Is ZFS scrub or compression screwing with thin provisioning unmap / zero space?
I'm at a loss what FreeBSD + ZFS + VMFS thin disk is doing - Any suggestions appreciated.

Thin disk is 80GB

# df -h
             Filesystem                   Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default            69G    5.0G     64G     7%    /
devfs                        1.0K      0B    1.0K     0%    /dev
/dev/gpt/efiboot0            260M    1.3M    259M     1%    /boot/efi
zroot/var/mail                64G    160K     64G     0%    /var/mail
zroot/var/log                 64G     31M     64G     0%    /var/log
zroot/usr/src                 64G     96K     64G     0%    /usr/src
zroot/tmp                     64G    206M     64G     0%    /tmp
zroot                         64G     96K     64G     0%    /zroot
zroot/usr/ports               64G     96K     64G     0%    /usr/ports
zroot/var/audit               64G     96K     64G     0%    /var/audit
zroot/home                    64G     96K     64G     0%    /home
zroot/var/crash               64G     96K     64G     0%    /var/crash
zroot/var/tmp                 64G    388K     64G     0%    /var/tmp
devfs                        1.0K      0B    1.0K     0%    /var/unbound/dev
/usr/local/lib/python3.13     69G    5.0G     64G     7%    /var/unbound/usr/local/lib/python3.13
/lib                          69G    5.0G     64G     7%    /var/unbound/lib
/dev/md43                    484M     48K    445M     0%    /usr/local/zenarmor/output/active/temp
fdescfs                      1.0K      0B    1.0K     0%    /dev/fd
procfs                       8.0K      0B    8.0K     0%    /proc
tmpfs                        100M     24K    100M     0%    /usr/local/zenarmor/run/tracefs

# zpool status
  pool: zroot
 state: ONLINE
config:

   NAME        STATE     READ WRITE CKSUM
   zroot       ONLINE       0     0     0
     da0p4     ONLINE       0     0     0

errors: No known data errors

Edit:  Reviewing backup logs the fresh VM template VMDK was 14GB prior to few GB of Zenarmor / NTOPNG data accumulated over weekend.
VMDK has grown around ~29GB beyond what it should be in around 3 days.

Attempting to clean up (delete) old Sub CA cert and it's issued certs.
The cert won't delete because its in use by legacy OpenVPN which has since been removed from WebUI.
So apparently the legacy OpenVPN configuration has remained.  When I switched long time ago I believe I had just disabled the legacy OpenVPN servers.

Does this now require some manual edits of XML config and import?
What's simplest way to purge legacy OpenVPN config to free up certs for deletion?

Hi.

Can anyone shed some light to help me better understand the Default deny / State violations rule?  The causes and fix?
In the past when traffic isn't flowing that should be and is logging blocked by state violation I just reboot.
But I'm curious about resolving without reboot. 

For example
Yesterday my Spectrum home connection did a IP change on me for first time in 5+ years.
I re-configured a IPSEC VPN on both sides for new IP but traffic wasn't flowing despite phase 2s online.
Then I saw all VPN tunnel traffic being denied by state violation rule.
Rebooted the remote Opnsense and VPN began flowing again.

Would have reset states table under Firewall --> Diag --> States --> Actions fix issue?

Needing to adjust some NGINX ACLs today and ACL issue.

Upon editing ACL to add new IP I am getting "Unexpected error, check log for details"
Then tried creating new ACL and it worked.  Then tried editing same ACL.  Same error.

So then created brand new ACL with all needed IPs and saved ok.
Then went to HTTP server and attempted to change ACL to new ACL.  Same error. 
Cannot change HTTP server ACL.  :-/

Also tried all same above with NGINX service stopped.  Same error.

Reviewed NGINX log, Opnsense general + backend logs. Don't see anything mentioned about NGINX config / ACL.

Anyone aware of NGINX ACL issues or suggestions?

OPNsense 24.7.8
os-nginx 1.34_2

Setup HA / CARP today on OPNsense 24.1.10_8.
LAN CARP VIP only and using single this DHCP WAN script to toggle master WAN INT
https://gist.github.com/spali/2da4f23e488219504b2ada12ac59a7dc
Outgoing NAT - LAN , Lan adresss --> LAN CARP VIP  (Source firewall traffic to use CARP LAN VIP downstream)
After a bit of struggle failover and sync is working.  (Make those OPT# match + administration - listen interfaces - All Recommended!)  IPSEC states don't though with this WAN toggle :(

Anyway - now noticing redis won't start.  Exact same error issue as discussed here
https://forum.opnsense.org/index.php?topic=38845.15

WARNING: The TCP backlog setting of 511 cannot be enforced because kern.ipc.somaxconn is set to the lower value of 128.

Warning: Could not create server TCP listening socket Real_Int_IP:6379: bind: Address already in use

Failed listening on port 6379 (tcp), aborting.

Just like other post.  I can CLI

redis-server --port 6379 --daemonize yes

It runs on same WebUI default port 6379 just fine.  Doesn't show running via WebUI.
And then can run Ntopng Enterprise via WebUI.

Other post was left without known cause...
Any ideas?  New HA setup shouldn't cause this right?

Edit:
Starting redis this way it is a fresh DB.
I haven't reset it via webui and HA sync shouldn't of reset it.  Only one way master to backup sync.  Hmm.

On 24.1.6 attempting upgrade to 24.1.8

After upgrade Unbound won't start.

2024-05-31T11:27:54-07:00   Critical   unbound   [8463:0] fatal error: failed to setup modules   
2024-05-31T11:27:54-07:00   Error   unbound   [8463:0] error: module init for module python failed   
2024-05-31T11:27:54-07:00   Error   unbound   [8463:0] error: python exception in Py_InitializeFromConfig: init_fs_encoding: failed to get the Python codec of the filesystem encoding


Additionally getting NGINX errors on certificates and extremely delayed boot.  I believe its just because of Unbound issue and lack of DNS

failed to setup nginx
performing sanity check on nginx configuration
nginx ssl_stapling ignored, host not found in ocsp responder r3.o.lencr.org

The other odd behavior with 24.1.8 is the extracting hung on extracting net-snmp for over 10 minutes.  During which the webui was completely unresponsive on other tabs.
It finally did extract and finished update.

[142/161] Upgrading net-snmp from 5.9.4,1 to 5.9.4_2,1...
===> Creating groups
Creating group 'snmpd' with gid '344'
===> Creating users
Creating user 'snmpd' with uid '344'
[142/161] Extracting net-snmp-5.9.4_2,1: ........

---

Any suggestions what is going on with unbound?  I've attempted upgrade twice, same issue.
I've since restored backup of 24.1.6.
Thanks

Been learning IPSEC VTI route tunnels and Opnsense config.  Need to migrate from legacy.
Having been accustomed to policy VPNs I've always filtered the ipsec at source.  Specific tunnels / hosts terminated.  But seems the route vpn  standard is no filter and all 0.0.0.0/0 is processed.

So then all lans pass traffic for every ipsec# gateway route.

So effectively, the only way to control traffic is with IPSEC# INT outbound rules.  And subsequent inbound rules at destination firewall on global ipsec interface.

I've read on the praises of routed based VPNs as there is no filter rules required.  But my understanding is it is just trading places.  Rather than define them on ipsec filter rules, I'm applying same rules to IPSEC# INT outbound rules.  (I don't want everything allowed at source and only block at destination inbound)

So I explored the tunables
net.inet.ipsec.filtertunnel = 1
net.inet6.ipsec6.filtertunnel = 1
net.enc.in.ipsec_filter_mask = 0
net.enc.out.ipsec_filter_mask = 0
https://docs.opnsense.org/manual/vpnet.html#route-based-vti

The filtering appears to work but one cannot define multiple source / destination in same phase2.  Each needs its own phase2 / child.

The other benefit I see is if "install polices" accidentally gets enabled with 0.0.0.0/0 it will totally lock you out until console access + service strongswan stop.  I don't believe this can happen now if_ipsec(4) filtering enabled?  I want to test this more with filtering enabled as I find the ability to totally break firewall access with a simple "install policy" toggle to be a bit concerning.

So - are there any drawbacks to filtering VTIs (if_ipsec(4)?
(Other than that policy VPNs can no longer be used)

Thank you.

Edit
Actually now noticing different behavior with tunables if_ipsec(4) filtering enabled.  Traffic on destination firewall now appears to be flowing inbound to VTI IPSEC# interface rather than inbound via global IPSEC interface.  I'm guessing this by design?

I'm mistaken.  (if_ipsec(4) does not appear to be working.
If I disable a Phase 2 with defined local / remote subnets traffic still flows?
So my other phase 2 which also has local / remote subnets defined must not be actually using them and is still using 0.0.0.0/0
No idea WTF is happening here.  Other than policy vpn is simpler.

Updated to 23.7 today.  System resource usage out of control and crashing opnsense within 20 minutes.
VMware VM - 12 CPU core / 12GB RAM / 8GB SWAP.
Around 20 min of uptime RAM + SWAP 100% + CPU 100%. 
Been running this VM long time throughout many Opnsense versions without issue.  Normally very low CPU and ~60-70% RAM use with both zenarmor + surcata + few others.

Despite turning off Zenarmor from starting engine + elasticsearch at boot I still see Zenarmor processes in a "ps axmfv" using excessive CPU - So thinking it's zenarmor...

Anyone seen this? 
Going to give Zenarmor full uninstall + reinstall momentarily.  If that doesn't fix then Veeam VM restore back to 23.1

Does anyone know of a good GUI guide for the new 23.x IPSEC policy setup?  (swantcl.conf)

Following below example I cannot get a tunnel to come up.  Nor do I see anything under IPSEC log?  Does "new connections" even use GUI IPSEC log?  Or is that just for "legacy" IPSEC tunnels only?

I've setup a lab exactly like this example just with different subnets.
https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn.html

Been having issue with Nropng enterprise plugin for last week on multiple opnsense 22.7.10_2

I have the external repo pkg installed discussed here:
https://www.ntop.org/guides/ntopng/third_party_integrations/opnsense.html

pkg add https://packages.ntop.org/FreeBSD/FreeBSD:13:amd64/latest/ntop-1.0.pkg

Upon checking for updates the ntopng repo is checked but returns nothing to update.
I inquired to ntopng and they said repo is fine and just directed me to just run pkg install ntopng.
But this installs regular ntopng5.5 for freebsd.  Not the opnsense GUI version os-ntopng-enterprise -5.5

Any ideas why opnsense isn't working with ntopng repo?  Thanks.

I've been using NGINX to proxy http backend.  Today trying Opnsense Nginx TLS upstream to Nginx Proxy Manager.

Getting http error: 
1 upstream SSL certificate verify error: (18:self signed certificate) while SSL handshaking to upstream

At first I had two different lets encrypt wildcard certs.  1 on Opnsense + 1 on NPM proxy host.
Then tried a host cert on NPM to see if it was cause of validation failure.  Still same error. 

Connects if I disable Opnsense Nginx TLS verify.
And 2nd lets encrypt cert is valid on NPM host if accessed directly.

Opnsense has the Lets encrypt X1 + Intermediate in its trusted store.
Under Nginx upstream I have X1 CA set as TLS: Trusted Certificate

Ideas as to what I'm doing wrong with certificate chain / Nginx TLS verify?  Thanks.

Hi,

Been learning some Vaultwarden which lead me to also learn some fail2ban.
Wasted some time but eventually realized fail2ban wasn't going to work behind separate NGINX proxy regardless of forwarded header.  (nftables / iptables have correct ban IP yet packets inbound only from Opnsense / NGINX IP - thus no block)

So that lead me to find this Opnsense API solution to create ban Alias which is working well.
https://github.com/fail2ban/fail2ban/pull/2761/files

curl -k -s -XPOST -d '{"address":"<ip>"}' -H "Content-Type: application/json" -k -u "<oskey>":"<ossecret>" \
            https://<osaddress>/api/firewall/alias_util/add/<osalias>

However it won't block until you stop trying to brute force for a bit and let the state expire.

What I'm wondering if I can append another API Curl to this that will delete all active states for the same ban IP to force it to restart which would give desired result of immediate block.

Possible?  Any tips or API documentation pointers appreciated.

Edit:  I missed this before posting but still not sure exactly what I need being completely unfamiliar with API.  Perhaps I'll spin up my opnsense VM template and explore.
https://docs.opnsense.org/development/api/core/diagnostics.html#
POST diagnostics firewall delState  $stateid,$creatorid
POST diagnostics firewall queryStates

Having Opnsense crash / hang following two vmotions.

Have Opnsense on 2 host ESXi7 cluster.  Have a switch's vlans interconnecting Internet Modem / Opnsense ESXi hosts WAN links + Opnsense ESXi hosts LAN links.

I can vmotion a single time to other host.  And all is well.  However when I vmotion back to original host after it completed Opnsense will hang.  Internet drops.  Pings latency high + drops on LAN INT.  WebGUI stops responding and cannot login via console.  Just hangs after pass.

The moment I go to reboot it via vsphere web console it responds again.  So I did bit of troubleshooting.  Believe I've got in narrowed down to IPSEC strongswan service as culprit.  If I stop service I'm able vmotion back and forth without issue.  The moment I start it I can vmotion 1 time.  But after the 2nd vmotion - hangs.

Any ideas why strongswan is causing this behavior?

Hello.

I'm encountering a lingering issue from the expired LE cert with NTOPNG external repo.
I've just confirmed the issue is with my specific install as I tested on another opnsense template I have and working fine.

I can update opnsense using default repo + sunny valley repo.
NTOPNG repo returning:

Updating ntop repository catalogue...
Certificate verification failed for /O=Digital Signature Trust Co./CN=DST Root CA X3
2602498781184:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:

I've tried re-locating /usr/local/etc/pkg/repos/ntop.conf up one dir.
Then tried re-adding repo with:
pkg add https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/ntop-1.0.txz
Same cert error and doesn't add repo.

I'm thinking issue is because ntopng was configured with old LE cert which was same cert as webui.  (Which I sinced refreshed all my certs)  My opnsense template that updated NTOPNG successfully always been configured with default webui self sign cert.

Anybody know how I can get NTOPNG cert reset?  Thanks.

Really struggling with getting a few Apache virtual servers to work consistently behind Opnsense NGINX.
I know its more of Apache2 + NGINX thing here - But I'm bit confused with the Opnsense wizardry not using standard nginx configs that I reference online.

Which is my first question.  Possible to completely override the opnsense NGINX plugin config - and use basic config?  Is that supported or must use GUI?
-----

I have a single page website - index.html being served via port 80.  It has a javascript that connects to a external yellowpages API server also on port 80.

yp.mydomain1.com.conf
<VirtualHost *:80>
        ServerName yp.mydomain1.com
        DocumentRoot /home/www/yp.mydomain1.com
        DirectoryIndex index.html
        ErrorLog ${APACHE_LOG_DIR}/wp.mydomain1.com-error.log
        CustomLog ${APACHE_LOG_DIR}/wp.mydomain1.com-access.log combined
</VirtualHost>

Setup the 4 NGINX components. 
Server, Local IP, 80
Location:  /
HTTP Server:  yp.mydomain.com1 , 80

It works.  THEN.
I'm also wanting apache multi-site wordpress.  Doing the following gets me my first working TLS offload wordpress site, but then breaks my yellowpages site.

wordpress confg
                                                                                                                                                        mydomain2.com.conf                                                                                                                                                                         
<VirtualHost *:80>

        ServerName mydomain2.com
        ServerAlias www.mydomain2.com

        DocumentRoot /home/www/mydomain2.com
        DirectoryIndex index.html index.php
       
        <Directory /home/www/mydomain2.com/>
        Options FollowSymLinks
        AllowOverride Limit Options FileInfo
        DirectoryIndex index.php
        Require all granted
        </Directory>
        <Directory /home/www/mydomain2.com/wp-conent>
         Options FollowSymLinks
         Require all granted
        </Directory>

ErrorLog ${APACHE_LOG_DIR}/mydomain2.com-error.log
CustomLog ${APACHE_LOG_DIR}/mydomain2.com-access.log combined
</VirtualHost>

--------

wp-config.php  (A LOT of time)

define('WP_HOME', 'http://mydomain2.com');
define('WP_SITEURL', 'http://mydomain2.com');

/** Make sure WordPress understands it's behind an SSL terminator */
define('FORCE_SSL_ADMIN', true);
define('FORCE_SSL_LOGIN', true);

if ($_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
$_SERVER['HTTPS']='on';
---------------
.htaccess

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
----------------

Opnsense NGINX:
Server:  LOCAL IP, Access 80
Location:  URL /
HTTP server:  mydomain2.com, X-Forward, 443 + 80, Cert
---------

So now I have my first TLS offload wordpress with wp-admin functional (although port 80 to 443 redirect rule not working yet - must provide https) , however now my yellowpages site now always returns 404. I've played with URL pattern.  I've played with NGINX specifying root dir + index.html - No go.  I'm lost with http header troubleshooting.  Not a web dev...but this shouldn't be so difficult. 
Any suggestions welcome.  Thanks.

Successfully upgrade to 21.1
Now attempting final update to get all packages current.

The first attempt hung at downloading #7 readline-8.1.0.txz
Since they where still downloading I left it for ~20min then did graceful reboot.
Then tried again with sensei + surcata off
This time it started with readline-8.1.0txz and downloaded
Now hung again at downloading #5- postfix-sasl-3.5.9,1.txz
I do have functional internet while its hung on update.

System log has
2021-02-18T15:27:22   kernel   842.209109 [1174] netmap_extra_free breaking with head 1734082816   
2021-02-18T15:27:22   kernel   842.209078 [1071] netmap_obj_free ouch, double free on buffer 2025

Any suggestions be helpful.  Is a VM on ESXi v7
Thanks

Making changes to environment for WIFI client bridge / Cell phone hotspot failover.

Adding additional VMXNET3 nics on existing VM throws entire VMXNET3 to VMX# out of whack.  Breaks entire FW.  One needs to re-assign going through manual pairing referencing MAC on VM to VMX MAC shown in Opnsense.  Assign port groups appropriately.  It works but kinda dirty - especially if not documented for the next guy.

So - my approach in past 20.1 has been OPNsense must just always have all 10 NICs added - so if future changes are required, then VMXNET3 NICs already present, thus no re-ordering on next boot.  I just use NULL Vmware Portgroup with no uplinks + not connected for unused NICs.

Just tried a a fresh 20.7 install with all 10 VMXNET3 from the start.  Same behavior of random ordering.
Reading up a little bit - appears to be an ongoing issue with more than 4 VMXNET3 that throws ordering out of whack.

Anybody have a trick that I can manually adjust for a 1:1 assignment.  I want Network adapter 1 in VM to = VMX0 , Network adapter 2 in VM = VMX1 , ect.

Thanks

Having played / used Opnsense for ~4 months I just stopping by to share how awesome I think it is...
Coming from having an environment with Cisco FMC / FTD is night and day.  The irony.

Recently I did upgrade from 20.1.9 to to 20.7.2.  Using VMware I do have vmx interfaces and did run into known netmap issue with both Sensei + Suricata.  Following https://www.sunnyvalley.io/post/opnsense-kernel-netmap-status I was able to fix.  (running:  opnsense-update -kr 20.7.2-netmap)

When I first tested it out I was foolish and only gave opnsense a 20GB disk.  So part of this upgrade to 20.7 was also to migrate to a fresh install.  This is what got me even happier with opnsense today to make this post.  Doing a full backup / restore to new VM with 60GB disk worked flawlessly.  Everything restored except for disable hardware offloads from what I can tell.  And I am using quite few plugins.  Even the mismatching Vmware interfaces restored just in the order I wanted. Just pure awesome.

Also briefly share that on my lab environment which is 2x Dell PER710s with Dual X5650s running vmware 2 node VSAN cluster the Vmotion / failover also works flawlessly.  Quite pleased with Opnsense.

Thank you Devs!