Messages

It depends on a lot of factors.

Hi.  In the past I've read about state violation rule but like you say, lots of factors, so I've never really felt that I understand when issue surfaces.  Which is not often.

In regards to last time with IPSEC.  I restarted IPSEC service multiple times on both firewalls and Phase 2s were good and established fresh.  Traffic wouldn't flow yet I made no further config changes, rebooted remote Opnsense and VPN came backup up and traffic flowed through tunnel.  So I'm not sure.

I know there's implicit deny all rule on ACLs, but on all interfaces I have a manual deny all rule.  So when I see "Default deny / State violation" I know its Opnsense's internal rule and something with TCP states is wrong.  Otherwise if it was regular flow - it would be logged denied against my manual rule.

Both locations use Opnsense LAN INT on /29 subnet connected to L3 switch VLANs.  I have Zenarmor set on Opnsense LAN INT.  Perhaps Zenarmor be culprit for your mentioned reasons states get messed up - duplicated packets back, packet reordering, etc?

Bump.

Hi.

Can anyone shed some light to help me better understand the Default deny / State violations rule?  The causes and fix?
In the past when traffic isn't flowing that should be and is logging blocked by state violation I just reboot.
But I'm curious about resolving without reboot. 

For example
Yesterday my Spectrum home connection did a IP change on me for first time in 5+ years.
I re-configured a IPSEC VPN on both sides for new IP but traffic wasn't flowing despite phase 2s online.
Then I saw all VPN tunnel traffic being denied by state violation rule.
Rebooted the remote Opnsense and VPN began flowing again.

Would have reset states table under Firewall --> Diag --> States --> Actions fix issue?

Probably unrelated as my unbound wasn't crashing the service but I was having constant unreliable DNS resolution back on 24.1.x.  Both internal 53 + external DOT 853.  DNS resolutions would randomly fail and then try second later and work.  Constant SERVFAILS.

The fix I found finally was interfacing binding.  For years I had unbound both "Network Interfaces" + "Outgoing Network Interfaces" on specific interfaces.  Upon setting both to ALL interfaces (0.0.0.0) all my unbound problems went away.  Unbound ACLs to control access.  Finally returned to being reliable.

Probably not your situation but worth a check if your not binding unbound all interfaces.

(Also had numerous other binding issues.  Like Webui not running at boot due to specific MGT interfaces and redis also not starting when not set on LAN interfaces.  Believe either changes to freebsd itself or IPv6 work being done on Opnsense - Specific interface binding became quite problematic if not set on ALL within last year.)

opnsense-patch -c plugins 1e23572

Patch is the fix, however my setup did require a full opnsense reboot after installing patch.

Probably best to open an issue on Github

Created github issue - https://github.com/opnsense/plugins/issues/4358

I reviewed it a bit more and attempting to edit any config results in error.  Hunch is a writable permission issue since nothing of my nginx config can be changed.

Believe main config directory is /usr/local/etc/nginx  , directory root / wheel , root has rwx , wheel only has r-x, other r-x.

Also - never used but looked into opnsense-revert tool.  But nginx has been on 1.34 since opnsense 24.7.x
I also don't understand the _2 version of os-nginx.  Per release notes its just 1.34 yet opnsense plugin is 1.34_2
https://github.com/opnsense/plugins/blob/stable/24.7/www/nginx/pkg-descr

Tried the patch - opnsense-patch -c plugins 1e23572

Unfortunately did not fix it.  Tried Nginx restart.  Reverted patch.

Still cannot edit existing Nginx ACL or change active ACL on HTTP server.

Edit: actually just editing HTTP server with zero changes and attempting to save results in error.

Needing to adjust some NGINX ACLs today and ACL issue.

Upon editing ACL to add new IP I am getting "Unexpected error, check log for details"
Then tried creating new ACL and it worked.  Then tried editing same ACL.  Same error.

So then created brand new ACL with all needed IPs and saved ok.
Then went to HTTP server and attempted to change ACL to new ACL.  Same error. 
Cannot change HTTP server ACL.  :-/

Also tried all same above with NGINX service stopped.  Same error.

Reviewed NGINX log, Opnsense general + backend logs. Don't see anything mentioned about NGINX config / ACL.

Anyone aware of NGINX ACL issues or suggestions?

OPNsense 24.7.8
os-nginx 1.34_2

Interface binding issue as a result of HA setup - VIP / CARP IP.
Redis had been always set to LAN interface in past.

The reason CLI command -  redis-server --port 6379 --daemonize yes - had worked is because its implied loopback interface.  In Webui not selecting any redis interface also make it also use loopback and work.  I assumed like MGT binding it would mean it would listen on all interfaces 0.0.0.0 , not correct.  Just leave it with no interface selected.

With the new HA setup, either VIP itself and/or my LAN INT IP NAT config caused this issue.
I have all downstream traffic flowing through fortigate in transparent mode.  I have it using LAN VIP IP downstream, rather than LAN INT real IP of each firewall.

Whichever cause with VIP or my NAT, if Opnsense plugins are only thing to use redis - Don't select an interface and use loopback.

Post Bump.

I've ignored this Redis WebUI issue / bug since Aug. when I first post.

Today I upgraded Opnsense HA to 24.7.5
Went very smoothly.  Kudos to devs.

However still have this same Redis GUI bug with WebUI.  Won't start.  And strangely I'm not seeing any log errors now when attempting to start via WebUI like I had before on 24.1.

But same as 24.1 , Redis continues to work just fine if I use CLI -  redis-server --port 6379 --daemonize yes
However while running WebUI shows that it isn't.

Tried another redis reset + uninstall / reinstall + mv /var/db/redis /var/db/redis-OLD
No luck.

The exact same situation in previous post:
https://forum.opnsense.org/index.php?topic=38845.15

Any suggestions would be appreciated.

Setup HA / CARP today on OPNsense 24.1.10_8.
LAN CARP VIP only and using single this DHCP WAN script to toggle master WAN INT
https://gist.github.com/spali/2da4f23e488219504b2ada12ac59a7dc
Outgoing NAT - LAN , Lan adresss --> LAN CARP VIP  (Source firewall traffic to use CARP LAN VIP downstream)
After a bit of struggle failover and sync is working.  (Make those OPT# match + administration - listen interfaces - All Recommended!)  IPSEC states don't though with this WAN toggle :(

Anyway - now noticing redis won't start.  Exact same error issue as discussed here
https://forum.opnsense.org/index.php?topic=38845.15

WARNING: The TCP backlog setting of 511 cannot be enforced because kern.ipc.somaxconn is set to the lower value of 128.

Warning: Could not create server TCP listening socket Real_Int_IP:6379: bind: Address already in use

Failed listening on port 6379 (tcp), aborting.

Just like other post.  I can CLI

redis-server --port 6379 --daemonize yes

It runs on same WebUI default port 6379 just fine.  Doesn't show running via WebUI.
And then can run Ntopng Enterprise via WebUI.

Other post was left without known cause...
Any ideas?  New HA setup shouldn't cause this right?

Edit:
Starting redis this way it is a fresh DB.
I haven't reset it via webui and HA sync shouldn't of reset it.  Only one way master to backup sync.  Hmm.

Hmm - I'm leaning toward the default opnsense repo itself got an update/change..?

Nope. The only change was DNS in your settings, and now things work properly.

Just to make sure everything is fine, do a health check in Firmware - Audit section

Disagree - Had nothing to do with my dns settings.

Only thing was to populate system --> settings --> general DNS 8.8.8.8 so that 24.1.8 with broken unbound could get access to repo.
I did the same 24.1.8 2nd update run twice before on 2 other VMs with same 8.8.8.8 general setting and it never provided unbound 1.20 those two times.
The 3rd time it found unbound-1.20.0_1

As soon as unbound working I removed dns server 8.8.8.8 from system - settings - general and my existing unbound config working just like it did before.

Edit:  Ran health checks - Good to go + firmware status refresh now the usual ~20-30 seconds rather than 5+ minutes it was doing before.

[2024-06-05T21:10:28-07:00 Notice pkg-static unbound upgraded: 1.19.3 -> 1.20.0_1]

I guess 4th times a charm.

Restored 24.1.6 VM , Disabled NGINX in preparation of of unbound breaking.
Did update to 24.1.8 which updated same as before.  Hung on snmp package extract + unbound failed to start on first 24.1.8 boot.
Set DNS under general settings.
2nd update run completely different result compared to previous 2nd update run (shared in previous post).  Among the changes was unbound update from 1.19.3 to 1.20.0_1 that wasn't there before.  As well as python 3.9 cleanup.

Unbound now works.  And Opnsense firmware section much more responsive.
Hmm - I'm leaning toward the default opnsense repo itself got an update/change..?

----
Here are changes of 2nd update of 24.1.8

2024-06-05T21:11:11-07:00   Notice   pkg-static   hiredis-1.2.0.15 installed   
2024-06-05T21:11:11-07:00   Notice   pkg-static   ndpi-4.8.d20240223,1 installed   
2024-06-05T21:11:10-07:00   Notice   pkg-static   lua54-5.4.6_1 installed   
2024-06-05T21:10:35-07:00   Notice   pkg-static   python39-3.9.18_2 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   nmap-7.94_3 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-typing-extensions-4.11.0 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-idna-3.7 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-tzdata-2024.1 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-sniffio-1.3.1 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-pylsqpack-0.3.18 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-setuptools-63.1.0_1 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-pycparser-2.22 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-packaging-23.2 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-zipp-3.18.1 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-exceptiongroup-1.2.0 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-pyasn1-0.6.0 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-attrs-23.2.0 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-vici-5.9.11 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-importlib-metadata-7.1.0 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-pyotp-2.9.0 deinstalled   
2024-06-05T21:10:34-07:00   Notice   pkg-static   py39-anyio-4.3.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-ujson-5.9.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-sortedcontainers-2.4.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-filelock-3.13.4 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-pysocks-1.7.1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-sqlite3-3.9.18_7 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-certifi-2024.2.2 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-webencodings-0.5.1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-yaml-6.0.1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-pytz-2024.1,1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-hpack-4.0.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-six-1.16.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-cffi-1.16.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-async_generator-1.10 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-outcome-1.3.0_1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-markupsafe-2.1.5 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-lxml-4.9.3 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-h11-0.14.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-soupsieve-2.0.1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-hyperframe-6.0.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-socksio-1.0.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-h2-4.1.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-html5lib-1.1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-urllib3-1.26.18_1,1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-python-dateutil-2.9.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-cryptography-42.0.5_1,1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-trio-0.25.0 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-beautifulsoup-4.12.3 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-openssl-23.2.0,1 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   py39-httpcore-1.0.5 deinstalled   
2024-06-05T21:10:33-07:00   Notice   pkg-static   php82-opcache upgraded: 8.2.18 -> 8.2.19   
2024-06-05T21:10:33-07:00   Notice   pkg-static   netdata reinstalled: 1.43.2_1 -> 1.43.2_1   
2024-06-05T21:10:28-07:00   Notice   pkg-static   shadowsocks-libev upgraded: 3.3.5_2 -> 3.3.5_3   
2024-06-05T21:10:28-07:00   Notice   pkg-static   unbound upgraded: 1.19.3 -> 1.20.0_1   
2024-06-05T21:10:27-07:00   Notice   pkg-static   ntopng upgraded: 6.1.240606 -> 6.0.d20240307_1,1   
2024-06-05T21:10:23-07:00   Notice   pkg-static   libzmq4 upgraded: 4.3.5 -> 4.3.5_2   
2024-06-05T21:10:23-07:00   Notice   pkg-static   libsodium upgraded: 1.0.18 -> 1.0.19

sh:/usr/local/lbexec/opnsense-auth: not found

Lab VM is broke cause pkg install opnsense-24.1.8 wouldn't reinstall back after removing it - need to duplicate 24.1.6 VM again.

I've already updated my 24.1.6 to 24.1.8 three times now.  Every update via opnsense updater resulted in broken unbound - and a somewhat sluggish opnsense.

Can see in previous posts that after the reboot of 24.1.8, the 2nd update run it does bunch of cleanup of python 3.9 (meaning doesn't do python cleanup during first 24.1.6 to 24.1.8 update)

After new lab VM I am going to try just manual update of unbound:
pkg install unbound-1.20.0

But will opnsense allow unbound update this way?  Or give another reference to opnsense-24.1.8 being marked as vital and prevent update of unbound 1.20.0 over top of existing 1.19.3?  I'll find out.

Ok believe I've identified issue...Its unbound version

Per 24.1.7 release notes it says unbound updated to unbound-1.20.0

Yet my 24.1.8 still has unbound-1.19.3 

Health check which appears to hang finally returned:
unbound-1.19.3 version mismatch, expected 1.20.0_1

So tried
pkg remove unbound-1.19.3
Installed packages to be REMOVED:
        opnsense: 24.1.8
        unbound: 1.19.3

pkg: Cannot delete vital package: opnsense!
pkg: If you are sure you want to remove opnsense,
pkg: unset the 'vital' flag with: pkg set -v 0 opnsense


pkg set -v 0 opnsense

pkg install opnsense-24.1.8

Installed packages to be UPGRADED:
        pkg: 1.19.2_1 -> 1.21.3 [FreeBSD]

pkg install unbound-1.20.0  (unbound-1.20.0_1 not found)

New packages to be INSTALLED:
        fontconfig: 2.15.0_2,1 [FreeBSD]
        freetype2: 2.13.2 [FreeBSD]
        graphite2: 1.3.14 [FreeBSD]
        libICE: 1.1.0_2,1 [FreeBSD]
        libSM: 1.2.3_1,1 [FreeBSD]
        libX11: 1.8.7_1,1 [FreeBSD]
        libXau: 1.0.9_1 [FreeBSD]
        libXdmcp: 1.1.5 [FreeBSD]
        libXext: 1.3.6,1 [FreeBSD]
        libXfixes: 6.0.0_1 [FreeBSD]
        libXrender: 0.9.10_2 [FreeBSD]
        libfontenc: 1.1.8 [FreeBSD]
        libxcb: 1.16.1 [FreeBSD]
        png: 1.6.43 [FreeBSD]
        unbound: 1.20.0 [FreeBSD]
        xorgproto: 2023.2 [FreeBSD]
        zstd: 1.5.6 [FreeBSD]


However upon reviewing installed packages opnsense 24.1.8 package did not install properly.
Re-attempted pkg install opnsense-24.1.8 and tells me not found in repo?

Now this lab VM is broken...I will re-attempt this one more time with just trying to update unbound rather than removing 1.19.3.  But it will likely tell me its locked?

Can anyone share manual install procedure of opnsense base / kernel or point to documentation? 
I'm unexperienced with manual package management as opnsense updater usually works.