Messages

S-NAT has no impact on routing at all.

If you want to route certain traffic to the non-default gateway you need to add a Policy based routing rule.

Thanks!  I'd figured that out just before you posted. I hadn't found the documentation on it.

I am an idiot.

I needed a firewall rule for WiFi right before the default deny that routes all traffic through the Surf gateway.

I'm certain I'm an idiot and I'm missing something obvious.

I've have multiple LAN networks.  Until recently my WAN was solely a Comcast Business with static IPs, meaning I have an ipv4 /29 and an ipv6 /59.  I now have a Surf internet fiber connection.  Ultimately, I'd like to have some LANs (WiFi, IOT, wired family computers) use the new fiber connection but have my two server LANs (internal servers and DMZ servers) use the Comcast one.

I have always had Manual Outbound NAT rules working for all networks, along with a few outbound port and inbound port rules.  I don't remember if the need to set explicit Outbound NAT was due to the tutorial I followed for Comcast Business or if it was a desire to block the round robin IP usage.  In any case, it has been working.

I've been trying to just get the WiFi working, and am failing.  Details below, but in a nutshell, WiFi is 192.168.253.0/24 and when setting up a NAT using my new fiber, packet captures show an un-NAT'd 192.168.253.x IP going out my Comcast WAN, which is the default route. 

If I leave the existing Comcast Outbound NAT in place, everything works.  Packet captures show traffic leaving my Comcast WAN interface using the public IP set in that outbound NAT rule.

If I disable the Comcast Outbound NAT and set up a Surf Fiber Outbound NAT, traffic doesn't work.  Packet captures show the un-NAT'd 192.168.253.0/24 IP of devices on WiFi going out the Comcast WAN interface.  Since those are not routable packets, Comcast drops them.

Clearly SNAT isn't working when I'm trying to use Surf fiber.  I can't figure out what I'm doing wrong.

Gateway settings in GUI:

You cannot view this attachment.

I can ping both the Comcast and Surf upstream gateways.

This screenshot of the Outbound NAT rule for Surf fiber shows the rule disabled, but if I disable the Comcast Outbound rule for WiFi and enable this one, then the WiFi traffic is not SNAT'd and routes out the Comcast default route.

You cannot view this attachment.

So, while there are likely other things I'll need to tweak, it looks to me like the SNATing is not happening.  I'm certain I'm missing something obvious.

I agree with dseven.  Run a packet capture on the WAN. Assuming you see them, then alter your one to one NAT rule to log. Also create a rule on the LAN side to permit logging of packets to the Debian 12 box.

You do mention that even with the one to one in place, you are getting the wrong IP for outbound packets, so it's possible the packets are making it all the way to the Debian box, but the replies to the TCP handshake come from the .105 ip instead of .106. A packet capture on the Debian 12 box could be informative too.

I too have a /29. I've only done port forwards, not 1 to 1, so I can't help with settings there.

I will say you should do more to narrow down in what point in the process are things breaking. With that info, others that have worked with 1 to 1 NATs may have some insight.

Is it possible something on the Debian firewall is permitting LAN but not non-LAN?

I have noticed something similar after the upgrade to OpnSense 24.10.

In digging, I saw that /usr/local/zenarmor/output/active/temp was full.  I expanded Memory Disk Size in Settings / Reporting & Data / Database and ran a health check.  That indicated issues with MongoDb and suggested a reset or repair.  I tried a repair first and when that didn't work tried a reset.

ZenArmor was working again, but the expanded /usr/local/zenarmor/output/active/temp filled up again.

I've sent in a report, but my experience sounds very similar to the original report here.

Thanks!  Removing all the various certificates downloaded since the upgrade to 24.10 and re-running an update worked.

Code Select Expand

# rm /tmp/li
libfetch_crl.24101713  libfetch_crl.24102022  libfetch_crl.24102420
libfetch_crl.24101722  libfetch_crl.24102122  libfetch_crl.24102422
libfetch_crl.24101822  libfetch_crl.24102222  lighttpdcompress/
libfetch_crl.24101922  libfetch_crl.24102322
root@OPNsense:~ # rm /tmp/libfetch_crl.2410*

The tab complete shows roughly one a day since the upgrade on 17 Oct.

I am now at 24.10_7.

I'm also at 24.10 BE and receive the could not load CRL file error:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10 at Thu Oct 24 20:15:38 UTC 2024
Fetching subscription information, please wait... Could not load CRL file /tmp/libfetch_crl.24102420
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/subscription: Authentication error
Fetching changelog information, please wait... Could not load CRL file /tmp/libfetch_crl.24102420
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/sets/changelog.txz: Authentication error
Updating OPNsense repository catalogue...
Waiting for another process to update repository OPNsense
Updating SunnyValley repository catalogue...
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/meta.txz: Authentication error
repository SunnyValley has no meta file, using default settings
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/packagesite.pkg: Authentication error
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/packagesite.txz: Authentication error
Unable to update repository SunnyValley
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

I did attempt

# opnsense-patch 372c9c98 70df0a15f7e
# /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com

and the update-crl-fetch.py worked.  However, a check for udpates in the GUI still fails.

I'm on 24.4.1_3 of the Business version of opnsense.

I had set up git, running on a proxmox VM, for configuration backups just under a month ago.  I recently noticed a crash report that's filled with php fatal errors of what looks to be the scheduled git backup saying that the ssh hostkey has changed.  See below.

Now, there is an outside chance that the hostkey was changed as I was tweaking some things.

The changes in backups are making it to git.  A test of the git in the backups section of configuration succeeds, so obviously even though this is a FATAL error, it's not blocking the functionality.

Where can I reset that key, so these stop happening.  I care because right now, I basically ignore the fact that there are Crash Reporter errors becuase I know they aren't errors.  I'd like to clear this so that a crash report means I should do something.

Sample php error:

Code Select Expand

[15-Aug-2024 15:17:00 Etc/UTC] PHP Fatal error:  Uncaught Exception: ssh hostkey changed in /usr/local/opnsense/mvc/app/library/OPNsense/Backup/Git.php:181
Stack trace:
#0 /usr/local/opnsense/scripts/system/remote_backup.php(17): OPNsense\Backup\Git->backup()
#1 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Backup/Git.php on line 181
[15-Aug-2024 17:17:00 Etc/UTC] PHP Fatal error:  Uncaught Exception: ssh hostkey changed in /usr/local/opnsense/mvc/app/library/OPNsense/Backup/Git.php:181
Stack trace:
#0 /usr/local/opnsense/scripts/system/remote_backup.php(17): OPNsense\Backup\Git->backup()
#1 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Backup/Git.php on line 181

I'm on the same versions of opnsense and zenarmor as lwwarner.  I attempted the same update via the System: Firmware.  I saw the same fail to rename error, though my random bits at the end of the directory were different.

After that no ZenArmor pages would load.

I did a reinstall of the os-sensei package.  When I went to the ZenArmor dashboard and had a splash screen over it indicating that I had a different version and needed a ZenArmor restart.  After that all attempts to load any opnsense pages gave a 503 service unavailable error.

I ssh'd into opnsense and chose menu option 11: Reload all services.  That looks to be hung at "Info: conf-yaml-loader: Including configuration file custom.yaml." just after loading suricata.  However, the web interface and ZenArmor are back up.

I have suricata enabled.  I'd played with some rules, but ultimately disabled all rules.

I'm hoping my reload all services eventually comes back to a command prompt.  If it doesn't, I may pretend this is a Windows system and just restart.

A few comments:

That first link seems to imply this is from some blocking tool sending website info. That sounds like something like ZenArmor set up to respond with a blocking web page info.

Any chance you have dns filtering set up to respond on the WAN interface?

ShadowServer.org describes how they do this: https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-ddos-middlebox-report/

It's definitely doing a GET against a blocked URL. This is not something a stock opnsense box will do. This is being done by some plug-in mistakenly set to block WAN traffic.

If you have static IPs, you can sign up with ShadowServer to get reports on your IP's with exact timestamps.

Some more digging (with packet captures on lots of different interfaces), and I believe it is my switch not respecting defined vlans when it comes to multicast.

I'm playing with igmp snooping on the one interface I'd most like to isolate, but not having much luck thus far.

So, I've been playing with IPv6 after avoiding it for years, and after moving a linux server acting as a router to opnsense.

I've got a number of different networks including a separate IOT and DMZ.  I had permitted IPv6 on WiFi, IOT and DMZ.  I wanted to get a sense of what types of traffic I was seeing in the DMZ and turned on tcpdump to capture 1000 packets.

There are the expected things like router/neighbor advertisement/solicitation.

However, the bulk of the traffic is mdns addressed to the link local address of ff02::fb. 

The disturbing thing is that most of these are from iOS devices on WiFi.  ff02:: addresses are IPv6 link local multicast.  They are not supposed to be routed, as I understand it.

I have checked and as far as I can find, there is nothing handling mdns proxying.  I've played with it in the past to permit airprinting from WiFi to IOT using avahi on linux.  The avahi daemon is not running on that system.  Besides, this DMZ network is for moving various services currently hosted on that linux box.  The linux box has no route to the DMZ.

I did install the mdns repeater on opnsense, but it is currently disabled--and DMZ (where the packet captures happened) is not selected even in the disabled interfaces.

Do I need to do something special in my firewall rules to make link local multicast stay link local and not act as site local?  I've got explicit blocks from all LAN interfaces to DMZ except for my computer to manage them.  That's what makes me think this ff02::fb traffic is somehow coming from the opnsense box.

It's also possible I'm fundamentally misunderstanding something about IPv6 because it's all relatively new to me.

The subject is basically the question. I know DOH is much more common that DOT.  There's no ZenArmor policy for blocking DOT. Does the DOH block also block DOT or is there no way in ZenArmor to so that?

Fixed this.  I am an idiot.  What's worse, I didn't even describe enough in the original post to let anyone guess at the idiocy.

The multi-NIC OPNsense box is replacing a vlan aware linux box that has been acting as firewall/router and also hosting services on public IPs.  The machine I was testing printer connectivity to had had static routes set up to the other vlans using that still existing linux box.

I realized something weird was going on when I realized my ALL_LANS rule to permit this traffic wasn't showing any counters.

So packets were going my system -> linux router -> printer.  Return packets were going printer -> OPNsense, which dropped them because no state existed for the return packets.

EDIT: see second post for details.  I had static routes on the system I was trying to reach the printer using an older (still existing) router/firewall.

I'm clearly missing something with inter-LAN traffic rules.

I've got multiple VLANs on my network.  However, I have zero VLANs defined on my OPNsense box.  OPNsense has multiple NICs and each is plugged into a switch interface where the untagged traffic is the LAN defined for that interface.  So, VLANs, but just different LANs as far as OPNsense is concerned.

For the most part, I want all the various LANs isolated, but I do want to permit some LANs to contact a printer in the IOT LAN.

It appears that whatever I do, the return traffic for that (e.g. https) is getting blocked on the IN of the IOT firewall.

Here's a simplified explanation.  I'm actually trying to permit a few different ports, but this example just uses https/443 since that is easy to test.


WAN -- (WAN/igb0) OPNsense (LAN/igb5) -- LAN (192.168.252.0/24)
                 (opt3/igb3)
                      |
                      |
                     IOT (192.168.255.0/24)
                      |
                      |
                   printer (192.168.255.196)   


Traffic seems to make it from LAN to printer but is always blocked from printer back to LAN at the IOT interface.

Outbound NATs for all LANs.  All using the same WAN address.

I've created a Group called ALL_LANS.  That group includes LAN and IOT.

There's a rule before any DENIES in ALL_LANS that permits "ALL_LANS net" to connect to the printer IP/32 using TCP/UDP port 443.

I even added an explicit ALL_LANS rule that permits the src IP/32 of the printer using a src port of 443, though I would think that was not needed.  I added this after having issues reaching the printer from the default LAN.

The IOT chain of rules looks like:

1. permit any traffic to the IOT address for DNS

2. permit any ICMP traffic to the IOT address.

3. permit printer ip/32 src port 443 to any.

4. block any traffic to dst port DNS and log.

5. block any traffic to dst net LAN and log.

6. permit src IOT net to any.

7. Deny any to any and log.


When I try to connect a LAN computer to the printer via https, it fails, and firewall logs show it's the return traffic from IOT printer to LAN computer that is blocked.  I suspect this is due to the return traffic not being recognized as related traffic.  I'm stumped as to what to do.

With rules set as above, that return IOT printer traffic is blocked and logged using rule 5.  If I disable rule 5, it is blocked and logged using rule 7.  If I disable rule 7, it is blocked and logged using the automatically generated rule of "Default deny / state violation rule."

I'm terribly confused.  I'd think if this return traffic is causing a state violation, the block would always happend in the autogenerated rules.

I've even tried creating NATs for LAN/IOT and placing those above the default WAN nat for LAN/IOT.  That did not seem to make any difference.

I'm certain I'm missing something simple and obvious. Any suggestions or questions welcome.  Pointing out where I'm clearly an idiot is really welcome if it helps me fix this and solve my issue.