Messages

I was ultimately able to get Comcast to roll back my firmware and prefix delegation is working again. It took about a week, but I also opened the case the Wednesday before Thanksgiving.

Comcast is aware of the issue. They will eventually roll out updated firmware, but it sounds like that will take time. If you are affected by this, you'll want to get a ticket opened and request a firmware rollback.

I lost my IPv6 prefix delegation about a week ago. Seriously dug into it yesterday and have packet captures of the modem telling me no prefix delegation.

For anyone else using anything besides the base /64 of your IPv6 statics, don't waste much time on this. There's a forum thread on the Comcast support that makes it clear this is an issue with the latest firmware. It even includes someone who got their modem swapped out. The swapped out modem came with older firmware. Prefix delegation worked again. A few days later, the new modem switched to the latest firmware and prefix delegation broke.

https://forums.businesshelp.comcast.com/conversations/ipv6/prefix-delegation-disabled/690fa973a2c50219bf21c6e6

It's pretty clear that firmware CGA4332COM_8.2p5s1_PROD_sey breaks prefix delegation for Comcast customers.

S-NAT has no impact on routing at all.

If you want to route certain traffic to the non-default gateway you need to add a Policy based routing rule.

Thanks!  I'd figured that out just before you posted. I hadn't found the documentation on it.

I am an idiot.

I needed a firewall rule for WiFi right before the default deny that routes all traffic through the Surf gateway.

I'm certain I'm an idiot and I'm missing something obvious.

I've have multiple LAN networks.  Until recently my WAN was solely a Comcast Business with static IPs, meaning I have an ipv4 /29 and an ipv6 /59.  I now have a Surf internet fiber connection.  Ultimately, I'd like to have some LANs (WiFi, IOT, wired family computers) use the new fiber connection but have my two server LANs (internal servers and DMZ servers) use the Comcast one.

I have always had Manual Outbound NAT rules working for all networks, along with a few outbound port and inbound port rules.  I don't remember if the need to set explicit Outbound NAT was due to the tutorial I followed for Comcast Business or if it was a desire to block the round robin IP usage.  In any case, it has been working.

I've been trying to just get the WiFi working, and am failing.  Details below, but in a nutshell, WiFi is 192.168.253.0/24 and when setting up a NAT using my new fiber, packet captures show an un-NAT'd 192.168.253.x IP going out my Comcast WAN, which is the default route. 

If I leave the existing Comcast Outbound NAT in place, everything works.  Packet captures show traffic leaving my Comcast WAN interface using the public IP set in that outbound NAT rule.

If I disable the Comcast Outbound NAT and set up a Surf Fiber Outbound NAT, traffic doesn't work.  Packet captures show the un-NAT'd 192.168.253.0/24 IP of devices on WiFi going out the Comcast WAN interface.  Since those are not routable packets, Comcast drops them.

Clearly SNAT isn't working when I'm trying to use Surf fiber.  I can't figure out what I'm doing wrong.

Gateway settings in GUI:

You cannot view this attachment.

I can ping both the Comcast and Surf upstream gateways.

This screenshot of the Outbound NAT rule for Surf fiber shows the rule disabled, but if I disable the Comcast Outbound rule for WiFi and enable this one, then the WiFi traffic is not SNAT'd and routes out the Comcast default route.

You cannot view this attachment.

So, while there are likely other things I'll need to tweak, it looks to me like the SNATing is not happening.  I'm certain I'm missing something obvious.

I agree with dseven.  Run a packet capture on the WAN. Assuming you see them, then alter your one to one NAT rule to log. Also create a rule on the LAN side to permit logging of packets to the Debian 12 box.

You do mention that even with the one to one in place, you are getting the wrong IP for outbound packets, so it's possible the packets are making it all the way to the Debian box, but the replies to the TCP handshake come from the .105 ip instead of .106. A packet capture on the Debian 12 box could be informative too.

I too have a /29. I've only done port forwards, not 1 to 1, so I can't help with settings there.

I will say you should do more to narrow down in what point in the process are things breaking. With that info, others that have worked with 1 to 1 NATs may have some insight.

Is it possible something on the Debian firewall is permitting LAN but not non-LAN?

I have noticed something similar after the upgrade to OpnSense 24.10.

In digging, I saw that /usr/local/zenarmor/output/active/temp was full.  I expanded Memory Disk Size in Settings / Reporting & Data / Database and ran a health check.  That indicated issues with MongoDb and suggested a reset or repair.  I tried a repair first and when that didn't work tried a reset.

ZenArmor was working again, but the expanded /usr/local/zenarmor/output/active/temp filled up again.

I've sent in a report, but my experience sounds very similar to the original report here.

Thanks!  Removing all the various certificates downloaded since the upgrade to 24.10 and re-running an update worked.

Code Select Expand

# rm /tmp/li
libfetch_crl.24101713  libfetch_crl.24102022  libfetch_crl.24102420
libfetch_crl.24101722  libfetch_crl.24102122  libfetch_crl.24102422
libfetch_crl.24101822  libfetch_crl.24102222  lighttpdcompress/
libfetch_crl.24101922  libfetch_crl.24102322
root@OPNsense:~ # rm /tmp/libfetch_crl.2410*

The tab complete shows roughly one a day since the upgrade on 17 Oct.

I am now at 24.10_7.

I'm also at 24.10 BE and receive the could not load CRL file error:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.10 at Thu Oct 24 20:15:38 UTC 2024
Fetching subscription information, please wait... Could not load CRL file /tmp/libfetch_crl.24102420
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/subscription: Authentication error
Fetching changelog information, please wait... Could not load CRL file /tmp/libfetch_crl.24102420
fetch: https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:14:amd64/24.10/sets/changelog.txz: Authentication error
Updating OPNsense repository catalogue...
Waiting for another process to update repository OPNsense
Updating SunnyValley repository catalogue...
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/meta.txz: Authentication error
repository SunnyValley has no meta file, using default settings
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/packagesite.pkg: Authentication error
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
Could not load CRL file /tmp/libfetch_crl.24102420
pkg: https://updates.zenarmor.com/opnsense/FreeBSD:14:amd64/24.7/${SUBSCRIPTION}/packagesite.txz: Authentication error
Unable to update repository SunnyValley
Error updating repositories!
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***

I did attempt

# opnsense-patch 372c9c98 70df0a15f7e
# /usr/local/opnsense/scripts/system/update-crl-fetch.py opnsense-update.deciso.com

and the update-crl-fetch.py worked.  However, a check for udpates in the GUI still fails.

I'm on 24.4.1_3 of the Business version of opnsense.

I had set up git, running on a proxmox VM, for configuration backups just under a month ago.  I recently noticed a crash report that's filled with php fatal errors of what looks to be the scheduled git backup saying that the ssh hostkey has changed.  See below.

Now, there is an outside chance that the hostkey was changed as I was tweaking some things.

The changes in backups are making it to git.  A test of the git in the backups section of configuration succeeds, so obviously even though this is a FATAL error, it's not blocking the functionality.

Where can I reset that key, so these stop happening.  I care because right now, I basically ignore the fact that there are Crash Reporter errors becuase I know they aren't errors.  I'd like to clear this so that a crash report means I should do something.

Sample php error:

Code Select Expand

[15-Aug-2024 15:17:00 Etc/UTC] PHP Fatal error:  Uncaught Exception: ssh hostkey changed in /usr/local/opnsense/mvc/app/library/OPNsense/Backup/Git.php:181
Stack trace:
#0 /usr/local/opnsense/scripts/system/remote_backup.php(17): OPNsense\Backup\Git->backup()
#1 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Backup/Git.php on line 181
[15-Aug-2024 17:17:00 Etc/UTC] PHP Fatal error:  Uncaught Exception: ssh hostkey changed in /usr/local/opnsense/mvc/app/library/OPNsense/Backup/Git.php:181
Stack trace:
#0 /usr/local/opnsense/scripts/system/remote_backup.php(17): OPNsense\Backup\Git->backup()
#1 {main}
  thrown in /usr/local/opnsense/mvc/app/library/OPNsense/Backup/Git.php on line 181

I'm on the same versions of opnsense and zenarmor as lwwarner.  I attempted the same update via the System: Firmware.  I saw the same fail to rename error, though my random bits at the end of the directory were different.

After that no ZenArmor pages would load.

I did a reinstall of the os-sensei package.  When I went to the ZenArmor dashboard and had a splash screen over it indicating that I had a different version and needed a ZenArmor restart.  After that all attempts to load any opnsense pages gave a 503 service unavailable error.

I ssh'd into opnsense and chose menu option 11: Reload all services.  That looks to be hung at "Info: conf-yaml-loader: Including configuration file custom.yaml." just after loading suricata.  However, the web interface and ZenArmor are back up.

I have suricata enabled.  I'd played with some rules, but ultimately disabled all rules.

I'm hoping my reload all services eventually comes back to a command prompt.  If it doesn't, I may pretend this is a Windows system and just restart.

A few comments:

That first link seems to imply this is from some blocking tool sending website info. That sounds like something like ZenArmor set up to respond with a blocking web page info.

Any chance you have dns filtering set up to respond on the WAN interface?

ShadowServer.org describes how they do this: https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-ddos-middlebox-report/

It's definitely doing a GET against a blocked URL. This is not something a stock opnsense box will do. This is being done by some plug-in mistakenly set to block WAN traffic.

If you have static IPs, you can sign up with ShadowServer to get reports on your IP's with exact timestamps.

Some more digging (with packet captures on lots of different interfaces), and I believe it is my switch not respecting defined vlans when it comes to multicast.

I'm playing with igmp snooping on the one interface I'd most like to isolate, but not having much luck thus far.

So, I've been playing with IPv6 after avoiding it for years, and after moving a linux server acting as a router to opnsense.

I've got a number of different networks including a separate IOT and DMZ.  I had permitted IPv6 on WiFi, IOT and DMZ.  I wanted to get a sense of what types of traffic I was seeing in the DMZ and turned on tcpdump to capture 1000 packets.

There are the expected things like router/neighbor advertisement/solicitation.

However, the bulk of the traffic is mdns addressed to the link local address of ff02::fb. 

The disturbing thing is that most of these are from iOS devices on WiFi.  ff02:: addresses are IPv6 link local multicast.  They are not supposed to be routed, as I understand it.

I have checked and as far as I can find, there is nothing handling mdns proxying.  I've played with it in the past to permit airprinting from WiFi to IOT using avahi on linux.  The avahi daemon is not running on that system.  Besides, this DMZ network is for moving various services currently hosted on that linux box.  The linux box has no route to the DMZ.

I did install the mdns repeater on opnsense, but it is currently disabled--and DMZ (where the packet captures happened) is not selected even in the disabled interfaces.

Do I need to do something special in my firewall rules to make link local multicast stay link local and not act as site local?  I've got explicit blocks from all LAN interfaces to DMZ except for my computer to manage them.  That's what makes me think this ff02::fb traffic is somehow coming from the opnsense box.

It's also possible I'm fundamentally misunderstanding something about IPv6 because it's all relatively new to me.

The subject is basically the question. I know DOH is much more common that DOT.  There's no ZenArmor policy for blocking DOT. Does the DOH block also block DOT or is there no way in ZenArmor to so that?