For some reason after the update to 25.1, i cant access my domain interface anymore. Im connecting via wireguard to my network and try to access 'photos.mydomain.com' and it does not load.
The specific apps im having problems with: immich, jellyfin, plex.
Im using: Unraid (with SWAG/nginx). Other apps seem to load fine. But im having issues with those 3. I dont even see any logs for the requests being sent to my nginx, so it seems like whatever the update changed on opnsense, may have caused it. (Was working fine in 24.7.12)
im starting to notice the same, 0 changes from before the upgrade
I'm experiencing what seems to be a very similar issue after the update to OPNsense 25.1. Like you, I use WireGuard to connect to my network, but in my case, I'm using Private Internet Access (PIA) and rely on their port forwarding feature.
Before the update, everything was working perfectly. I'm using a script https://github.com/FingerlessGlov3s/OPNsensePIAWireguard to manage the WireGuard tunnel and automatically retrieve the assigned forwarded port from PIA. This script creates the WireGuard tunnel and dynamically configures the OPNsense firewall rules.
My primary issue is that Plex, which relies on external access via the forwarded port through PIA, is no longer accessible. The script still appears to be functioning correctly: the tunnel establishes, the port is retrieved, and my NAT rule is in place to forward traffic on that port from the WireGuard interface to my Plex server.
To troubleshoot, I've taken a few steps to isolate the problem:
Replaced Plex with a Minimal Webserver: I set up a simple webserver listening on the same port that Plex uses. I can see connections hitting the firewall logs on OPNsense, confirming that traffic is arriving at the correct port and being processed by the NAT rule. However, the webserver page fails to load in my client's browser. It just says the site did not load.
Confirmed Tunnel Integrity: The WireGuard tunnel itself seems to be stable, as other services that don't rely on port forwarding from PIA are working without issue.
So it seems like its a bug with the new update and NAT. I dont really want to find a workaround if its an issue with OPNSense. Hopefully a fix is pushed out for it.
Without showing the NAT rules in question, some info on the network topology, and preferrably packet traces, there is little chance anything will be fixed.
Don't assume a problem you experience is widely known. Always assume it's particular to your specific setup. Seriously.
Not claiming there is no bug, or "it's your fault" or some such - but the most important part of a bug report is "how to reproduce". "There's a bug in NAT" is not a problem description. Neither is "I cannot connect", sorry.
If everybody using NAT reflection was experiencing the same problem, Q&A would probably have caught it before shipping. And even if not, the forum would now be full with reports. Apparently that is not the case.
My updates went completely painless apart from some cosmetic issues in the dashboard.
Hi Patrick,
Thanks for the response. If I can provide the information you requested should I post it here or start a new thread specific to my issue. I am pretty sure that I am having the same issue and I can recreate it. I even rolled back to 24.7 and reupdated to 25.1 and was able to reproduce the problem.
Quote from: Patrick M. Hausen on February 06, 2025, 06:45:31 PMWithout showing the NAT rules in question, some info on the network topology, and preferrably packet traces, there is little chance anything will be fixed.
Don't assume a problem you experience is widely known. Always assume it's particular to your specific setup. Seriously.
Not claiming there is no bug, or "it's your fault" or some such - but the most important part of a bug report is "how to reproduce". "There's a bug in NAT" is not a problem description. Neither is "I cannot connect", sorry.
If everybody using NAT reflection was experiencing the same problem, Q&A would probably have caught it before shipping. And even if not, the forum would now be full with reports. Apparently that is not the case.
My updates went completely painless apart from some cosmetic issues in the dashboard.
Sorry, first time im using the forum here. Wasnt sure what else to put in the description
I'm not really sure how to replicate/demonstrate the issue or what settings would be useful to post here. I just started to notice it today and was curious if i was alone or not in the matter. Nothing changed in my setup besides the version from 24.7.12 to 25.1.
I suggest we just continue in this thread. Thanks.
Quote from: Patrick M. Hausen on February 06, 2025, 10:18:34 PMI suggest we just continue in this thread. Thanks.
okay sounds good. I already made a forum post and cannot see a way to delete it.
I'm experiencing an issue after updating to OPNsense 25.1, regarding external access to services through a WireGuard tunnel. I'm using Private Internet Access (PIA) and rely on their port forwarding feature.
Before the update, everything worked flawlessly. I use a script (https://github.com/FingerlessGlov3s/OPNsensePIAWireguard) to manage the WireGuard tunnel, automatically retrieve the assigned forwarded port from PIA, and dynamically configure the OPNsense firewall rules.
My primary issue is that Plex, which requires external access via the forwarded port from PIA, is no longer accessible. The script still appears to be functioning correctly: the tunnel establishes, the port is retrieved from PIA, and my NAT rule is in place to forward traffic on that port from the WireGuard interface (wg2) to my Plex server.
To troubleshoot, I've taken several steps to isolate the problem:
Replaced Plex with a Minimal Webserver: I set up a simple webserver and created a new NAT rule. I can see connections hitting the firewall logs on OPNsense, confirming that traffic is arriving at the correct port and being processed by the NAT rule. However, the webserver page fails to load in my client's browser. It just shows "site cannot be reached" or a similar error.
Confirmed Tunnel Integrity: The WireGuard tunnel itself seems stable, as other services that don't rely on port forwarding from PIA are working without issue. This suggests the core WireGuard connection is healthy.
Verified Firewall and NAT Rule Activity: As additional context, I've included screenshots of my NAT rule and the corresponding allow rule on the WireGuard interface. I've confirmed that both rules are active.
I have also included pictures of the firewall logs showing the incoming connection and it being redirected to the right ip/port.
I've captured packet traces on my WireGuard interface. These packet traces show that the TCP SYN packets from the external client reach the firewall via the WireGuard interface. However, despite this, a TCP connection cannot be established.
I've performed a rollback to OPNsense 24.7, and the issue is immediately resolved. After confirming functionality in 24.7, I re-upgraded to 25.1, and the problem reappears.
I'm including these packet traces and screenshots to provide as much detail as possible. Thanks.
Can you add a bit about how this PIA port forwarding works? Where is the connection terminated from the outside and which system is forwarding which port where? Thanks!
PIA assigns the port on their end, and they provide a script (https://github.com/pia-foss/manual-connections/blob/master/port_forwarding.sh) that allows me to request a port for forwarding traffic through their network and over the WireGuard VPN tunnel I have set up with PIA.
For testing, I set up a basic web server running on port 80 on an internal machine. When the script runs, PIA assigned me, for example, port 51476 for forwarding. I then configured a NAT rule in OPNsense to watch for incoming TCP traffic on port 51476 on the WireGuard interface (wg2) and forward it to my test web server's internal IP address on port 80.
I know that some port forwarding is happening, but it's not working correctly, because I can see incoming connections hitting the OPNsense firewall. I have provided firewall log images showing these connections. These logs indicate the NAT rule is being triggered and appears to be routing traffic correctly, but I cannot establish a full TCP connection with the web server from the outside. The connection either times out, or I receive a reset (RST) packet.
To clarify your question about where the connection is terminated from the outside:
The initial connection is terminated at PIA's servers. When an external client attempts to connect to my service, they connect to PIA's public IP address on the assigned forwarded port (e.g., 51476). PIA then forwards that traffic through the WireGuard tunnel to my OPNsense firewall.
My OPNsense firewall then receives the traffic on the WireGuard interface (wg2) and, based on the NAT rule, forwards it to my internal web server on port 80. The web server is not directly exposed to the internet; it's behind the OPNsense firewall and NAT.
I hope this clarifies the port forwarding setup. Please let me know if you need any further details!"
Just asking for clarity here: You say that this worked before 25.1 including Plex? The reason I am asking is that while Plex can have a different port than 32400, but it must know which IP to connect to. Since you cannot specify a DNS name in Plex, it probably is essential to use the same IP for inbound and outbound traffic, which is potentially (or per default) not the case.
The PIA approach seems to be that they provide you with a public IP and an abitrary port, which you could use as a target for inbound by specifying it directly or via DNS. All of your normal outbound traffic would go over the external NATed IP your ISP provides. This is all that Plex can see, so it would try to connect back to the IP that was reaching out to them.
So, IMHO, I think you would also have to direct all outbound Plex traffic over your PIA IP, maybe that is the problem. IDK what magic the PIA script does, but potentially, it has not been modified to work with 25.1, yet.
Quote from: meyergru on February 07, 2025, 11:45:09 AMJust asking for clarity here: You say that this worked before 25.1 including Plex? The reason I am asking is that while Plex can have a different port than 32400, but it must know which IP to connect to. Since you cannot specify a DNS name in Plex, it probably is essential to use the same IP for inbound and outbound traffic, which is potentially (or per default) not the case.
The PIA approach seems to be that they provide you with a public IP and an abitrary port, which you could use as a target for inbound by specifying it directly or via DNS. All of your normal outbound traffic would go over the external NATed IP your ISP provides. This is all that Plex can see, so it would try to connect back to the IP that was reaching out to them.
So, IMHO, I think you would also have to direct all outbound Plex traffic over your PIA IP, maybe that is the problem. IDK what magic the PIA script does, but potentially, it has not been modified to work with 25.1, yet.
Yes, I dont run any PIA scripts or anything. I had my plex setup working as follows: plex.mydomain.com would redirect me internally and externally, same with jellyfin.mydomain.com (plex settings > network > custom server url I have my domain). I did get my immich to run/load internally, thats due to me enabling my cloudflare proxy, jellyfin and plex are the only 2 apps that I dont route through CF proxy. So it seems to point to an issue with the 'hairpin NAT"
My port forwards consist of 2 things:
80/443 --> point to my SWAG(nginx) instance
my NAT has 2 settings enabled:
Reflection for port forwards and automatic outbound nat for reflection.
I updated to v25.1 and changed 0 settings, just ran the update and let it do its thing. If theres other info I can provide I can, I'm not sure what settings would be useful though, so just let me know :)
The simplest way to check whats up is if you create manual NAT rules using this tutorial page. If it works with them, then maybe there's something up with the automatic generation.
Please note that NAT is complicated.
https://docs.opnsense.org/manual/how-tos/nat_reflection.html
Quote from: Monviech (Cedrik) on February 07, 2025, 02:36:46 PMThe simplest way to check whats up is if you create manual NAT rules using this tutorial page. If it works with them, then maybe there's something up with the automatic generation.
Please note that NAT is complicated.
https://docs.opnsense.org/manual/how-tos/nat_reflection.html
Yeah i can do that and try, my only concern is that it was working before the update and nothing besides the version changed. So wasnt sure if others had experienced the same issue as well, if others have experienced the same issue, then it would point to something consistent in the update that changed.
Thats what we can find out. If you have snapshots of before and after the update, we could also compare the pf ruleset.
Just store whats in /tmp/rules.debug before and after the update and diff it for obvious changes regarding rdr or nat.
Quote from: meyergru on February 07, 2025, 11:45:09 AMJust asking for clarity here: You say that this worked before 25.1 including Plex? The reason I am asking is that while Plex can have a different port than 32400, but it must know which IP to connect to. Since you cannot specify a DNS name in Plex, it probably is essential to use the same IP for inbound and outbound traffic, which is potentially (or per default) not the case.
The PIA approach seems to be that they provide you with a public IP and an abitrary port, which you could use as a target for inbound by specifying it directly or via DNS. All of your normal outbound traffic would go over the external NATed IP your ISP provides. This is all that Plex can see, so it would try to connect back to the IP that was reaching out to them.
So, IMHO, I think you would also have to direct all outbound Plex traffic over your PIA IP, maybe that is the problem. IDK what magic the PIA script does, but potentially, it has not been modified to work with 25.1, yet.
I'm routing all my internet traffic, on the interface plex lives on, through the PIA VPN tunnel using a firewall rule that forces all internet traffic through the PIA tunnel.
The PIA server assigns a specific port to my VPN's internal IP address (for example, 10.10.8.2) and PIA routes to that via NAT. I then automatically update an alias on my firewall with this assigned port. This allows me to specify that port in Plex's remote access settings. A NAT rule is in place to forward any incoming traffic on that PIA-assigned port directly to my Plex server.
The key here is that PIA handles the port assignment. I've confirmed that I am receiving a port from them and that traffic on that port reaches my firewall. The problem occurs after the firewall - the connection is not successfully forwarded to Plex. I provied packet capture and firwall logs showing the connection were hitting the firewall in my previous posts.
This setup functioned perfectly on OPNsense version 23 and it survived the upgrade to all the way to 24.7 but now with 25.1 there are problems. I have been using this setup for over a year and through many Opnsense upgrades. I have even went as far as reinstalling 24.7 and restoring a backup which resulted in 100% working NAT over Wireguard using the setup I have explained. However, after upgrading to version 25.1. I can pull any logs or whatever is needed from 24.7 so just let me know what I can do.
Quote from: Monviech (Cedrik) on February 07, 2025, 02:46:28 PMThats what we can find out. If you have snapshots of before and after the update, we could also compare the pf ruleset.
Just store whats in /tmp/rules.debug before and after the update and diff it for obvious changes regarding rdr or nat.
I luckily backup my config every night. I did a quick compare on the XML, and the NAT section remained the same, no changes. The only differences were the UUID's that were added. Other than that, its pretty much the same.
Good that the rules are the same, that means that can be ruled out (pun not intended xD)
If VPN is involved, the next possible cause can be PMTU (Path MTU Discovery), since VPN reduces possible MSS sizes.
In 25.1 there have been some issues regarding that.
https://github.com/opnsense/src/issues/235
You could try to install the test kernel and see if that fixes your issues:
https://github.com/opnsense/src/issues/235#issuecomment-2636702333
Quote from: Monviech (Cedrik) on February 07, 2025, 03:35:37 PMGood that the rules are the same, that means that can be ruled out (pun not intended xD)
If VPN is involved, the next possible cause can be PMTU (Path MTU Discovery), since VPN reduces possible MSS sizes.
In 25.1 there have been some issues regarding that.
https://github.com/opnsense/src/issues/235
You could try to install the test kernel and see if that fixes your issues:
https://github.com/opnsense/src/issues/235#issuecomment-2636702333
I can definitely try that, but the issue is happening within the network as well. any device within the LAN cant reach my subdomains that arent routed through CF proxy.
Quote from: Monviech (Cedrik) on February 07, 2025, 03:35:37 PMGood that the rules are the same, that means that can be ruled out (pun not intended xD)
If VPN is involved, the next possible cause can be PMTU (Path MTU Discovery), since VPN reduces possible MSS sizes.
In 25.1 there have been some issues regarding that.
https://github.com/opnsense/src/issues/235
You could try to install the test kernel and see if that fixes your issues:
https://github.com/opnsense/src/issues/235#issuecomment-2636702333
This resolved my issue 100% thank you.
Quote from: Monviech (Cedrik) on February 07, 2025, 03:35:37 PMGood that the rules are the same, that means that can be ruled out (pun not intended xD)
If VPN is involved, the next possible cause can be PMTU (Path MTU Discovery), since VPN reduces possible MSS sizes.
In 25.1 there have been some issues regarding that.
https://github.com/opnsense/src/issues/235
You could try to install the test kernel and see if that fixes your issues:
https://github.com/opnsense/src/issues/235#issuecomment-2636702333
This also solved my issue
Working for me now :) Realized that i never tested the LAN, and only VPN. So at least my VPN is backup and running to access the domains :D
The kernel fix for the issue that some people here were having is going into 25.1.1 later this week.
Cheers,
Franco
I upgraded to 25.1 last night and did also notice issues with accessing my wireguard server in OPNsense. After a few hours of digging around, checking logs, firewall rules and various other settings, I found that a setting in Firewall normalization for my "WireGuard (Group)" was misconfigured and not allowing any peer's handshake to go through.
What fixed it for me was:
Firewall -> Settings -> Normalization -> "WireGuard (Group)" [or what ever your instance name is] -> Edit.
Direction was set to in, and needed to be set to "Any" according to the documentation.
Immediately after I changed this one setting, all of my WireGuard clients were able to connect again. I have no idea if this was a bug in the update (I'm not able to compare old configuration yet), or was just working in the old version out of sheer luck and broke when updated.
Anyway, I hope this helps someone else with this issue.
It does appear that this Normalization setting (on both 25.1 and 25.1.1) is getting corrupted following the upgrade from 24.x. I was able to get LAN traffic but not WAN following the upgrade with our Wireguard VPN setup. Re-applying this setting (mine still said "any") resolved the issue.
Thanks for the find GrantasarusRex!
Quote from: GrantasarusRex on February 10, 2025, 04:52:13 PMI upgraded to 25.1 last night and did also notice issues with accessing my wireguard server in OPNsense. After a few hours of digging around, checking logs, firewall rules and various other settings, I found that a setting in Firewall normalization for my "WireGuard (Group)" was misconfigured and not allowing any peer's handshake to go through.
What fixed it for me was:
Firewall -> Settings -> Normalization -> "WireGuard (Group)" [or what ever your instance name is] -> Edit.
Direction was set to in, and needed to be set to "Any" according to the documentation.
Immediately after I changed this one setting, all of my WireGuard clients were able to connect again. I have no idea if this was a bug in the update (I'm not able to compare old configuration yet), or was just working in the old version out of sheer luck and broke when updated.
Anyway, I hope this helps someone else with this issue.