I have two VLANS, 100 for my personal devices, and 110 for my website servers.
At 110.2 is my Caddy reverse proxy. I have a port forward rule that forwards http & https traffic to 110.2. If someone is off my network entirely, they can connect to my website just fine.
From any of the computers on 100, for example 100.53, they cannot connect to the websites through their FQDNs. If I type the internal IP into the browser, it connects fine. In Firewall:Settings:Advanced I have "Reflection for port forwards" and "Automatic outbound NAT for Reflection" turned on.
Using traceroute to the FQDN, it does a single hop straight to my external IP address. Doing packet capture in OPNsense, if I curl the website I can see packets being blocked from 100.53 -> 110.2. VLAN 100 has a firewall rule that explicitly allows all traffic internally.
I have looked through all sorts of different pieces of documentation to try and fix this. I setup manual hairpin NAT exactly as shown here (https://docs.opnsense.org/manual/how-tos/nat_reflection.html) to no avail. Everything has said that I am doing the right thing.
I am at a total loss with this. Two days ago it was working. All I did was try to set up automatic wireguard VPN tunnelling for all external traffic. It was extremely finicky, so I turned it all off and carefully deleted everything (gateways, interfaces, firewall rules). Somewhere in the process of setting wireguard up I was no longer able to access the websites. I feel like a massive idiot for not backing up before doing the changes.
Any help is appreciated. I have been tearing my hair out for about 10 hours at this point.
Instead of enabling NAT reflection, configure a host override for your domains in you internal DNS (e.g. Unbound). So your internal devices get the local address, when resolving the website instead of the public.
Quote from: Gryphon on September 29, 2025, 01:20:02 AMDoing packet capture in OPNsense, if I curl the website I can see packets being blocked from 100.53 -> 110.2. VLAN 100 has a firewall rule that explicitly allows all traffic internally.
This is presumably due to asymmetric routing. Enable the firewall rule logging and check the TCP flag.
Quote from: viragomann on September 29, 2025, 06:09:07 PMInstead of enabling NAT reflection, configure a host override for your domains in you internal DNS (e.g. Unbound). So your internal devices get the local address, when resolving the website instead of the public.
I really do not want to maintain a second list of subdomains in OPNsense if at all possible. A wildcard would not work as not every address in my domain points to that public address. A general solution is far more preferable.
Quote from: viragomann on September 29, 2025, 06:09:07 PMThis is presumably due to asymmetric routing. Enable the firewall rule logging and check the TCP flag.
Hm. I have realized that I somewhat mixed two of my states I apologize. A rule wasn't applied from what I can tell, and just enabling the logging didn't seem to give any useful results. Here is a better description of what is going on.
Automatic ReflectionThis setup is simple. In Firewall:Settings:Advanced I have "Reflection for port forwards" and "Automatic outbound NAT for Reflection" turned on. There is a Firewall:NAT:Outbound rule for 443 and 80, pointing them to 110.2, with automatically generated Rules made.
WindowsCurl from windows will fail to respond. In the firewall logs all I can see is a pass going 100.53 -> 110.2, and a pass coming back. It doesn't make any sense that there are two passes yet the response fails.
LinuxCurl from Linux will also fail to respond. In the firewall logs I can see blocks going 100.51 -> 110.2. There are no rules set anywhere that should differentiate these two hosts.
Manual ReflectionBoth system settings for reflection in Firewall:Settings:Advanced are turned off. In Firewall:NAT:Outbound there are two rules, one for 443 and one for 80.
Interface: vlan0.110
Source: vlan0.110 net
Source Port: *
Destination 110.0/24
Destination Port: 443/80
NAT Address: vlan0.110 address
NAT Port: *
Static Port: NO
Windows and LinuxThe same thing happens on both OSes now. The request seems to be re-routed to point at OPNsense's webUI. No firewall logs are generated related to 110.2.
I get the feeling that the manual reflection setup is close, but I'm not knowledgeable enough to actually see the solution.
Just a question before we go on with troubleshooting this issue. I reread you initial post and noticed that it's a Caddy reverse proxy, which you're forwarding the traffic to and need to access from the other VLAN.
Why don't you run Caddy on OPNsense and let it listen on the WAN IP?
This would make the use of the WAN IP inside your network much easier. It just requires to allow access to it from inside, which might be given anyway already.
Quote from: viragomann on September 30, 2025, 10:09:55 PMWhy don't you run Caddy on OPNsense and let it listen on the WAN IP?
A very reasonable question. I found OPNsense's Caddy plugin incredibly frustrating to use, and it could not do a few things that standalone could. Partially related to my lack of desire to keep another list of domains, and also related to some niche capabilities.
Odd. I found OPNsense's Caddy plugin to be the most straightforward and easy to use reverse proxy ever.
@Gryphon maybe open a new thread about your Caddy concerns? Technically a reverse proxy on "WAN address" is the best method to publish services to the Internet and the local network.
Quote from: Patrick M. Hausen on September 30, 2025, 11:54:15 PM@Gryphon maybe open a new thread about your Caddy concerns? Technically a reverse proxy on "WAN address" is the best method to publish services to the Internet and the local network.
There are certain things that I do which I know are completely impossible with the Caddy plugin. The other methods of achieving the same results as I currently have involve systems which I strongly dislike using due to how clunky and frustrating to manage they are. Solving this forwarding problem will be a single solution that should work indefinitely and is what I would like to pursue.
Really not sure where to go from here. Would still definitely appreciate any help that doesn't surround changing my Caddy configuration.
Quote from: Gryphon on September 30, 2025, 04:50:19 AMCurl from windows will fail to respond. In the firewall logs all I can see is a pass going 100.53 -> 110.2
No need to obscure private IP addresses at all. Nobody outside of your network will be able to access it.
Clear text makes troubleshooting easier.
Please post your NAT port forwarding rule and and the firewall rules on VLAN 100.
Run a packets capture on VLAN 100 and on VLAN 110, while you curl your website. Best to run both simultaneously.
Post the capture then.
Also check if Caddy logs anything regarding this.
It would also be helpful to enable the logging of default blocks and in all rules on both involved interfaces and post the firewall logs.
Quote from: viragomann on October 03, 2025, 09:44:33 PMNo need to obscure private IP addresses at all. Nobody outside of your network will be able to access it.
Clear text makes troubleshooting easier.
Fair, was just removing a bunch of what I thought to be redundant '10.10.' is all.
Port Forward(https://i.postimg.cc/kGMFVx0b/Port-Forward.png)
Outbound(https://i.postimg.cc/hv4bJ9km/NAT-Outbound.png)
Advanced Settings(https://i.postimg.cc/tJrdFsbC/Advanced-Settings.png)
Packet Capture on both VLANsTo be very clear, in this setup (called manual reflection in a past post) CURL is certainly being redirected to the OPNSense UI (which will throw an SSL error because .dev domains forcibly require proper https)
https://limewire.com/d/ENgvp#9aLgLWS3Ap
(If there is a better file sharing site I'd be happy to use it, this was just what I found quickly)
Quote from: viragomann on October 03, 2025, 09:44:33 PMIt would also be helpful to enable the logging of default blocks and in all rules on both involved interfaces and post the firewall logs.
I did this, but for some reason I do not see anything in any of Firewall:Log Files other than in Live View. I grabbed an image from after I sent a CURL request:
(https://i.postimg.cc/T1SzCYqV/Live-Firewall.png)
Well do you run the Opnsense WebUI on another port than 443 and 80?
Check in your administrative settings, and change to something like 8443 and disable the automatic redirect to free port 80.
NAT reflection is the way to go and it works (Im the guy who wrote this:
https://docs.opnsense.org/manual/how-tos/nat_reflection.html
If you wanne see which services listen locally on your OPNsense:
# sockstat -l
Yes, I also suspect that the OPNsense web interface is catching the traffic. Otherwise you should see also packets on the server interface going out.
You can change the listening port in System: Settings: Administration. Also disable HTTP Redirect.
Since you have disabled the global NAT reflection now, did you enable it in the port forwarding rules?
If you want to access your local server by the public IP, NAT reflection is necessary, or you nat the traffic manually on the Personal interface.
The shown outbound NAT rules do just hairpining for access from the server subnet, since it is stated as source. But I think, you don't need access from the servers themself.
For access from the Personal subnet there is no outbound NAT rule necessary, since requests go back to OPNsense anyway.
Quote from: Monviech (Cedrik) on October 04, 2025, 08:07:20 AMNAT reflection is the way to go and it works (Im the guy who wrote this:
https://docs.opnsense.org/manual/how-tos/nat_reflection.html (https://docs.opnsense.org/manual/how-tos/nat_reflection.html)
This page is what I have been going off of for the current setup, specifically method one. I made a mistake in the setup, I had not selected all of the interfaces on the port forward. I have changed that now.
(https://i.postimg.cc/cHf3Jc0h/Fixed-Forward.png)
Quote from: viragomann on October 04, 2025, 09:16:52 AMSince you have disabled the global NAT reflection now, did you enable it in the port forwarding rules?
If you want to access your local server by the public IP, NAT reflection is necessary, or you nat the traffic manually on the Personal interface.
Regarding this then, the page above specifically says to not turn that on. I will test with both for the sake of it.
Quote from: Monviech (Cedrik) on October 04, 2025, 08:07:20 AMWell do you run the Opnsense WebUI on another port than 443 and 80?
Quote from: viragomann on October 04, 2025, 09:16:52 AMYes, I also suspect that the OPNsense web interface is catching the traffic. Otherwise you should see also packets on the server interface going out.
You're both absolutely correct, and it was a little silly of me to forget that. I have fixed it.
Now, I have tried curling while watching the live firewall with the NAT reflection setting both on and off. The results are the same on both sides, but here are the live firewall outputs for both. Clearly we can see steps are being made in the right direction for sure, I am getting a redirect, the reflection rule is being triggered, a connection seems to get to Caddy, but then everything starts getting blocked for some reason.
Reflection Off(https://i.postimg.cc/C5Z4J8gX/Reflection-Off.png)
Reflection On(https://i.postimg.cc/6q7hzZXj/Reflection-On.png)
Quote from: Gryphon on October 04, 2025, 07:58:30 PMThis page is what I have been going off of for the current setup, specifically method one. I made a mistake in the setup, I had not selected all of the interfaces on the port forward. I have changed that now.
QuoteSince you have disabled the global NAT reflection now, did you enable it in the port forwarding rules?
If you want to access your local server by the public IP, NAT reflection is necessary, or you nat the traffic manually on the Personal interface.
Regarding this then, the page above specifically says to not turn that on. I will test with both for the sake of it.
Sure if you define the port forwarding rule on the Personal interface as well, NAT reflection is not needed then. Forwarding traffic from the other interfaces as well, is what NAT reflection does in fact.
Quote from: Gryphon on October 04, 2025, 07:58:30 PMClearly we can see steps are being made in the right direction for sure, I am getting a redirect, the reflection rule is being triggered, a connection seems to get to Caddy, but then everything starts getting blocked for some reason.
It's strange that you can see the same traffic passed and then blocked. I guess, it due to state violation. Click the "i" at the right and check the tcp flag to get sure.
Maybe there is something wrong in your network setup. Are the involved subnets VLANs by any chance?
Quote from: viragomann on October 05, 2025, 10:50:02 AMMaybe there is something wrong in your network setup. Are the involved subnets VLANs by any chance?
Yes, they are Vlans. I take it there is something I have missed then...
Yes, I suspect, that the VLANs are leaking somewhere outside of OPNsense, so that not every packet passes it.
As mentioned, check out the TCP flag of the blocked packets. Presumably they are not SYN packets. This would mean, that the respective SYN packet of the connection didn't pass OPNsense.
Normally only SYN packets are logged. If they are allowed, OPNsense sets a state, which passes following belonging packets.
Quote from: viragomann on October 06, 2025, 09:47:11 AMNormally only SYN packets are logged. If they are allowed, OPNsense sets a state, which passes following belonging packets.
From the image below (which I have re-posted for posterity), the first three packets that are allowed are SYN. Everything afterwards that is blocked are RST. At least I think so, they are 'S' and 'R' in the fields, not the full names.
(https://i.postimg.cc/6q7hzZXj/Reflection-On.png)
I'm not sure where exactly the packets could be leaking, though this is definitely something I'm completely unfamiliar with.
The passed packets are from client to server, probably HTTPS. The RST packets are a reaction of the client to a packet from the server that the client did not expect. So when the server answers the client, these packets seem to go past your OPNsense instead of through it.
Does the server have a network interface in the same network the client is connected to?
Quote from: Patrick M. Hausen on October 06, 2025, 04:00:32 PMDoes the server have a network interface in the same network the client is connected to?
Ah, when the last comment was mentioned leaking I had a thought that this might be the problem. Yes, 10.10.110.2 on VLAN110 is also on VLAN100 as 10.10.100.7. I can see how this might be an issue. Is there any way to avoid the problem without taking that machine off both interfaces?
Quote from: Gryphon on October 06, 2025, 07:58:54 PMQuote from: Patrick M. Hausen on October 06, 2025, 04:00:32 PMDoes the server have a network interface in the same network the client is connected to?
Ah, when the last comment was mentioned leaking I had a thought that this might be the problem. Yes, 10.10.110.2 on VLAN110 is also on VLAN100 as 10.10.100.7. I can see how this might be an issue. Is there any way to avoid the problem without taking that machine off both interfaces?
If the client and the server share a network why do you want the client traffic to go through OPNsense in the first place?
So options are:
- remove VLAN100 from server
- use address on VLAN100 directly for communication without going through OPNsense
- create outbound NAT rule on OPNsense on VLAN110 to NAT the connections from VLAN100 to the interface address.
What is important to understand:
IP is a connectionless protocol, so called packet switching as opposed to circuit switching. There is no "connection". The illusion of a connection is created at higher layers, e.g. TCP.
Unless special policy routing is in place a packet is routed bases on it's destination IP address only. And this decision is made for each individual packet without taking any history into account.
Specifically the server system has no knowledge about on which interface the packets from the client arrived when sending the answer. It sees a packet destined for the client, in VLAN100, it has got a direct connection to that network, it sends the packet via that network.
We get the same topic frequently in the TrueNAS forum when people try to run e.g. the web UI and the services (SMB and the like) via different VLANs.
A generic Unix system without any additional firewall or policy routing mechanisms configured has absolutely no knowledge about on which interface a packet it received actually came in. That information is simply not stored anywhere. And it will send out each packet to any destination via the "best", i.e. shortest way.
HTH,
Patrick
Quote from: Patrick M. Hausen on October 06, 2025, 08:13:47 PMIf the client and the server share a network why do you want the client traffic to go through OPNsense in the first place?
Optimally I don't want them to, and I would even prefer to have two instances of Caddy on the separate networks to remove the need for the current setup. In the past when I wanted to do that it wasn't possible due to my domain registrar, but I think it is possible now that I have switched to one that supports API access. Since that should be possible I'll just solve the problem that way.
I was unaware of how the network connections worked, I did not expect it to change the packets route in the way that it does. Thank you for the information.