Unable to access self hosted websites across vlans

[Gryphon]

I have two VLANS, 100 for my personal devices, and 110 for my website servers.

At 110.2 is my Caddy reverse proxy. I have a port forward rule that forwards http & https traffic to 110.2. If someone is off my network entirely, they can connect to my website just fine.

From any of the computers on 100, for example 100.53, they cannot connect to the websites through their FQDNs. If I type the internal IP into the browser, it connects fine. In Firewall:Settings:Advanced I have "Reflection for port forwards" and "Automatic outbound NAT for Reflection" turned on.

Using traceroute to the FQDN, it does a single hop straight to my external IP address. Doing packet capture in OPNsense, if I curl the website I can see packets being blocked from 100.53 -> 110.2. VLAN 100 has a firewall rule that explicitly allows all traffic internally.

I have looked through all sorts of different pieces of documentation to try and fix this. I setup manual hairpin NAT exactly as shown here to no avail. Everything has said that I am doing the right thing.

I am at a total loss with this. Two days ago it was working. All I did was try to set up automatic wireguard VPN tunnelling for all external traffic. It was extremely finicky, so I turned it all off and carefully deleted everything (gateways, interfaces, firewall rules). Somewhere in the process of setting wireguard up I was no longer able to access the websites. I feel like a massive idiot for not backing up before doing the changes.

Any help is appreciated. I have been tearing my hair out for about 10 hours at this point.

[viragomann]

Instead of enabling NAT reflection, configure a host override for your domains in you internal DNS (e.g. Unbound). So your internal devices get the local address, when resolving the website instead of the public.

Doing packet capture in OPNsense, if I curl the website I can see packets being blocked from 100.53 -> 110.2. VLAN 100 has a firewall rule that explicitly allows all traffic internally.

This is presumably due to asymmetric routing. Enable the firewall rule logging and check the TCP flag.

[Gryphon]

Instead of enabling NAT reflection, configure a host override for your domains in you internal DNS (e.g. Unbound). So your internal devices get the local address, when resolving the website instead of the public.

I really do not want to maintain a second list of subdomains in OPNsense if at all possible. A wildcard would not work as not every address in my domain points to that public address. A general solution is far more preferable.

This is presumably due to asymmetric routing. Enable the firewall rule logging and check the TCP flag.

Hm. I have realized that I somewhat mixed two of my states I apologize. A rule wasn't applied from what I can tell, and just enabling the logging didn't seem to give any useful results. Here is a better description of what is going on.


Automatic Reflection
This setup is simple. In Firewall:Settings:Advanced I have "Reflection for port forwards" and "Automatic outbound NAT for Reflection" turned on. There is a Firewall:NAT:Outbound rule for 443 and 80, pointing them to 110.2, with automatically generated Rules made.

Windows
Curl from windows will fail to respond. In the firewall logs all I can see is a pass going 100.53 -> 110.2, and a pass coming back. It doesn't make any sense that there are two passes yet the response fails.

Linux
Curl from Linux will also fail to respond. In the firewall logs I can see blocks going 100.51 -> 110.2. There are no rules set anywhere that should differentiate these two hosts.

Manual Reflection
Both system settings for reflection in Firewall:Settings:Advanced are turned off. In Firewall:NAT:Outbound there are two rules, one for 443 and one for 80.

Interface: vlan0.110
Source: vlan0.110 net
Source Port: *
Destination 110.0/24
Destination Port: 443/80
NAT Address: vlan0.110 address
NAT Port: *
Static Port: NO

Windows and Linux
The same thing happens on both OSes now. The request seems to be re-routed to point at OPNsense's webUI. No firewall logs are generated related to 110.2.


I get the feeling that the manual reflection setup is close, but I'm not knowledgeable enough to actually see the solution.

[viragomann]

Just a question before we go on with troubleshooting this issue. I reread you initial post and noticed that it's a Caddy reverse proxy, which you're forwarding the traffic to and need to access from the other VLAN.
Why don't you run Caddy on OPNsense and let it listen on the WAN IP?

This would make the use of the WAN IP inside your network much easier. It just requires to allow access to it from inside, which might be given anyway already.

[Gryphon]

Why don't you run Caddy on OPNsense and let it listen on the WAN IP?

A very reasonable question. I found OPNsense's Caddy plugin incredibly frustrating to use, and it could not do a few things that standalone could. Partially related to my lack of desire to keep another list of domains, and also related to some niche capabilities.

[Patrick M. Hausen]

Odd. I found OPNsense's Caddy plugin to be the most straightforward and easy to use reverse proxy ever.

@Gryphon maybe open a new thread about your Caddy concerns? Technically a reverse proxy on "WAN address" is the best method to publish services to the Internet and the local network.

[Gryphon]

@Gryphon maybe open a new thread about your Caddy concerns? Technically a reverse proxy on "WAN address" is the best method to publish services to the Internet and the local network.

There are certain things that I do which I know are completely impossible with the Caddy plugin. The other methods of achieving the same results as I currently have involve systems which I strongly dislike using due to how clunky and frustrating to manage they are. Solving this forwarding problem will be a single solution that should work indefinitely and is what I would like to pursue.