Messages

If the client and the server share a network why do you want the client traffic to go through OPNsense in the first place?

Optimally I don't want them to, and I would even prefer to have two instances of Caddy on the separate networks to remove the need for the current setup. In the past when I wanted to do that it wasn't possible due to my domain registrar, but I think it is possible now that I have switched to one that supports API access. Since that should be possible I'll just solve the problem that way.

I was unaware of how the network connections worked, I did not expect it to change the packets route in the way that it does. Thank you for the information.

Does the server have a network interface in the same network the client is connected to?

Ah, when the last comment was mentioned leaking I had a thought that this might be the problem. Yes, 10.10.110.2 on VLAN110 is also on VLAN100 as 10.10.100.7. I can see how this might be an issue. Is there any way to avoid the problem without taking that machine off both interfaces?

Normally only SYN packets are logged. If they are allowed, OPNsense sets a state, which passes following belonging packets.

From the image below (which I have re-posted for posterity), the first three packets that are allowed are SYN. Everything afterwards that is blocked are RST. At least I think so, they are 'S' and 'R' in the fields, not the full names.



I'm not sure where exactly the packets could be leaking, though this is definitely something I'm completely unfamiliar with.

Maybe there is something wrong in your network setup. Are the involved subnets VLANs by any chance?

Yes, they are Vlans. I take it there is something I have missed then...

NAT reflection is the way to go and it works (Im the guy who wrote this:

https://docs.opnsense.org/manual/how-tos/nat_reflection.html

This page is what I have been going off of for the current setup, specifically method one. I made a mistake in the setup, I had not selected all of the interfaces on the port forward. I have changed that now.

Since you have disabled the global NAT reflection now, did you enable it in the port forwarding rules?
If you want to access your local server by the public IP, NAT reflection is necessary, or you nat the traffic manually on the Personal interface.

Regarding this then, the page above specifically says to not turn that on. I will test with both for the sake of it.

Well do you run the Opnsense WebUI on another port than 443 and 80?

Yes, I also suspect that the OPNsense web interface is catching the traffic. Otherwise you should see also packets on the server interface going out.

You're both absolutely correct, and it was a little silly of me to forget that. I have fixed it.


Now, I have tried curling while watching the live firewall with the NAT reflection setting both on and off. The results are the same on both sides, but here are the live firewall outputs for both. Clearly we can see steps are being made in the right direction for sure, I am getting a redirect, the reflection rule is being triggered, a connection seems to get to Caddy, but then everything starts getting blocked for some reason.

Reflection Off


Reflection On

No need to obscure private IP addresses at all. Nobody outside of your network will be able to access it.
Clear text makes troubleshooting easier.

Fair, was just removing a bunch of what I thought to be redundant '10.10.' is all.

Port Forward


Outbound


Advanced Settings


Packet Capture on both VLANs
To be very clear, in this setup (called manual reflection in a past post) CURL is certainly being redirected to the OPNSense UI (which will throw an SSL error because .dev domains forcibly require proper https)
https://limewire.com/d/ENgvp#9aLgLWS3Ap
(If there is a better file sharing site I'd be happy to use it, this was just what I found quickly)

It would also be helpful to enable the logging of default blocks and in all rules on both involved interfaces and post the firewall logs.

I did this, but for some reason I do not see anything in any of Firewall:Log Files other than in Live View. I grabbed an image from after I sent a CURL request:

Really not sure where to go from here. Would still definitely appreciate any help that doesn't surround changing my Caddy configuration.

@Gryphon maybe open a new thread about your Caddy concerns? Technically a reverse proxy on "WAN address" is the best method to publish services to the Internet and the local network.

There are certain things that I do which I know are completely impossible with the Caddy plugin. The other methods of achieving the same results as I currently have involve systems which I strongly dislike using due to how clunky and frustrating to manage they are. Solving this forwarding problem will be a single solution that should work indefinitely and is what I would like to pursue.

Why don't you run Caddy on OPNsense and let it listen on the WAN IP?

A very reasonable question. I found OPNsense's Caddy plugin incredibly frustrating to use, and it could not do a few things that standalone could. Partially related to my lack of desire to keep another list of domains, and also related to some niche capabilities.

[Automatic Reflection]

Instead of enabling NAT reflection, configure a host override for your domains in you internal DNS (e.g. Unbound). So your internal devices get the local address, when resolving the website instead of the public.

I really do not want to maintain a second list of subdomains in OPNsense if at all possible. A wildcard would not work as not every address in my domain points to that public address. A general solution is far more preferable.

This is presumably due to asymmetric routing. Enable the firewall rule logging and check the TCP flag.

Hm. I have realized that I somewhat mixed two of my states I apologize. A rule wasn't applied from what I can tell, and just enabling the logging didn't seem to give any useful results. Here is a better description of what is going on.


Automatic Reflection
This setup is simple. In Firewall:Settings:Advanced I have "Reflection for port forwards" and "Automatic outbound NAT for Reflection" turned on. There is a Firewall:NAT:Outbound rule for 443 and 80, pointing them to 110.2, with automatically generated Rules made.

Windows
Curl from windows will fail to respond. In the firewall logs all I can see is a pass going 100.53 -> 110.2, and a pass coming back. It doesn't make any sense that there are two passes yet the response fails.

Linux
Curl from Linux will also fail to respond. In the firewall logs I can see blocks going 100.51 -> 110.2. There are no rules set anywhere that should differentiate these two hosts.

Manual Reflection
Both system settings for reflection in Firewall:Settings:Advanced are turned off. In Firewall:NAT:Outbound there are two rules, one for 443 and one for 80.

Interface: vlan0.110
Source: vlan0.110 net
Source Port: *
Destination 110.0/24
Destination Port: 443/80
NAT Address: vlan0.110 address
NAT Port: *
Static Port: NO

Windows and Linux
The same thing happens on both OSes now. The request seems to be re-routed to point at OPNsense's webUI. No firewall logs are generated related to 110.2.


I get the feeling that the manual reflection setup is close, but I'm not knowledgeable enough to actually see the solution.

I have two VLANS, 100 for my personal devices, and 110 for my website servers.

At 110.2 is my Caddy reverse proxy. I have a port forward rule that forwards http & https traffic to 110.2. If someone is off my network entirely, they can connect to my website just fine.

From any of the computers on 100, for example 100.53, they cannot connect to the websites through their FQDNs. If I type the internal IP into the browser, it connects fine. In Firewall:Settings:Advanced I have "Reflection for port forwards" and "Automatic outbound NAT for Reflection" turned on.

Using traceroute to the FQDN, it does a single hop straight to my external IP address. Doing packet capture in OPNsense, if I curl the website I can see packets being blocked from 100.53 -> 110.2. VLAN 100 has a firewall rule that explicitly allows all traffic internally.

I have looked through all sorts of different pieces of documentation to try and fix this. I setup manual hairpin NAT exactly as shown here to no avail. Everything has said that I am doing the right thing.

I am at a total loss with this. Two days ago it was working. All I did was try to set up automatic wireguard VPN tunnelling for all external traffic. It was extremely finicky, so I turned it all off and carefully deleted everything (gateways, interfaces, firewall rules). Somewhere in the process of setting wireguard up I was no longer able to access the websites. I feel like a massive idiot for not backing up before doing the changes.

Any help is appreciated. I have been tearing my hair out for about 10 hours at this point.

The email has been changed to required since there are two issuers inside. ZeroSSL requires an Email, Let's Encrypt does not. If one fails the other is tried automatically.

To keep it simple I required the Email. Just put in whatever ;D

Ah, that makes sense. Any idea why Let's Encrypt would be claiming I have no valid A or AAAA records? They're definitely there.

This looks fantastic, but somehow it has managed to feel more complicated to setup than just a small VM with Caddy on it lol. Followed the instructions for the firewall and everything else for the initial setup, added my DNS provider with all the keys. I did turn off the ability to automatically create records, as many of my services I absolutely don't want accessible from off the network.

Despite having a functional A record in my DNS, I see a bunch of errors from Let's Encrypt claiming that there are no valid A or AAAA records for the domain I had entered. It actually errored enough times before I noticed that I got locked out.

Also, I've definitely never needed an email for Let's Encrypt before, not sure what that's about if I'm being honest, but whatever. Really I'd like to figure out why they're claiming there are no valid records despite having been perfectly fine on my VM that I paused 10 minutes before setting up.