Messages

Following up on this for anyone who may stumble on it in the future. The issue was not anything to do with the Layer4 configuration, but that I was redirecting traffic to three different servers all using their own certificates for the same wildcard domain, which caused browser issues when going back and forth across servers.

Everything works now that I've specified each individual subdomain in my Caddyfile vs using *.halp.app.

I tested this configuration and it worked for me:

Is there a chance it's because I'm forwarding subdomains to Caddy instances that are set up for wildcard domains?

No worries! I really appreciate the help you've given me so far.

I have since discovered that some of the subdomains still had Unbound overrides, which I've disabled. I'm now at a point where each of the subdomains tries to load externally, but comes up with a blank page (as opposed to throwing errors before).

I'm using a custom DNS module and a wildcard cert on the other Caddy instances, so I'm assuming I just need to configure something to allow OPNsense Caddy and them to speak properly.

Thanks, I just updated OPNsense to remove the port forward and configure the firewall rules.

Attached is an updated log file. It looks like the Layer4 proxy is working in that it's interpreting plan.halp.app and sending it to the right server - but nothing is loading in the browser. This is consistent across services when trying to access them externally.

Is there something special I need to enable on the internal Caddy instances on Servers 1, 2, and 3 to accept the layer4 proxy? It's almost like the traffic is being load balanced that only services on a single server work at a time.

It's not - I should have elaborated that I masked my real domain in the uploaded file for privacy reasons. I can confirm the domain and subdomains resolve correctly internally and externally - I've used the command line and various DNS lookup tools to ensure everything is pointing to my public IP address.

Within OPNsense, I also have a NAT port forward rule that takes HTTPS traffic with a destination of the WAN address and forwards it to OPNsense Caddy (192.168.1.1, port 443). In OPNsense Caddy settings, I've configured the port to 443 (it defaulted to 20443). My OPNsense install lives on port 440, not 443 - so I don't think that's the issue.

In a nutshell, this is what I'm experiencing:

I have three servers that serve different applications -

Server 1 - A, B, C
Server 2 - D, E, F
Server 3 - G, H I

When I restart Caddy and then externally visit one of Server 1's apps, they all work - but the services on Servers 2 and 3 do not load.

If I restart Caddy again and then try to access a service on Server 2, all of the apps on Server 2 now work - but the services on Servers 1 and 3 do not load.

The same thing happens if I restart and then start by loading a Server 3 app.

I've shared my Caddy config with masked domains if it helps.

Thanks for the quick response! I've attached the most recent lines from my Caddy log, where you can see I'm externally navigating to several subdomains unsuccessfully (nvr, plan, and sso).

The server I'm forwarding them to has a hostname of debian and Caddy exposed via port 443, which is also managing TLS via HTTPS.

Something that might be important to note: I've disabled Unbound overrides/split DNS while testing these subdomains. When I navigate to plan.halp.app from inside my network, the layer4 proxy seems to work fine. When I use my cell phone to navigate to plan.halp.app from outside my network, it does not work.

I've tried this on both Edge and Firefox mobile.

I have three servers running separate instances of Caddy for separate services that I'm trying to make externally available. To do so, I've deployed the Caddy plugin and have enabled Layer4 proxying.

I think the configuration is correct (TCP, TLS SNI) along with my basic Caddy setups on the other servers (with LE auto HTTPS) - but it only works intermittently.

By that, I mean some of the services can sometimes be accessed externally, but a few minutes later might become unavailable, and so on with other services.

Are there any watch-outs I should be on the lookout for to resolve this?

This worked perfectly - almost wish I had just asked here first! Thanks!

When I click 'Update' in the system settings, I receive the following error accompanied by the logs below:

The release type "opnsense" is not available on this repository.

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 24.1.10_8 at Tue Aug 27 09:40:02 EDT 2024
Fetching changelog information, please wait... fetch: https://pkg.opnsense.org/FreeBSD:14:amd64/24.1/sets/changelog.txz: Not Found
Updating OPNsense repository catalogue...
pkg: Repository OPNsense has a wrong packagesite, need to re-create database
Waiting for another process to update repository OPNsense
Updating mimugmail repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: ........ done
Processing entries: .......... done
mimugmail repository update completed. 202 packages processed.
All repositories are up to date.
pkg: Repository OPNsense has a wrong packagesite, need to re-create database
pkg: Repository OPNsense cannot be opened. 'pkg update' required
Checking integrity... done (0 conflicting)
Your packages are up to date.
pkg: Repository OPNsense has a wrong packagesite, need to re-create database
pkg: Repository OPNsense cannot be opened. 'pkg update' required

I've searched for this error and have tried as many solutions as possible (manually triggering updates from the command line, changing mirror, etc.) with no luck.

Is there something obvious I might be missing?

I have an Nginx reverse proxy configured to provide access to several services that I need outside my lan and use IP:port references for internal services (accessed through Wireguard if I'm away from my network).

Is there any easy way to assign domain names with valid HTTPS certs for the services that I'd like to remain internal only?

I'm new to OPNsense and Unbound, so I'm a little lost as to where to even start.

Most of my services are installed via Docker on two different servers, so it would be preferable to be able to point OPNsense/Unbound to an Nginx/Caddy reverse proxy installed on one of the two servers (depending on which subdomain is being requested) to prevent the need of having to expose ports on my network.

Open to any other thoughts, though!