Hello all,
I have two servers that host websites, one being Prod and the other being Test. I would like to be able to use one public IP. Each server has their own Apache and Mginx(reverse proxy) install. Right now it looks like everything gets pushed to the prod server, since that was the first server done. How can I get both servers to peacefully coexist thru one IP?
Thanks,
Steve
With a reverse proxy on OPNsense. There are at least three plugins that can do that: HAproxy, NginX and Caddy. Pick what you are most comfortable with.
You must access the webservers via two different FQDNs for that to work.
All of my websites have their own FQDNs, so thats not the issue.
As mentioned each web server has its own Nginx implementation, so I was hoping I could just reconfigure for the test server. My initial thought was to use a https://FQDN:different port and setup OPNsense firewall/NAT rules to take that and translate it to my test web server. The problem I have is one public IP and Let's Encrypt does not like multiple internal IPs thru one public IP.
So now I am thinking about assigning a separate public IP for the test web server. I have 2 extra public IPs and can use one to do this but not sure how OPNsense would be configured. Would I just add another WAN port to this public IP and then get my firewall/NAT rules in place for this?
As you mentioned I could move the reverse proxy capabilities up to the firewall but this is now production and I cannot monkey around with this now. As soon as I enable Nginx I need to be ready for all.
If you want
- single IP (v4) address
- single (standard) port
- letsencrypt
There is no way around moving the reverse proxy and SSL termination to that single IP address and port. Which product to use for that remains your choice.
Which product would you recommend based on my needs? I have Nginx already implemented so I am thinking this might be the path of least resistence?
If you use NginX you need to integrate the acme-client somehow. The os-caddy plugin is essentially fire and forget and will do letsencrypt automatically. Just enable TLS, it will take care of everything.
Does the os-caddy support TLS 1.3 and HTTP3?
TLS 1.3 - of course.
HTTP/3 - also yes for Caddy itself, although I do not know the state of the plugin in that regard.
Should some option be missing from the plugin, @monviech has been really accepting to feature and merge requests.
https://caddyserver.com/features
Caddy supports HTTP3 reverse proxying in the upcoming version 2.8.4 that will be in the plugin version 1.5.7.
https://github.com/caddyserver/caddy/commit/5f6758dab5fc02f74233a92c53ba3b654e476dc0
But the feature is marked experimental, so I didnt give it a GUI option yet.
// EXPERIMENTAL: "3" enables HTTP/3, but it must be the only
// version specified if enabled. Additionally, HTTPS must be
// enabled to the upstream as HTTP/3 requires TLS. Subject
// to change or removal while experimental.
Quote from: Monviech on June 12, 2024, 08:58:52 PM
Caddy supports HTTP3 reverse proxying in the upcoming version 2.8.4 that will be in the plugin version 1.5.7.
https://github.com/caddyserver/caddy/commit/5f6758dab5fc02f74233a92c53ba3b654e476dc0
But the feature is marked experimental, so I didnt give it a GUI option yet.
// EXPERIMENTAL: "3" enables HTTP/3, but it must be the only
// version specified if enabled. Additionally, HTTPS must be
// enabled to the upstream as HTTP/3 requires TLS. Subject
// to change or removal while experimental.
It definitely looks like a simpler implementation than HAProxy. Do you support both DNS-01 and HTTP-01? Is there a guide I should be looking to use, as I get this setup?
https://docs.opnsense.org/manual/how-tos/caddy.html
Thank you...it does not look too onerous!
One last question...I would assume that once I deploy Caddy I must turn off Nginx on my web servers.
Quote from: spetrillo on June 13, 2024, 08:48:56 PM
One last question...I would assume that once I deploy Caddy I must turn off Nginx on my web servers.
No. Caddy will receive and answer all requests from any client on the Internet to your public IP address, then relay these to your internal web hosts based on the respective FQDN. You can leave your internal hosts just as they are.
If the NginX on your web servers implements TLS you need to decide if you want OPNsense/Caddy to trust these certificates or simply enable the "don't care about the cert" button in the Caddy config.
Or have your NginX servers use plain HTTP only and Caddy to relay to that. This is what I do for most of my applications. I do not consider unencrypted traffic across a private switched network a security risk.
Ahh so I could consolidate SSL and certs with Caddy, and from Caddy to Nginx/Apache would be unenceypted port 80 access?
Does Caddy have a feature to allow internal access to the websites, sort of like split brain?
Why split brain? You access your external IP address from inside. If you have the default "allow all" rule on LAN, things are just working.
I thought split brain was for cert reasons...guess i was wrong.
The single interesting point with respect to certificates is the connection of any browser to your front end proxy or web server. If all requests, internal and external alike, connect to the Caddy server on your WAN IP, then there is only a single certificate to worry about and Caddy will already have taken care of that.
"Split" access is more of a DNS issue. If for some reason you prefer to have external access routed through Caddy but internal access directly to your services (by means of split DNS) then of course you also face the problem of generating and deploying valid certificates for both ways of access.
So the general recommendation is to just map the FQDN to your external IP address and use that everywhere. Caddy will take care of TLS and the certificate.
Quote from: Monviech on June 12, 2024, 09:22:07 PM
https://docs.opnsense.org/manual/how-tos/caddy.html
Monviech,
I noticed the Caddy website shows some functionality in orange, that are optional plugins. If I need SSH and PHP is that in the OPNsense plugin or do I need additional plugins?
Thanks,
Steve
I dont understand the question.
Sorry if I was not clear. If you look at the Caddy website you will see a statement that features in orange are provided by optional plugins. Further down there is a list of app modules and I would be looking for SSH and PHP. Since I am using the OPNsense plugin are these features in the plugin or do I need to add additional plugins to get SSH and PHP app functionality?
The Caddy plugin in OPNsense supports only HTTP and HTTPS.
What is your use case for SSH and PHP support on a firewall based proxy?
PHP bc we are full PHP environment and will end up putting out edge apps that will need to talk back. It might work over HTTPS though.
The php and ssh module are for when caddy is used as ssh server and php server.
For a reverse proxy, only http/https are needed. You dont want to host actual applications on the opnsense itself. You only want to reverse proxy to them.
Hi Monviech,
So first off this has been an incredibly simple process. I am up and rolling in about an hour. I do have a couple of questions.
I have two Plesk servers, that serve up multiple websites. I noticed your section on Reverse Proxy to a Webserver with Vhosts and the internal/external hostnames. I have setup serveral DNS overrides in Unbound, so the external URL name resolves to the internal IP address. I did this bc our code has some external URL calls in it and we found the app would timeout without knowing where it was. Should I remove this and implement your Vhost section?
Second we use Cloudflare as our DNS provider. I noticed you have a section for DNS Providers and the DNS API standard field. Do you want my global API key here, or can I put my DNS Edit key there?
Lastly when I try to hit my website I am getting a blank screen back. Its like it does not know where to go. I wonder if this is back to the vhost piece.
Thanks,
Steve
Ok I think I figured out the blank screen and that is around the private IPs but now when I try to hit a website from the outside I am being told the site redirected me too many times. Whats that all about?
Also I am seeing this in my Caddy log every 10 minutes...do I need to add something?
Informational caddy "info","ts":"2024-06-15T16:09:16Z","logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
I'm sorry I don't have much experience with this kind of setup. You just have to try things out now until you figure things out. If not, go to the https://caddy.community . Make sure you fill out their full help template otherwise they can't help.
Also, please read all help texts in the plugin, they help.
Some hinty:
- Cloudflare API key should be only a scoped one for DNS.
- For plesk servers, there could be client side redirects interfering.
- plesk servers use nginx as another reverse proxy to apache. Could be a challenge to have two reverse proxies if not everything is configured just right.