Docker in proxmox & outside access

[Andy259]

Community,

New user here well newish, so a bit of background:

For the last couple of years I've been running Plesk(webserver) on a homebased server, I then discovered Proxmox and Docker (I can run docker in Plesk, but I would much rather have a separate server, so I installed Proxmox on another machine, then came the inevitable port clashes etc, so I got a second IP that was a month ago, I got another machine and installed Opnsense, so here I am trying to figure out HOW to get to the docker containers from OUTSIDE my network i.e point services i.e the ARRS to a domain name.

Here's my setup:

ROUTER OPNSENSE (PUBLIC IP's 194.###.###.37 & 194.###.###.38) The first IP Is the first WAN IP and the one that is setup
during opnsense wizard, the second has been added as virtual IP
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

SERVER 1 PLESK - 194.###.###.37 >> 192.168.1.109 (via nat 1:1)

ALL WORKING WELL AND AS IT SHOULD.(just used this as an example as I'm sure 1:1 is setup correctly and this confirms)

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This is the confusing bit i'll be as clear as possible.

SERVER 2 PROXMOX - 194.###.###.38 >> 192.168.1.112

WORKS CORRECTLY WITH 1:1 NAT AS LONG AS THE BELOW 1:1 NAT DOESN'T EXIST

PROXMOX CONTAINER WITH DOCKER 194.###.###.38 >> 192.168.1.115

So in essence if I delete the Proxmox rules then Portainer is accesible through 194.###.###.38:9000 and also via a url.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Some additional info, I followed a couple of tutorials which I linked below

The first is to install docker, portainer and nginx proxy manager.

https://wiki.opensourceisawesome.co...er-docker-compose-and-more-in-under-5-minutes

This second one basically explains nginx proxy manager (however, this video is on the assumption you are either on a bare metal server with docker installed directly on Ubuntu OR you are using a vps from digital ocean and the like (so I followed the directions for this as it matched my use case more closely(ish) so using this method I CAN get to those services on my public IP (portainer, nginx proxy manager) but ONLY if I disable the portforward for proxmox which is then not accessible via the public ip, I'm missing something simple and im sure i saw a vid on exactly how t do this but can't find it there's plenty on installing docker in a proxmox container or vm but then they dump you and don't tell you how to reach the services from the outside world.

Video about nginx etc which i followed and got me to the point of accessing the docker containers.

https://www.youtube.com/watch?v=cjJVmAI1Do4

Please bear in my mind I am new to proxmox, opnsense and docker so please be fairly specific in replies and if you can tag tutorials that would be great.

[Vintage0765]

Take a look at gluetun it might be what you're looking for.

https://github.com/qdm12/gluetun

[meyergru]

I wonder why you use 1:1 NAT on your OpnSense, because that way, it does not control much. Also, you are obviously limited to just one RFC1918 IPv4 behind it.

If you use normal outbound NAT for a /24 RFC1918 subnet, you can make a management LAN available where both your docker VM and your proxmox server can reside. You can makes them available from outside via port-forwarding or name-based reverse-proxying on your OpnSense (with NGinx, Caddy or HAProxy). The latter is convenient also because you can delegate all TLS termination to your firewall.

You also have the choice of using one or more VLANs to separate the VMs from proxmox and one another. Assuming that proxmox is on your LAN and inaccessible from WAN, you may want that kind of separation in case your VMs get hijacked.

[Andy259]

Hi meyergru,

Thank you for your response, I am using 1:1 Nat because I'm still a bit poo-poo when it comes to networking, Not sure what ports are needed for Proxmox and Docker alike so to me all ports open is the best solution, I've seen many people say don't have ports open, well if you don't have ports open how does the outside world reach your containers, for example what if I have Plesk in a VM, Plesk requires many ports to be open to function correctly. 81, 81, 53, 22 etc and i have tested this by turning said ports on individually on my previous setup e.g if you don't have 21 open FTP works for no-one (none f the domains on the server) is it the same with Proxmox? lets say I want to install a 2 webservers in separate containers how would they even get routed if I don't have the required ports open?

Vlan seperation sounds like what I may need can you point me in the direction of tutorials or what should I google? I like the idea it sounds like easier managment, I'm enjoying the learning tbh and enjoy watching the video's.

So are you saying for my Proxmox install I only need to open port 8006, I already have nginx proxy manger installed in a docker container which is working(ish, can only forward to whatever is inside docker on the same docker network).

My Ultimate aim is to have a hassle free possible way of having docker containers, Proxmox containers and VM's accesible via any port without the hassle of port forwarding EACH time a service requires a new port for example what if i wanted to have a mail server in a VM that would require port 465 and others to be open but HOW do i get the SAME public ip to point to a container that may have the same port requirements as they would just clash?? BUT I do have a 3rd IP so my plan was Public IP 1 >>>>> Plesk (up and running, no issues) Public IP 2 >>>> Proxmox Box (Some Issues, Due to 3rd IP not registering more about that below) Public IP 3 >>>> Docker (Some issues more below)

OK about the above mentioned issues as I said I have a 3rd IP which I want to use as mapped above BUT I can't register that in OPNsense (that's another issue and ill open a new thread as it might help others)so back to the issues that are not really issues so we go back to forwarding so at present I have my public IP no 2 pointing to the DOCKER VM with nginx setup as a reverse proxy which IS working BUT I obvioulsy can't get to my Proxmox when i have it this way does the port forwarding just need to be for the proxmox box?

[meyergru]

If your aim is to have many services available without using opening another port (or IP), then obviously a reverse proxy which does that by name is the way to go.

If those services are to be made available from outside your LAN, you would probably use a reverse proxy like Caddy, HAproxy or NGinx (for the first two, there are HowTos available in the tutorial section of the forum).

If these are only different docker services that you want accessible from your LAN, you can use Traefik.

The reverse proxies take care of TLS termination and map DNS names to internal IPs and Ports. So, if that proxy is on OpnSense, it can access anything behind it without the need for many public IPs - one is fully sufficient.

Because all containers on a docker host are on the same hardware and internal docker networks anyway, there is no specific need to separate them network-wise via VLANs - that would only come into play when you have multiple VMs or physical machines.

I use HAproxy with one public IP on OpnSense, this distributes DNS names to several internal VMs, which are categorized by how/if they are to be accessed from outside my LAN. One is a docker VM for internal services, one a docker VM for external services - the former is in my LAN, the latter in a DMZ VLAN. Same goes for full VMs, like Home Assistant and Plex, each one is located in one VLAN or the other depending on if I deem it safe to have it in my LAN. I could separate each "unsafe" VM into a different DMZ VLAN, but I don't.

On the docker machines, I use just the normal networking where I expose application-specific ports. Then, I only need a DNS name and a mapping to the specific docker IP and service port in the reverse proxy, which also handles TLS termination. All services can be made available as subdomains of one official domain to issue ACME certificates, preferably through a wildcard certificate (this is covered in the reverse proxy tutorials).

The only thing left is how to block specific services from outside (i.e. limit them to LAN access only). For HAproxy, this can be done by a backend rule that only allows RFC1918 IPs.