Connection but not Stable, Proxmox, OPNsense

[mikelinsi]

Hi everyone, 👋 Mike here from Zürich

I've got a setup at Hetzner with OPNsense running behind Proxmox, and I'm running into a pretty stubborn issue with internet connectivity in my VMs — especially when trying to run docker compose pull. Maybe someone here has seen something similar or has ideas on where I should dig deeper?

My setup:
Hosting provider: Hetzner (Root Server)
Virtualization: Proxmox VE 8.4.1

Public IPs:
Proxmox host: x.x.x.49/26
OPNsense WAN: x.x.x.47/26 (bridged via vmbr1)

Bridge setup:
vmbr0: for Proxmox (direct to internet)
vmbr1: internal bridge for OPNsense WAN and LAN

Firewall: OPNsense 24.7
WAN: x.47
LAN: 192.168.1.2, DHCP enabled (192.168.1.0/24)

Example client VM:
immich at 192.168.1.102

The issue:
VMs behind OPNsense get proper IPs via DHCP and generally can access the internet.
Funny enough, the OPNsense box itself can ping IPs (like 1.1.1.1) but can't resolve domains (e.g. google.com). That's less critical to me — my focus is on the VMs.
However, docker compose pull often fails, with errors like:
request canceled while waiting for connection (Client.Timeout exceeded)
context deadline exceeded
dial tcp [IPv6 address]: connect: network is unreachable
Error Get "https://ghcr.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Error Get "https://registry-1.docker.io/v2/": context deadline exceeded
Error Get "https://registry-1.docker.io/v2/": dial tcp [2600:...]:443: connect: network is unreachable
Get "https://ghcr.io/v2/": dial tcp 140.8...
Error response from daemon: Get "https://ghcr.io/v2/": dial tcp 140.82.121.33:443: i/o timeout (Client.Timeout exceeded while awaiting headers)

My questions for you:
What could be causing this instability?
How would you go about debugging this — what would you check?
I really appreciate any ideas, insights, or experiences you can share. Thanks a lot in advance for your help!

[meyergru]

You know that hetzner expects different MACs for each IP? So you have changed the MAC for one interface/bridge?

Also, although they tell you that you should use a netmask (/26), all IPs are isolated. Thus, it is better to use /32 for any IP and specify a pointopoint gateway. This is especially of concern for you, because .47 and .49 are in the same subnet, but probably cannot talk, unless they both use the gateway.

[EricPerl]

A network diagram wouldn't hurt. I don't understand the topology (OPN "behind" Proxmox, single bridge for WAN & LAN).
OPNsense is a VM too?

[mikelinsi]

Cool, thanks EricPerl and meyergru for answering so fast.

yeah, this setup was made over a year a go, I lost the most stuff in my head. because most of the time it works. just no upgrade on the open-sense and the strange connection on the clients with the docker stuff.
i think i followed something like this:
https://www.youtube.com/watch?v=-XpUhoiR_HA


here is my diagram
https://imgur.com/SAMlvGm


I have teste on Friday with a stock ubuntu server VM installation Instance of the OpenSense VM, that worked well. so I did get Internet on that machine, and with curl on ifconfig.me I got the second ip back which I have rented from hetzner.
But if I do this with the OPNsense box, I see that the traffic goes out, but I never get an answer for the ping, drill, dig, curl

I will try to next reinstall the OPNsense and see if this fixes. I have no idea where I need to look on OPNsense. Could be I destroyed my config with some NAT because some clients should be public, which works. like the nginx is on the second ip accessible


Mike

[meyergru]

IDK how your topology is supposed to work when both networks overlap. As I already said: x.x.x.47/26 and x.x.x.49/26 are on the same subnet, so how can you distribute that over two bridges connected to the same host?

[EricPerl]

The YT video is clearly using vmbr0 (vtnet0) for OPN WAN and vmbr1 (vtnet1) for OPN LAN.
vmbr0 is also used by proxmox, linked to the only NIC on the machine.
vmbr1 is an internal bridge, only accessible by VMs.

That makes sense.
The OP's config does not match and does not make sense. You can't have WAN and LAN of OPN on the same NIC without VLANs.

[meyergru]

Correct. I presented here how this is done correctly.

[EricPerl]

Yep. I'm aware. Although when I first read it, I was pretty green with OPN (and greener with virtualization) and missed the part about vmbr1 being internal only...
I later set up a 1 NIC virtualized OPN at home with physical machines on its LAN and still didn't understand how this datacenter use case worked.
Then I realized my mistake...

[mikelinsi]

update;

I think I got it working now... I will update here if I found out what was the solution.

what I did, I check the new installation and have seen that the second ip can be set with DHCP from hetzer without any config, and this works as a WAN. I needed to change the port sorting on the WM, so that one is WAN and the second is LAN, then auto discovory/mapping worked.

will update.