I finally managed to get the first WAN IPv4 Address working on (Virtualized) OPNSense on top of Proxmox VE on Hetzner Dedicated Server :D.
Now, I ordered an additional Single IPv4 Address from Hetzner Robot, and apparently this is where the issue start.
To summarize I have the following IPs:
- (NOT OPNSense) xxx.xxx.xxx.proxmox (Main IPv4 Address, cannot get a dedicated MAC Address)
- vtnet0: xxx.xxx.xxx.opnsense_01 (First Additional IPv4 Address, with a dedicated MAC Address)
- vtnet2: xxx.xxx.xxx.opnsense_02 (Second Additional IPv4 Address, with a dedicated DIFFERENT MAC Address OR without MAC Address at all)
I ended up in the same /26 Subnet as the "Main" Additional IP. The Gateway has therefore the same IP Address.
As a first Approach, I tried to simply add a new Network Interface to the OPNSense VM mapped to the new MAC Address, then configure the Interface in OPNSense using DHCPv4. Weird stuff happened with regards to the ARP Table, with the IP getting associated to both vtnet0 (Main WAN for OPNSense, configured via DHCPv4 ALWAYS) and vtnet2 (this additional IP Address with DIFFERENT MAC Address).
That didn't work, most likely because Multi-WAN is NOT Supported on Multiple Interfaces with the same Gateway ? Some posts on OPNSense Forum as well as PFSense Forum/Tutorials seem to indicate that this is NOT possible with PF on FreeBSD. Also https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html#multiple-wans-sharing-a-single-gateway-ip seem to suffest that it's only doable with NAT.
I also tried to add Static IPv4 for the Interface, then creating a separate Gateway (or trying with "Dynamic gateway policy - This Interface does not required an Intermediary System to act as a Gateway"), but it will NOT work :(.
I tried, as an alternative, to add simply the new IPs as a Virtual IP / IP Alias. The vtnet0 (wan) interface gets both IPs, but nothing happens :(.
My guess is that Hetzner checks that the SRC MAC and SRC IP Address match (MAC/IP Address Filtering). So I will get yet another MAC Address Abuse Email soon most likely :(.
To recap, this is done via Proxmox VE vmbr0 Linux Bridge, therefore (in the first approach) the vtnet0 (main OPNSense WAN Interface) and vtnet2 (secondary OPNSense WAN Interface) are effectively in the same L2 Subnet (that's why you can see them all in the ARP Table).
But I think that unless the SRC MAC and SRC IP "match" what Hetzner is expecting, traffic from the Server just gets dropped.
I couldn't ping TO the new WAN IP from my Home Network, neither could I ping FROM the new WAN IP in Interfaces -> Diagnostic -> Ping.
Those tests also failed when I removed the dedicated MAC Address in Hetzner Robot (so the traffic for that IP should be bound to the Proxmox VE Host now).
What would be the best Solution to solve this, assuming I cannot get Hetzner to "point" to the same MAC Address as the Main OPNSense WAN IP ?
From what I can see, either:
- Proxmox VE Host: Configure 2nd OPNSense IP as "Routed" -> NAT to another Linux Bridge say 192.168.100.1 -> OPNSense Configures an additional Interface there (double NAT most likely needed)
- OPNSense VM: Configure 2nd IP as "Routed" to an Intermediary Network -> (double NAT most likely needed)
Or is there some "weird" way to add a "Virtual" Interface ? Or possible a WAN / Transparent Filter Bridge or even a "LAN Bridge" ?
Can you use the second IP address with the same MAC address as the first one as far as Hetzner is concerned?
If yes, then simply add it under Interfaces > Virtual IPs to your existing WAN.
You cannot have addresses from the same subnet on two different interfaces.
Quote from: Patrick M. Hausen on May 22, 2024, 12:24:31 PM
Can you use the second IP address with the same MAC address as the first one as far as Hetzner is concerned?
If yes, then simply add it under Interfaces > Virtual IPs to your existing WAN.
You cannot have addresses from the same subnet on two different interfaces.
I just asked if it's something they can do. By the automatic Robot Service, this does NOT appear to be possible.
Either I can click on "Request a separate [DIFFERENT] MAC Address", or "Reset MAC" (which "deletes" the separate [different] MAC Address and re-binds that IP to the Main IP of the Server).
So I would need to do some Routing ?
Or possibly a 2nd OPNSense VM :D ? The advantage with a 2nd OPNSense VM would be, if nothing else, that it could avoid Double NAT.
(on a separate note: for IPv6 I can configure the /64 Subnet to POINT to one of these already-existing MAC Addresses, didn't manage to get it working though. For IPv4 this is however NOT possible to do)
Yes, so you use the main IP of your server for OPNsense and all alias IP addresses assigned to that with the same MAC and you use one extra IP with a different MAC for Proxmox ...
Quote from: Patrick M. Hausen on May 22, 2024, 01:59:15 PM
Yes, so you use the main IP of your server for OPNsense and all alias IP addresses assigned to that with the same MAC and you use one extra IP with a different MAC for Proxmox ...
Just got the reply from Hetzner. Unfortunately it's NOT possible (see attached).
So either I'll have to enable Routing on the Proxmox VE Host and Forward from there, or spin up another Router/Firewall VM (be it OPNSense or maybe OpenWRT etc) and do the routing there using a separate MAC Address :(.
That's what I wrote: if no virtual MAC is set it uses the same as the main IP.
So use main IP for OPNsense and an additional with virtual MAC for Proxmox ...
HTH,
Patrick
Quote from: Patrick M. Hausen on May 22, 2024, 02:12:15 PM
That's what I wrote: if no virtual MAC is set it uses the same as the main IP.
So use main IP for OPNsense and an additional with virtual MAC for Proxmox ...
HTH,
Patrick
You don't understand ... This is what I am doing for my Main Server IP (Proxmox Host) AND My FIRST Additional IP for OPNSense VM ;).
The Issue right now is how to handle the next (SECOND Additional or THIRD in Total) IP for OPNSense :(.
EDIT 1: Re-reading your Previous Post ...
>> you use one extra IP with a different MAC for Proxmox ..
That would be possible
>> you use the main IP of your server for OPNsense and all alias IP addresses assigned to that with the same MAC
That is NOT possible. The Main IP is tied to the Server physical MAC Address. There is no option to change that. Unless of course you are suggesting spoofing the Server Hardware MAC Address into the OPNSense VM, that is NOT going to work ...
Quote from: luckylinux on May 22, 2024, 02:14:41 PM
Unless of course you are suggesting spoofing the Server Hardware MAC Address into the OPNSense VM, that is NOT going to work ...
I am suggesting exactly that. Spoofing the MAC of the Proxmox host to the additional virtual MAC for the second IP address, then spoofing the MAC address of the OPNsense VM to the original one of the physical server.
Quote from: Patrick M. Hausen on May 22, 2024, 02:53:25 PM
Quote from: luckylinux on May 22, 2024, 02:14:41 PM
Unless of course you are suggesting spoofing the Server Hardware MAC Address into the OPNSense VM, that is NOT going to work ...
I am suggesting exactly that. Spoofing the MAC of the Proxmox host to the additional virtual MAC for the second IP address, then spoofing the MAC address of the OPNsense VM to the original one of the physical server.
Ah ... So basically what I wanted to do initially, but reversing the role of Proxmox and OPNSense VM.
Won't this cause additional issues though, making Hetzner send me yet other MAC Abuse Emails ?
Or for all intents and Purposes, since Hetzner will receive all traffic from the eth0 Port of the Linux Bridge (Main MAC of the Host, which you suggest me to Spoof into OPNSense, making the Additional IP MAC Address take the identity of the Linux Bridge / Proxmox Host instead), all traffic will appear from being originated from eth0 / Main MAC anyways ?
Or since this is simply L2 Networking and not Routing, it doesn't really matter (there is no "Network" Address "Translation", just a switch and 2 MACs sending/receiving stuff) ?
Quote from: luckylinux on May 22, 2024, 02:58:04 PM
Or since this is simply L2 Networking and not Routing, it doesn't really matter (there is no "Network" Address "Translation", just a switch and 2 MACs sending/receiving stuff) ?
I guess so - all of that is just a suggestion. I do run OPNsense at Hetzner, just not virtualised.
EDIT: thinking about my two node OPNsense HA cluster at Hetzner ...
You could IMHO:
- order a vswitch for your proxmox host
- define the vswitch VLAN in proxmox
- pass that as a virtual interface to OPNsense as WAN
- order a /29 or whatever connected to that vswitch and use these addresses in OPNsense
Quote from: Patrick M. Hausen on May 22, 2024, 03:18:58 PM
Quote from: luckylinux on May 22, 2024, 02:58:04 PM
Or since this is simply L2 Networking and not Routing, it doesn't really matter (there is no "Network" Address "Translation", just a switch and 2 MACs sending/receiving stuff) ?
I guess so - all of that is just a suggestion. I do run OPNsense at Hetzner, just not virtualised.
Well ... That would actually be the only way to make this work, otherwise I'd have to do Routing on the Host or Routing with Additional VM.
Alternatively I could ask for an Extra NIC + LAN Connection, totalling approx. 5 EUR/month extra.
Or if I want to try my luck, just pass the NIC directly to OPNSense via PCIe Passthrough (with the **big** Issue that if the OPNSense VM goes down or breaks, I cannot even access the Host).
Quote from: Patrick M. Hausen on May 22, 2024, 03:18:58 PM
Quote from: luckylinux on May 22, 2024, 02:58:04 PM
Or since this is simply L2 Networking and not Routing, it doesn't really matter (there is no "Network" Address "Translation", just a switch and 2 MACs sending/receiving stuff) ?
I guess so - all of that is just a suggestion. I do run OPNsense at Hetzner, just not virtualised.
EDIT: thinking about my two node OPNsense HA cluster at Hetzner ...
You could IMHO:
- order a vswitch for your proxmox host
- define the vswitch VLAN in proxmox
- pass that as a virtual interface to OPNsense as WAN
- order a /29 or whatever connected to that vswitch and use these addresses in OPNsense
I just tried "reversing" / spoofing as you suggested, so everything goes to the OPNSense VM (original MAC that the Host has) except one additional IP that goes to a dedicated MAC which I assigned to the Linux Bridge vmbr0 (Proxmox VE Host).
It **seems** to work :). Although I guess too early to tell still :-\.
@Patrick: Did you have any special Configuration needed for IPv6 to work with Hetzner ?
I couldn't manage to get it working. Neither on Proxmox VE, neither on OPNSense (tried both MAC Addresses to "point" the /64 Subnet to the right Appliance).
Maybe it's just because I'm used to IPv4, where Gateway and Address are in the same Subnet, but their Default Gateway fe80::1 doesn't play Ball at all. Interfaces -> Diagnostics -> NDT returns only the Local/Global (if set Static, otherwise none) IPv6 Addresses. DHCPv6 Configuration of the WAN Interface doesn't get an IP Address.
Strangely enough, I can see in OPNSense Firewall Logs that my Home Address is managing to Ping6 through (inbound), but I do not see any outbound reply (and Ping6 fails).
I had this issue previously with IPv4 as well (Inbound Traffic showing up in the logs, but Outbound not working), which I solved by setting Firewall -> Settings -> Advanced -> Disable reply-to on WAN rules -> CHECK
For IPv6 it's not working though. I tried to set Static IPv6 but nope. I cannot ping the gateway.
Probably the OPNSense VM cannot receive the Route Advertisement from the fe80::1 Gateway ? I don't see anything in the Proxmox VE Firewall Logs though ...
EDIT: I see lots of traffic on the Loopback Interface, not sure if this is normal. I don't really see anything going Out of the Firewall though. Loopback Interface has Source = Destination IPv6 Address, corresponding to the Static WAN IPv6 Address I had set.
Making progress somewhat ...
On Proxmox VE Host I added this to /etc/network/interfaces:
#auto eth0
iface eth0 inet manual
auto vmbr0
iface vmbr0 inet static
hwaddress XX:XX:XX:XX:XX:XX
address 94.XX.XX.XX
netmask 255.255.255.192
gateway 94.XXX.XXX.XXX
#pointopoint 94.XXX.XXX.XXX
bridge-ports eth0
bridge-stp off
bridge_waitport 0
bridge-fd 0
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
pre-up ip addr flush dev eth0
post-up ip addr flush dev eth0
iface vmbr0 inet6 static
hwaddress XX:XX:XX:XX:XX:XX
address 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001
netmask 96
gateway fe80::1
bridge-mcsnoop no
bridge-ports eth0
bridge-stp off
bridge_waitport 0
bridge-fd 0
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
pre-up ip addr flush dev eth0
post-up ip addr flush dev eth0
On OPNSense WAN Interface I set Static IPv6:
address: 2a01:XXXX:XXXX:XXXX:0001:0000:0000:**0010**
netmask: 96
gateway: 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001
Gateway fe80::1 would NOT work in OPNSense. I guess that's because Routing (of Subnet) is required. I read on Hetzner Docs that Bridging is supported for single IPv4 Addresses, but when dealing with an IPv4 Subnet, Routing is required. Well I guess it makes sense, if OPNSense IP is NOT in the same subnet as the Remote (Google DNS Server) Destination, then the Host must perform Routing, right ?
On Proxmox VE Host I enable IPv6 Forwarding. Maybe this was the most critical bit missing ? I could IPv6 ping between Proxmox VE Host and the OPNSense VM without Issues without, but as soon as trying to Ping e.g. Google DNS Server 2001:4860:4860::8888 it would fail.
sysctl -w net.ipv6.conf.default.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.default.forwarding=1
I don't think this brings me closer to a solution however ...
From what you @Patrick told me in the Past, IPv6 Networks are ALWAYS /64. I used /96 here because I was trying to follow the Instructions https://community.hetzner.com/tutorials/distributing-ipv6-over-libvirt-guests which, together with Host IPv6 forwarding Enabled, does Indeed work.
But I guess if I really want to do it "right", I need to order a /56 IPv6 Subnet, in that way I can have a /64 Subnet for WAN on Proxmox VE Host and OPNSense VM, then several other /64 Subnets for DMZ1/DMZ2/LAN1/LAN2/.... on OPNSense.
@Patrick: sorry If I didn't say it explicitely until now, but THANK YOU VERY MUCH for all your Help !
Quote from: luckylinux on May 23, 2024, 09:40:08 AM
Gateway fe80::1 would NOT work in OPNSense. I guess that's because Routing (of Subnet) is required.
It definitely does. An IPv6 interface will
always have a link local address in addition to the global unicast address you assigned. So it can use another link local address on the same interface just fine.
See attached screenshots. This is OPNsense on bare metal, though.
Quote from: Patrick M. Hausen on May 23, 2024, 10:14:00 AM
Quote from: luckylinux on May 23, 2024, 09:40:08 AM
Gateway fe80::1 would NOT work in OPNSense. I guess that's because Routing (of Subnet) is required.
It definitely does. An IPv6 interface will always have a link local address in addition to the global unicast address you assigned. So it can use another link local address on the same interface just fine.
See attached screenshots. This is OPNsense on bare metal, though.
I'm not doubting it works in your case. In mine it doesn't though for some weird reason. Maybe because the Linux Bridge (Switch) causes fe80::1 to become Ambiguous (is it Hetzner's Gateway, the Proxmox VE Gateway, etc) ?
Anyway, it works with the "Direct" Proxmox VE Host as a Gateway.
I also have both Link Local Address and Global Unicast Address in OPNSense:
2a01:XXXX:XXXX:XXXX:1::10/96
fe80::YYYY:YYYY:YYYY:YYYY/64 Weird Stuff ... Could also be because I disabled some multicast/unicast flooding in the Bridge though (I don't want to receive yet other MAC Address Abuse from Hetzner).
On another note, I don't understand why, unless I put a Firewall Rule in place (which another User on Proxmox Forums recommended to me), ALL other "NEIGHBOR" Hetzner Servers in my IPv4 Subnet are dumping traffic on me (like I am seeing & dropping Traffic that is destined to my Neighbors IP Address, and if I don't, Hetzner claims I have some non-authorized SOURCE MAC Addresses coming out of my Ethernet Port).
Why should I get traffic destined to another Server IP in the First Place ? That seems some weird Stuff going on.
I am receiving IPv4 TCP/UDP/ICMP Traffic with "Destination = Neighbor Server IP" (not destined to my Server !), of about all range of ports (for TCP/UDP I mean, NOT only the 32768-65535 Range, also e.g. port 8008 etc). Is this a Multicast Traffic ?
Do you also face this Issue at Hetzner ? Funnily the ARP Table (and NDT for IPv6 for that Matter) only shows "my" MAC Addresses / IPv6 Addresses (plus the IPv4 Gateway MAC Address for ARP) in OPNSense.
EDIT 1: By the Way, testing IPv6 via Interfaces -> Diagnostics -> Trace Route I always get "timeout reached", but from SSH into the Router, the following work Correctly:
# ICMP Traceroute
root@OPNSense:/ # traceroute6 -I 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2a01:XXXX:XXXX:XXXX:1::10, 64 hops max, 20 byte packets
1 2a01:XXXX:XXXX:XXXX:1::1 0.149 ms * 0.211 ms
2 2a01:XXXX::XXXX:XXXX:b 0.365 ms 2.220 ms 0.287 ms
3 2a01:XXXX:0:XXXX::XXXX 0.571 ms 0.474 ms 0.341 ms
4 2a01:XXXX:0:XXXX::XXXX 4.964 ms 4.898 ms 4.898 ms
5 2001:4860:1:1::1a4 5.157 ms 5.038 ms 5.094 ms
6 2a00:1450:8155::1 5.205 ms 5.219 ms 5.211 ms
7 dns.google 5.204 ms 5.189 ms 5.201 ms
# UDP Traceroute
root@OPNSense:/ # traceroute6 -U -p 33434 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2a01:XXXX:XXXX:XXXX:1::10, 64 hops max, 28 byte packets
1 2a01:XXXX:XXXX:XXXX:1::1 0.135 ms * *
2 2a01:XXXX::XXXX:XXXX:b 0.457 ms 0.326 ms 0.304 ms
3 coreXXX.XXXX.hetzner.com 11.941 ms 0.622 ms
2a01:XXXX:XXXX:XXXX::XXXX 0.497 ms
4 coreXXX.XXXX.hetzner.com 4.958 ms
2a01:XXXX:XXXX:XXXX::XXXX 5.180 ms
2a01:XXXX:XXXX:XXXX::XXXX 5.102 ms
5 2001:4860:1:1::19c8 4.948 ms
2001:4860:1:1::624 6.293 ms
2001:4860:1:1::19c8 5.235 ms
6 2a00:1450:8037::1 5.239 ms
2a00:1450:8057::1 6.570 ms
2a00:1450:8463::1 5.356 ms
7 dns.google 5.270 ms 5.112 ms 5.248 ms
# TCP Traceroute gets Lost sometimes once it gets out of Hetzner Datacenter I guess
root@OPNSense:/ # traceroute6 -T 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2a01:XXXX:XXXX:XXXX:1::10, 64 hops max, 20 byte packets
1 2a01:XXXX:XXXX:XXXX:1::1 0.158 ms * 0.250 ms
2 2a01:XXXX::XXXX:XXXX:b 0.364 ms 2.267 ms 0.339 ms
3 2a01:XXXX:XXXX:XXXX::XXXX 0.577 ms 0.495 ms
coreXX.XXX.hetzner.com 0.453 ms
4 coreXX.XXX.hetzner.com 5.096 ms 4.982 ms 5.170 ms
5 *
2001:4860:1:1::19c8 5.257 ms *
6 * * *
7 * * *
From the Host it seems to behave better with regards to TCP Traceroute:
root@Proxmox:~# traceroute6 -T -p 443 2001:4860:4860::8888
traceroute to 2001:4860:4860::8888 (2001:4860:4860::8888), 30 hops max, 68 byte packets
1 2a01:XXXX::XXXX:XXXX:b (2a01:XXXX::XXXX:XXXX:b) 0.401 ms 0.398 ms *
2 * * coreXX.XXX.hetzner.com (2a01:XXXX:XXXX:XXXX::XXXX) 0.395 ms
3 coreXX.XXX.hetzner.com (2a01:XXXX:XXXX:XXXX::XXXX) 4.840 ms * *
4 * * *
5 * * 2a00:1450:814d::1 (2a00:1450:814d::1) 4.912 ms
6 dns.google (2001:4860:4860::8888) 5.331 ms * *
Although I must say that when I'm doing TCP Traceroute from Proxmox VE, I can see in the Firewall Logs that it's "consistent". It's always targeting Port 443 as Instructed -> DPT=443
Whereas when I do TCP Traceroute from OPNSense, I can see in the Proxmox VE Host Firewall Logs that the Destination Port is changing A LOT during the Traceroute: DPT=444, DPT=445, DPT=446, ... , DPT=465, ...
So I don't know ...
Is the Traceroute Implementation *that* different from GNU/Linux to FreeBSD ?
When you spoof MAC addresses the interface is set to promiscuous mode which means you will see all traffic on that particular network destined at broadcast or multicast addresses or ones where the switch has not yet learned the correct port.
Why are you trying to force a virtualised firewall setup into an environment that is clearly not designed for that?
Quote from: Patrick M. Hausen on May 23, 2024, 11:01:53 AM
When you spoof MAC addresses the interface is set to promiscuous mode which means you will see all traffic on that particular network destined at broadcast or multicast addresses or ones where the switch has not yet learned the correct port.
Why are you trying to force a virtualised firewall setup into an environment that is clearly not designed for that?
I'm actually NOT spoofing (from inside OPNSense VM) the MAC Address and in OPNSense VM the Interface is NOT set to Promiscuous Mode.
The MAC Spoofing was done one level higher up (in Proxmox VE Network Configuration).
And according to Proxmox Forums Posts, this Part of /etc/network/interfaces
should disable Promiscous Mode in GNU/Linux:
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
>> Why are you trying to force a virtualised firewall setup into an environment that is clearly not designed for that?
Why do you say that ("environment that is clearly not designed for that") ?
What would be the "Proper" way to handle it otherwise ? Your vswitch Idea seems a bit complicated/overkill/somewhat expensive from what I need.
Would it be better to order an additional NIC + LAN Connection, do PCIe passthrough to OPNSense of one of the NICs (depending on which IP can get assigned the MAC Address, as discussed before), so basically keep a Dedicated NIC for Proxmox VE, and one Dedicated NIC (via PCIe Passthrough) for OPNSense VM ?
That at least could get around the Double-Firewalling Issue I guess ...
I have dedicated OPNsense hosts at Hetzner and private LAN connections to the equally dedicated web server hosts. But that might be overkill in your case.
When you order an additional private network interface you cannot connect that to the Internet as far as I know. Only to other equally private infrastructure.
I see another "proper" way out of your dilemma - routed setup. We do this for all our web servers at Hetzner. I assume (I don't use Proxmox) that Proxmox can create an internal bridge not connected to any physical interface that can serve as the uplink for VMs? Correct?
I also assume that Proxmox acts as a router for both IPv4 and IPv6.
So on the external Proxmox interface:
- configure the single IPv4 address you ordered with your server
- configure the IPv4 default gateway
- configure one IPv6 address from that /64 you got with your server, but with a /128 prefix length.
E.g. dead:beef:dead:beef::1/128
- configure the IPv6 default gateway: fe80::1
You should be able to ping the Proxmox host via IPv4 and IPv6 from outside.
Now you would need to order at least one additional IPv4 /29 from Hetzner to have IPv4 for your VMs.
Let's assume it's 8.16.32.64/29.
- assign the first (or last, or any ...) address to that internal bridge, e.g. 8.16.32.65/29
- you can now use 8.16.32.66-70 as IP addresses for VMs connected to that bridge. 8.16.32.65 would be their default gateway
- now assign another address from your /64 to that same bridge, but with the correct /64 prefix length, e.g. dead:beef:dead:beef::2/64. This is the default gateway for your VMs for IPv6.
The VMs connected to that bridge should be reachable on their public IPv4 and IPv6 addresses that way.
We have >20 hosts setup exactly in that fashion, only that our "VMs" are in fact FreeBSD jails and the hosts are not Proxmox but plain FreeBSD.
HTH,
Patrick
This is what we run with OPNsense at Hetzner:
(proServer is our own managed hosting product)
Private LAN Virtual Switch
┏━━━━┓ ┏━━━━━━━━━━━━━━━━┓ ┌────┐
┃ ┃ ┃ OPNsense ┃ │ │
┃ ┃ 10G ┃ (EX 44) ┃ 1G │ │
┃ ┣━━━━━━━━━━━━━┃ ┃─────────────┤ │
┃ ┃ ┃ HAproxy ┃ │ │
┃ ┃ ┗━━━━━━━━━━━━━━━━┛ │ │
┃ ┃ ┃ │ │
┃ ┃ ┃ │ │ Uplink Public Services
┃ ┃ HA-Sync ┃ 1G │ │━━━━━━━━━━━━━━━━━━▶ ◀════════════════════
┃ ┃ ┃ │ │ Public IP
┃ ┃ ┃ │ │
┃ ┃ ┏━━━━━━━━━━━━━━━━┓ │ │
┃ ┃ ┃ OPNsense ┃ │ │
┃ ┃ 10G ┃ (EX 44) ┃ 1G │ │
┃ ┣━━━━━━━━━━━━━┃ ┃─────────────┤ │
┃ ┃ ┃ HAproxy ┃ │ │
┃ ┃ ┗━━━━━━━━━━━━━━━━┛ └────┘
┃ ┃
┃ ┃
┃ ┃
┃ ┃
┃ ┃
┃ ┃ ┏━━━━━━━━━━━━━━━━┓
┃ ┃ 10G ┃ proServer Host ┃
┃ ┣━━━━━━━━━━━━━┫ (AX 102) ┃
┃ ┃ ┃ ┃
┃ ┃ ┃ ┌────┐ ┃
┃ ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃
┃ ┃ ┃ └────┘ Mgmt ┃
┃ ┃ ┃ ┌────┐ ┃ Uplink
┃ ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━▶
┃ ┃ ┃ └────┘ Mgmt ┃ Management only
┃ ┃ ┃ ┌────┐ ┃
┃ ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃
┃ ┃ ┃ └────┘ Mgmt ┃
┃ ┃ ┃ ... ┃
┃ ┃ ┃ ┃
┃ ┃ ┗━━━━━━━━━━━━━━━━┛
┃ ┃ ┏━━━━━━━━━━━━━━━━┓
┃ ┃ 10G ┃ proServer Host ┃
┃ ┣━━━━━━━━━━━━━┫ (AX 102) ┃
┃ ┃ ┃ ┃
┃ ┃ ┃ ┌────┐ ┃
┃ ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃
┃ ┃ ┃ └────┘ Mgmt ┃
┃ ┃ ┃ ┌────┐ ┃ Uplink
┃ ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━▶
┃ ┃ ┃ └────┘ Mgmt ┃ Management only
┃ ┃ ┃ ┌────┐ ┃
┃ ┃━ ━ ━ ━ ━ ━ ━┃━│Jail│───────▶ ┃
┃ ┃ ┃ └────┘ Mgmt ┃
┃ ┃ ┃ ... ┃
┃ ┃ ┃ ┃
┗━━━━┛ ┗━━━━━━━━━━━━━━━━┛
...
Quote from: luckylinux on May 23, 2024, 11:24:01 AM
Why do you say that ("environment that is clearly not designed for that") ?
Because Hetzner routes based on MAC address. For example they "throw" your entire /64 at the MAC address of your server. Similar for the first IPv4 and all subsequent additional subnets you might order. They don't allow multiple external links outside of simple dedicated MAC addresses for VMs (as you found out) or completely custom built solutions (different department, definitely expensive).
But then that configuration (just throw all destination IPs at that particular MAC) is why the setup with the single /128 externally and an internal bridge with the correct /64 works as supposed to as long as the host (Proxmox in your case) performs the routing.
Quote from: Patrick M. Hausen on May 23, 2024, 11:38:20 AM
I have dedicated OPNsense hosts at Hetzner and private LAN connections to the equally dedicated web server hosts. But that might be overkill in your case.
When you order an additional private network interface you cannot connect that to the Internet as far as I know. Only to other equally private infrastructure.
I see another "proper" way out of your dilemma - routed setup. We do this for all our web servers at Hetzner. I assume (I don't use Proxmox) that Proxmox can create an internal bridge not connected to any physical interface that can serve as the uplink for VMs? Correct?
I also assume that Proxmox acts as a router for both IPv4 and IPv6.
So on the external Proxmox interface:
- configure the single IPv4 address you ordered with your server
- configure the IPv4 default gateway
- configure one IPv6 address from that /64 you got with your server, but with a /128 prefix length.
E.g. dead:beef:dead:beef::1/128
- configure the IPv6 default gateway: fe80::1
You should be able to ping the Proxmox host via IPv4 and IPv6 from outside.
Now you would need to order at least one additional IPv4 /29 from Hetzner to have IPv4 for your VMs.
Let's assume it's 8.16.32.64/29.
- assign the first (or last, or any ...) address to that internal bridge, e.g. 8.16.32.65/29
- you can now use 8.16.32.66-70 as IP addresses for VMs connected to that bridge. 8.16.32.65 would be their default gateway
- now assign another address from your /64 to that same bridge, but with the correct /64 prefix length, e.g. dead:beef:dead:beef::2/64. This is the default gateway for your VMs for IPv6.
The VMs connected to that bridge should be reachable on their public IPv4 and IPv6 addresses that way.
We have >20 hosts setup exactly in that fashion, only that our "VMs" are in fact FreeBSD jails and the hosts are not Proxmox but plain FreeBSD.
HTH,
Patrick
Uhm ... I really wanted to avoid /29 Subnet if possible (quite expensive Setup Fee and I would "lose" one IP for the Proxmox VE Host/Bridge).
Too bad they don't allow another WAN Connection (I guess this is what they call "Uplink").
I see they however offer "5-port 1 Gbit switch", but I'm not sure they would allow this on the Uplink side. I can try to ask them though ...
Right now stuff is "kinda" working (with like 2 out of 3 TCP/UDP/ICMP Traceroutes working, but NOT All 3, both for IPv4 and IPv6), although we can agree it's a bit of a weird "fashion" with:
- IPv4 in Bridge Mode (IPv4 Forwarding DISABLED since it's not currently needed)
- IPv6 in Brouter (Bridge+Router) Mode and IPv6 forwarding ENABLED
(I just discovered some stuff on Ubuntu where /usr/bin/traceroute pointed to a different location than what I have on Proxmox VE -> Different traceroute / traceroute6 executable, different behavior, different results)
The other alternative, as suggested previously, would be to pass ALL Traffic to the OPNSense VM using the existing NIC. And then access the Proxmox VE Host from there ... But again, if OPNSense VM breaks down (it broke down in the middle of an update a few days ago, I had to reinstall all packages from the Proxmox VE noVNC Console / Web Console for the VM), then I cannot repair anything anymore (besides going in Hetzner Rescue Console, disable PCIe Passthrough, then go back to the old setup [the one I currently have], fix OPNSense, then switch back PCIe passthrough and reboot again).
But your solution would not work with single additional IP, correct, since you need 1 IP of each Subnet for Gateway and 1 IP for Broadcast, right ?
Quote from: luckylinux on May 23, 2024, 11:53:29 AM
But your solution would not work with single additional IP, correct, since you need 1 IP of each Subnet for Gateway and 1 IP for Broadcast, right ?
Correct. 5 possible VMs with public IPv4, if you order a /29.
Quote from: luckylinux on May 23, 2024, 11:53:29 AM
The other alternative, as suggested previously, would be to pass ALL Traffic to the OPNSense VM using the existing NIC. And then access the Proxmox VE Host from there ... But again, if OPNSense VM breaks down ...
At least you can get a remote KVM for up to three hours for free if available. Works pretty well. I don't know if you can fix a broken Proxmox/OPNsense setup that way, but I would guess so.
Quote from: Patrick M. Hausen on May 23, 2024, 11:55:48 AM
Quote from: luckylinux on May 23, 2024, 11:53:29 AM
But your solution would not work with single additional IP, correct, since you need 1 IP of each Subnet for Gateway and 1 IP for Broadcast, right ?
Correct. 5 possible VMs with public IPv4, if you order a /29.
Quote from: luckylinux on May 23, 2024, 11:53:29 AM
The other alternative, as suggested previously, would be to pass ALL Traffic to the OPNSense VM using the existing NIC. And then access the Proxmox VE Host from there ... But again, if OPNSense VM breaks down ...
At least you can get a remote KVM for up to three hours for free if available. Works pretty well. I don't know if you can fix a broken Proxmox/OPNsense setup that way, but I would guess so.
Or double NAT with iptables & masquerading when using a single additional IPv4 Address in Routed Setup I guess (Proxmox VE Routes the /32 single IPv4 Address to say vmbr2 which is say 192.168.100.1, and then OPNSense VM takes it from there as an additional Interface).
EDIT 1: To be honest, for all the times I got locked out of the Server, I was ALWAYS (knock on wood :D) able to Recover it from the Rescue Console. Just needed to git clone my set of Scripts for Linux Setup / Recovery (https://github.com/luckylinux/linux-setup), copy a Config File a have at hand at Home, then run the mounting and access the Proxmox VE Host filesystem to fix then Configuration ;).
Now you are leaving my area of expertise because I have no idea how these vmbr things work exactly.
I always aim for a clean network setup even if more expensive, but then I work in a professional/enterprise context. We just order all hosts with a /29 by default and calculate that in.
Quote from: Patrick M. Hausen on May 23, 2024, 12:02:12 PM
Now you are leaving my area of expertise because I have no idea how these vmbr things work exactly.
I always aim for a clean network setup even if more expensive, but then I work in a professional/enterprise context. We just order all hosts with a /29 by default and calculate that in.
Fair enough, thanks for the tips ;).
Right now it's definitively NOT clean but I'd say it's working. UDP Traceroute needed a "Reject" Rule at the last hop on OPNSense to make it work. Still stumbled by the inconsistency on the TCP Implementation that OPNSense uses (traceroute -T flag actually uses ICMP ... duh).
I contacted Hetzner and they forwarded my Ticket to another Department. Let's see if they come back with something positive & not too expensive. I'm just a Homelabber after all ;D.
EDIT: Nope, not easy and not cheap :(. Either separate MAC (which I am already using with all the issues that come with it), or vswitch (but that's more expensive, traffic is limited, etc).
Maybe I'll just have to keep the "weird" setup as it is now. Plus maybe some fixes as you suggested @Patrick with Regards to IPv6.
Hey got same Problem with 3 IP's and Same Gateway, got ist solved.
You need seperate MAC for every IP and set it to the seperate Interface in Proxmox VM Conf for Opnsense.
Interfaces conf file add to Main vmbr0 with Main-IP
up ip route add WAN2-IP dev vmbr0
up ip route add WAN3-IP dev vmbr0
In Opnsense create Interfaces and set IP4 on DHCP then you are able to get same Gateway for Multiple Wan Interfaces. What a Heck :-)
Try and Error multiple Times and a lot of Beer :-)
Hope it helps you.
Quote from: whenthelight on May 25, 2024, 10:42:29 PM
Hey got same Problem with 3 IP's and Same Gateway, got ist solved.
You need seperate MAC for every IP and set it to the seperate Interface in Proxmox VM Conf for Opnsense.
Interfaces conf file add to Main vmbr0 with Main-IP
up ip route add WAN2-IP dev vmbr0
up ip route add WAN3-IP dev vmbr0
In Opnsense create Interfaces and set IP4 on DHCP then you are able to get same Gateway for Multiple Wan Interfaces. What a Heck :-)
Try and Error multiple Times and a lot of Beer :-)
Hope it helps you.
Thanks for the Reply. I'm not fully sure this is correct either though.
It seems you are using a Bridge+Router Configuration (Brouter), and on top of that you are using the same vmbr0 so you are not routing between interfaces. You are using a Bridge (with Separate MAC for OPNSense) linked to the Physical Interface (eth0 in my case), which is different than my "pure" Bridged IPv4 Configuration. Although, as said before, I am using it in "Reverse" now (OPNSense spoofing the MAC Address of the Ethernet Interface, Proxmox VE Bridge using one of the additional MACs). But, if you are at Hetzner, you need to beware of the MAC Abuse Emails as well in your case, since in the end it's still (mostly) a Bridged Configuration.
It seems a bit what I was doing with the IPv6 /64 Subnet originally, where I was using the IPv6 IP from vmbr0 in OPNSense, but I am also not sure that was correct.
Routing within the same Interface (vmbr0) or Separate Interfaces (vmbr0 <-> vmbr1) ? Not sure ... For IPv4 and related MAC Abuse Emails I think you are not out of the Woods yet :(.
This is my updated /etc/network/interfaces
auto lo
iface lo inet loopback
#auto eth0
iface eth0 inet manual
auto vmbr0
iface vmbr0 inet static
hwaddress XX:XX:XX:XX:XX:XX
address XXX.XXX.XXX.proxmox
netmask 255.255.255.192
gateway XXX.XXX.XXX.hetznergateway
bridge-ports eth0
bridge-stp off
bridge_waitport 0
bridge-fd 0
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
pre-up ip addr flush dev eth0
post-up ip addr flush dev eth0
iface vmbr0 inet6 static
hwaddress XX:XX:XX:XX:XX:XX
# First Address of the Main /64 Subnet
address 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001
netmask 128
gateway fe80::1
bridge-mcsnoop no
bridge-stp off
bridge_waitport 0
bridge-fd 0
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
pre-up ip addr flush dev eth0
post-up ip addr flush dev eth0
up ip -6 route add fe80::1 dev vmbr0
up ip -6 route add default via fe80::1 dev vmbr0
down ip -6 route del default via fe80::1 dev vmbr0
down ip -6 route del fe80::1 dev vmbr0
auto vmbr1
iface vmbr1 inet static
address 0.0.0.0
netmask 255.255.255.255
bridge_ports none
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
iface vmbr1 inet6 static
# First Address of the separate :0001 /80 Subnet of the Main /64 Subnet
address 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001
netmask 80
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge-mcsnoop no
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
up ip -6 route add 2a01:XXXX:XXXX:XXXX:0000::1/128 via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 0
down ip -6 route del 2a01:XXXX:XXXX:XXXX:0000::1/128 via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 0
up ip -6 route add 2a01:XXXX:XXXX:XXXX:0001::/80 via 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001 dev vmbr1 metric 256
down ip -6 route del 2a01:XXXX:XXXX:XXXX:0001::/80 via 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001 dev vmbr1 metric 256
up ip -6 route add 2a01:XXXX:XXXX:XXXX::/64 via 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0010 dev vmbr1 metric 500
down ip -6 route del 2a01:XXXX:XXXX:XXXX::/64 via 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0010 dev vmbr1 metric 500
up ip -6 route add default via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 800
down ip -6 route del default via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 800
auto vmbr2
iface vmbr2 inet static
address 0.0.0.0
netmask 255.255.255.255
bridge_ports none
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
iface vmbr2 inet6 static
# First Address of the separate :0000 /64 Subnet of the Additional /56 Subnet
address 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0001
netmask 64
bridge_stp off
bridge_waitport 0
bridge_fd 0
bridge-mcsnoop no
bridge-disable-mac-learning 1
bridge-unicast-flood off
bridge-multicast-flood off
bridge-vlan-aware yes
bridge-vids 2-4096
up ip -6 route add 2a01:XXXX:XXXX:YYYY::/64 via 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0001 dev vmbr2 metric 256
down ip -6 route del 2a01:XXXX:XXXX:YYYY::/64 via 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0001 dev vmbr2 metric 256
up ip -6 route add 2a01:XXXX:XXXX:YYYY::/56 via 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0010 dev vmbr2 metric 500
down ip -6 route del 2a01:XXXX:XXXX:YYYY::/56 via 2a01:XXXX:XXXX:YYYY:0000:0000:0000:0010 dev vmbr2 metric 500
up ip -6 route add default via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 800
down ip -6 route del default via 2a01:XXXX:XXXX:XXXX:0000:0000:0000:0001 metric 800
I ordered an additional /56 IPv6 Subnet, so right now I am using:
- Bridging for IPv4 (net0 / vtnet0 on OPNSense is WAN_IPv4)
- Routing for IPv6 (net2 /vtnet2 on OPNSense is WAN_IPV6_SUBNET_56 and net3 / vtnet3 on OPNSense is WAN_IPV6_SUBNET_64)
However, strangely enough, the WAN_IPV6_SUBNET_64 stubbornly refuses to work in OPNSense, the Gateway can be pinged but doesn't lead to "outside access" (e.g Google DNS Servers). I am using /80 for routing Traffic (which should be fine without SLAAC and Static IPv6 only), but for some reason OPNSense -> Google IPv6 DNS Servers is not working.
You can argue I'm being a bit nitpicky, as with a /56 IPv6 Subnet I should have 256 /64 Networks :D.
But still, kinda weird that it does not work.
Actually ... in OPNSense VM both the /64 and the Additional /56 IPv6 Subnet I purchased work correctly, it's just that one Gateway appears down / unpingable (typically /64 Gateway appears down), if the other Gateway is used instead (I guess OPNSense cannot have 2 "default" Gateways, which is fair).
Although it's weird that they don't allow to monitor an External IP when the Gateway isn't being used ???.
Is this a BUG or a "Feature" ?
I lost a couple of Days trying to understand why I cannot monitor 2 different Google DNS Servers ...
Turns out you need to [UNCHECK] Gateways -> <your Gateway> -> "Disable Host Route", so that traffic to that Monitor IP is forced out through the Specified Gateway. Of course this applies to ALL Traffic (even if you do a "manual" Traceroute etc), but unless you do that, the Gateway will appear down (most likely because the traffic will just go through the other Gateway).