Multiple WAN IP Addresses with different MAC but same Gateway on Hetzner Network

[luckylinux]

I finally managed to get the first WAN IPv4 Address working on (Virtualized) OPNSense on top of Proxmox VE on Hetzner Dedicated Server  .

Now, I ordered an additional Single IPv4 Address from Hetzner Robot, and apparently this is where the issue start.

To summarize I have the following IPs:

• (NOT OPNSense) xxx.xxx.xxx.proxmox (Main IPv4 Address, cannot get a dedicated MAC Address)
• vtnet0: xxx.xxx.xxx.opnsense_01 (First Additional IPv4 Address, with a dedicated MAC Address)
• vtnet2: xxx.xxx.xxx.opnsense_02 (Second Additional IPv4 Address, with a dedicated DIFFERENT MAC Address OR without MAC Address at all)

I ended up in the same /26 Subnet as the "Main" Additional IP. The Gateway has therefore the same IP Address.

As a first Approach, I tried to simply add a new Network Interface to the OPNSense VM mapped to the new MAC Address, then configure the Interface in OPNSense using DHCPv4. Weird stuff happened with regards to the ARP Table, with the IP getting associated to both vtnet0 (Main WAN for OPNSense, configured via DHCPv4 ALWAYS) and vtnet2 (this additional IP Address with DIFFERENT MAC Address).

That didn't work, most likely because Multi-WAN is NOT Supported on Multiple Interfaces with the same Gateway ? Some posts on OPNSense Forum as well as PFSense Forum/Tutorials seem to indicate that this is NOT possible with PF on FreeBSD. Also https://docs.netgate.com/pfsense/en/latest/multiwan/considerations.html#multiple-wans-sharing-a-single-gateway-ip seem to suffest that it's only doable with NAT.

I also tried to add Static IPv4 for the Interface, then creating a separate Gateway (or trying with "Dynamic gateway policy - This Interface does not required an Intermediary System to act as a Gateway"), but it will NOT work  .

I tried, as an alternative, to add simply the new IPs as a Virtual IP / IP Alias. The vtnet0 (wan) interface gets both IPs, but nothing happens  .

My guess is that Hetzner checks that the SRC MAC and SRC IP Address match (MAC/IP Address Filtering). So I will get yet another MAC Address Abuse Email soon most likely .

To recap, this is done via Proxmox VE vmbr0 Linux Bridge, therefore (in the first approach) the vtnet0 (main OPNSense WAN Interface) and vtnet2 (secondary OPNSense WAN Interface) are effectively in the same L2 Subnet (that's why you can see them all in the ARP Table).

But I think that unless the SRC MAC and SRC IP "match" what Hetzner is expecting, traffic from the Server just gets dropped.

I couldn't ping TO the new WAN IP from my Home Network, neither could I ping FROM the new WAN IP in Interfaces -> Diagnostic -> Ping.

Those tests also failed when I removed the dedicated MAC Address in Hetzner Robot (so the traffic for that IP should be bound to the Proxmox VE Host now).

What would be the best Solution to solve this, assuming I cannot get Hetzner to "point" to the same MAC Address as the Main OPNSense WAN IP ?

From what I can see, either:
- Proxmox VE Host: Configure 2nd OPNSense IP as "Routed" -> NAT to another Linux Bridge say 192.168.100.1 -> OPNSense Configures an additional Interface there (double NAT most likely needed)
- OPNSense VM: Configure 2nd IP as "Routed" to an Intermediary Network -> (double NAT most likely needed)

Or is there some "weird" way to add a "Virtual" Interface ? Or possible a WAN / Transparent Filter Bridge or even a "LAN Bridge" ?

[Patrick M. Hausen]

Can you use the second IP address with the same MAC address as the first one as far as Hetzner is concerned?

If yes, then simply add it under Interfaces > Virtual IPs to your existing WAN.

You cannot have addresses from the same subnet on two different interfaces.

[luckylinux]

Can you use the second IP address with the same MAC address as the first one as far as Hetzner is concerned?

If yes, then simply add it under Interfaces > Virtual IPs to your existing WAN.

You cannot have addresses from the same subnet on two different interfaces.

I just asked if it's something they can do. By the automatic Robot Service, this does NOT appear to be possible.

Either I can click on "Request a separate [DIFFERENT] MAC Address", or "Reset MAC" (which "deletes" the separate [different] MAC Address and re-binds that IP to the Main IP of the Server).

So I would need to do some Routing ?

Or possibly a 2nd OPNSense VM ? The advantage with a 2nd OPNSense VM would be, if nothing else, that it could avoid Double NAT.

(on a separate note: for IPv6 I can configure the /64 Subnet to POINT to one of these already-existing MAC Addresses, didn't manage to get it working though. For IPv4 this is however NOT possible to do)

[Patrick M. Hausen]

Yes, so you use the main IP of your server for OPNsense and all alias IP addresses assigned to that with the same MAC and you use one extra IP with a different MAC for Proxmox ...

[luckylinux]

Yes, so you use the main IP of your server for OPNsense and all alias IP addresses assigned to that with the same MAC and you use one extra IP with a different MAC for Proxmox ...

Just got the reply from Hetzner. Unfortunately it's NOT possible (see attached).

So either I'll have to enable Routing on the Proxmox VE Host and Forward from there, or spin up another Router/Firewall VM (be it OPNSense or maybe OpenWRT etc) and do the routing there using a separate MAC Address  .

[Patrick M. Hausen]

That's what I wrote: if no virtual MAC is set it uses the same as the main IP.

So use main IP for OPNsense and an additional with virtual MAC for Proxmox ...

HTH,
Patrick

[luckylinux]

That's what I wrote: if no virtual MAC is set it uses the same as the main IP.

So use main IP for OPNsense and an additional with virtual MAC for Proxmox ...

HTH,
Patrick

You don't understand ... This is what I am doing for my Main Server IP (Proxmox Host) AND My FIRST Additional IP for OPNSense VM  .

The Issue right now is how to handle the next (SECOND Additional or THIRD in Total) IP for OPNSense  .

EDIT 1: Re-reading your Previous Post ...

>>  you use one extra IP with a different MAC for Proxmox ..
That would be possible

>> you use the main IP of your server for OPNsense and all alias IP addresses assigned to that with the same MAC
That is NOT possible. The Main IP is tied to the Server physical MAC Address. There is no option to change that. Unless of course you are suggesting spoofing the Server Hardware MAC Address into the OPNSense VM, that is NOT going to work ...

[Patrick M. Hausen]

Unless of course you are suggesting spoofing the Server Hardware MAC Address into the OPNSense VM, that is NOT going to work ...

I am suggesting exactly that. Spoofing the MAC of the Proxmox host to the additional virtual MAC for the second IP address, then spoofing the MAC address of the OPNsense VM to the original one of the physical server.

[luckylinux]

Unless of course you are suggesting spoofing the Server Hardware MAC Address into the OPNSense VM, that is NOT going to work ...

I am suggesting exactly that. Spoofing the MAC of the Proxmox host to the additional virtual MAC for the second IP address, then spoofing the MAC address of the OPNsense VM to the original one of the physical server.

Ah ... So basically what I wanted to do initially, but reversing the role of Proxmox and OPNSense VM.

Won't this cause additional issues though, making Hetzner send me yet other MAC Abuse Emails ?

Or for all intents and Purposes, since Hetzner will receive all traffic from the eth0 Port of the Linux Bridge (Main MAC of the Host, which you suggest me to Spoof into OPNSense, making the Additional IP MAC Address take the identity of the Linux Bridge / Proxmox Host instead), all traffic will appear from being originated from eth0 / Main MAC anyways ?

Or since this is simply L2 Networking and not Routing, it doesn't really matter (there is no "Network" Address "Translation", just a switch and 2 MACs sending/receiving stuff) ?

[Patrick M. Hausen]

Or since this is simply L2 Networking and not Routing, it doesn't really matter (there is no "Network" Address "Translation", just a switch and 2 MACs sending/receiving stuff) ?

I guess so - all of that is just a suggestion. I do run OPNsense at Hetzner, just not virtualised.

EDIT: thinking about my two node OPNsense HA cluster at Hetzner ...

You could IMHO:

- order a vswitch for your proxmox host
- define the vswitch VLAN in proxmox
- pass that as a virtual interface to OPNsense as WAN
- order a /29 or whatever connected to that vswitch and use these addresses in OPNsense

[luckylinux]

Or since this is simply L2 Networking and not Routing, it doesn't really matter (there is no "Network" Address "Translation", just a switch and 2 MACs sending/receiving stuff) ?

I guess so - all of that is just a suggestion. I do run OPNsense at Hetzner, just not virtualised.

Well ... That would actually be the only way to make this work, otherwise I'd have to do Routing on the Host or Routing with Additional VM.

Alternatively I could ask for an Extra NIC + LAN Connection, totalling approx. 5 EUR/month extra.

Or if I want to try my luck, just pass the NIC directly to OPNSense via PCIe Passthrough (with the **big** Issue that if the OPNSense VM goes down or breaks, I cannot even access the Host).

[luckylinux]

Or since this is simply L2 Networking and not Routing, it doesn't really matter (there is no "Network" Address "Translation", just a switch and 2 MACs sending/receiving stuff) ?

I guess so - all of that is just a suggestion. I do run OPNsense at Hetzner, just not virtualised.

EDIT: thinking about my two node OPNsense HA cluster at Hetzner ...

You could IMHO:

- order a vswitch for your proxmox host
- define the vswitch VLAN in proxmox
- pass that as a virtual interface to OPNsense as WAN
- order a /29 or whatever connected to that vswitch and use these addresses in OPNsense

I just tried "reversing" / spoofing as you suggested, so everything goes to the OPNSense VM (original MAC that the Host has) except one additional IP that goes to a dedicated MAC which I assigned to the Linux Bridge vmbr0 (Proxmox VE Host).

It **seems** to work  . Although I guess too early to tell still  .

[luckylinux]

@Patrick: Did you have any special Configuration needed for IPv6 to work with Hetzner ?

I couldn't manage to get it working. Neither on Proxmox VE, neither on OPNSense (tried both MAC Addresses to "point" the /64 Subnet to the right Appliance).

Maybe it's just because I'm used to IPv4, where Gateway and Address are in the same Subnet, but their Default Gateway fe80::1 doesn't play Ball at all. Interfaces -> Diagnostics -> NDT returns only the Local/Global (if set Static, otherwise none) IPv6 Addresses. DHCPv6 Configuration of the WAN Interface doesn't get an IP Address.

Strangely enough, I can see in OPNSense Firewall Logs that my Home Address is managing to Ping6 through (inbound), but I do not see any outbound reply (and Ping6 fails).

I had this issue previously with IPv4 as well (Inbound Traffic showing up in the logs, but Outbound not working), which I solved by setting Firewall -> Settings -> Advanced -> Disable reply-to on WAN rules -> CHECK

For IPv6 it's not working though. I tried to set Static IPv6 but nope. I cannot ping the gateway.

Probably the OPNSense VM cannot receive the Route Advertisement from the fe80::1 Gateway ? I don't see anything in the Proxmox VE Firewall Logs though ...

EDIT: I see lots of traffic on the Loopback Interface, not sure if this is normal. I don't really see anything going Out of the Firewall though. Loopback Interface has Source = Destination IPv6 Address, corresponding to the Static WAN IPv6 Address I had set.

[luckylinux]

Making progress somewhat ...

On Proxmox VE Host I added this to /etc/network/interfaces:

Code: [Select]

#auto eth0
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        hwaddress XX:XX:XX:XX:XX:XX
        address 94.XX.XX.XX
netmask 255.255.255.192
gateway 94.XXX.XXX.XXX
#pointopoint 94.XXX.XXX.XXX
bridge-ports eth0
bridge-stp off
        bridge_waitport 0
bridge-fd 0
        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096
        pre-up ip addr flush dev eth0
        post-up ip addr flush dev eth0


iface vmbr0 inet6 static
        hwaddress XX:XX:XX:XX:XX:XX
        address 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001
        netmask 96
        gateway fe80::1
        bridge-mcsnoop no
        bridge-ports eth0
        bridge-stp off
        bridge_waitport 0
        bridge-fd 0
        bridge-disable-mac-learning 1
        bridge-unicast-flood off
        bridge-multicast-flood off
        bridge-vlan-aware yes
        bridge-vids 2-4096
        pre-up ip addr flush dev eth0
        post-up ip addr flush dev eth0


On OPNSense WAN Interface I set Static IPv6:

Code: [Select]

address: 2a01:XXXX:XXXX:XXXX:0001:0000:0000:**0010**
netmask: 96
gateway: 2a01:XXXX:XXXX:XXXX:0001:0000:0000:0001
Gateway fe80::1 would NOT work in OPNSense. I guess that's because Routing (of Subnet) is required. I read on Hetzner Docs that Bridging is supported for single IPv4 Addresses, but when dealing with an IPv4 Subnet, Routing is required. Well I guess it makes sense, if OPNSense IP is NOT in the same subnet as the Remote (Google DNS Server) Destination, then the Host must perform Routing, right ?

On Proxmox VE Host I enable IPv6 Forwarding. Maybe this was the most critical bit missing ? I could IPv6 ping between Proxmox VE Host and the OPNSense VM without Issues without, but as soon as trying to Ping e.g. Google DNS Server 2001:4860:4860::8888 it would fail.

Code: [Select]

sysctl -w net.ipv6.conf.default.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.default.forwarding=1
I don't think this brings me closer to a solution however ...

From what you @Patrick told me in the Past, IPv6 Networks are ALWAYS /64. I used /96 here because I was trying to follow the Instructions https://community.hetzner.com/tutorials/distributing-ipv6-over-libvirt-guests which, together with Host IPv6 forwarding Enabled, does Indeed work.

But I guess if I really want to do it "right", I need to order a /56 IPv6 Subnet, in that way I can have a /64 Subnet for WAN on Proxmox VE Host and OPNSense VM, then several other /64 Subnets for DMZ1/DMZ2/LAN1/LAN2/.... on OPNSense.

@Patrick: sorry If I didn't say it explicitely until now, but THANK YOU VERY MUCH for all your Help !

[Patrick M. Hausen]

Gateway fe80::1 would NOT work in OPNSense. I guess that's because Routing (of Subnet) is required.

It definitely does. An IPv6 interface will always have a link local address in addition to the global unicast address you assigned. So it can use another link local address on the same interface just fine.

See attached screenshots. This is OPNsense on bare metal, though.